.org Reassignment: Preliminary Staff Report on Evaluation of the Proposals for Reassignment of the .Org Registry
Posted: 19 August 2002
Preliminary Staff Report on Evaluation of the Proposals for Reassignment of the .Org Registry
This Preliminary ICANN Staff Report documents the results of the solicitation and evaluation process used to assist in the ICANN Board's selection of an entity to assume responsibility for operating the .org registry starting 1 January 2003 from the current operator of that registry, VeriSign, Inc. It is now posted for comments of the bidders and community prior to finalization and submission to the ICANN Board for its decision. Public comments on this draft report should be submitted by e-mail to email@example.com on or before 29 August 2002. A final version of this Staff Report, taking into account comments received, will be posted on 5 September 2002, and comments will also be invited on that final version.
Eleven strong proposals were received in response to the Request for Proposals issued by ICANN on 20 May 2002. Each bidder clearly took the task very seriously and invested considerable effort in proposal preparation and submission. ICANN should be very grateful for both the interest demonstrated and the effort and resources expended.
Unfortunately, only one bidder can be successful. ICANN owes it to those who submitted bids, therefore, to conduct as fair, thorough, impartial, open and transparent a process as is reasonably possible. Every effort has been made in the solicitation and evaluation processes to ensure that is the case. No process can be perfect there is always room for improvement and, indeed, a longer timescale might have produced more bids or more complete bids, and a more detailed evaluation. However, we have great confidence in both the bid process and in the evaluation methodology and results. We believe the evaluation process has been fair and impartial, and that more time for evaluation or a different approach would not have led to a different conclusion.
Because the selection criteria span a variety of subjects, the evaluation was conducted using a multi-team approach. The teams were:
These evaluators many of whom graciously volunteered their time worked very hard and under very tight deadlines to produce their evaluations. ICANN appreciates their efforts.
This report summarizes the results of the evaluation. Each of the three teams evaluating the technical and usage aspects produced three-tier rankings of the bids. Only one of the proposals that submitted by the Internet Society (ISOC) was accorded top-tier ranking by all three teams. Based on that fact, and consideration of procedural aspects, ICANN staff's preliminary (subject to public comment) recommendation is that the proposal submitted by the Internet Society (ISOC) be selected.
The bid solicitation arose from the revisions to the agreements among ICANN, VeriSign, and the U.S. Department of Commerce that were approved by the ICANN Board at its meeting on 2 April 2001 and signed in May 2001. One of the provisions of that agreement was that VeriSign would relinquish responsibility for operating the .org registry to an entity of selected by ICANN at the end of the calendar year 2002. As part of that provision, VeriSign also agreed to provide a US$5M endowment to be used to fund future operating costs of the successor registry operator, provided it is a non-profit entity.
In response to this provision, ICANN launched an open and transparent bid solicitation and evaluation process that was announced on 22 April 2002. Full details of this announcement, of the subsequent steps followed, and of the bids received can be found at the ICANN website in the Materials on .Org Reassignment. The bid solicitation was authorized by the ICANN Board at its meeting in Accra, Ghana in March 2002 following a report and recommendations it had received from the Domain Name Supporting Organization (DNSO). At that meeting, the ICANN Board stated that primacy of consideration should be given to stability of transition and operation of the .org registry so that there be no service interruptions in the .org registry.
As a result of the final Request for Proposals that was issued by ICANN on 20 May 2002, eleven proposals were received on or before the 18 June 2002 deadline. These and all subsequent materials received by the bidders are posted on the ICANN Website at http://www.icann.org/tlds/org/. Open and transparent procedures were maintained throughout the evaluation process. Bidders were requested to communicate only in writing with ICANN staff or board members, and any materials received from any of the bidders were posted on the website. All bidders were invited to make a brief presentation on their bid in the special ICANN Public Forum held for the purpose on 26 June 2002 in Bucharest, Romania. The process allowed for written questions to be submitted by prospective bidders prior to final submission of bids; all questions and answers were posted on the website.
This report summarizes the process used to evaluate the bids received and presents the resulting staff recommendation to the Board.
Eleven bids were received. Six of these were from not-for-profit organizations, most of which had obtained the commitment of other operating registries ("back-end operators") to operate the .org registry on the bidder's behalf should the bidder be successful, but the bidder would retain overall policy direction and community interface, that is, overall responsibility and accountability would remain with the bidder. All six not-for-profit bidders and one for-profit bidder plan to seek the US$5M endowment to be provided by VeriSign to assist with operating costs. One of the six (UIA) proposed to employ the services of the VeriSign registry should UIA be awarded the reassignment.
The eleven proposals in alphabetic order and, where applicable, associated back-partners including "back-end" registry operators are as follows:
Table 1: Proposal Submissions and Associated Primary Partners
The RFP stated eleven criteria that would be used in assessing the proposals. These are listed here for ease of reference:
Criteria 1, 7, 8, and 9 are primarily technical in nature and are referenced here as Technical Criteria. Criteria 4, 5 and 6 are focused on the extent to which the bidders address the needs of non-commercial registrants consistent with the primary purposes of the .org registry; these are referenced here as Usage Criteria. Criteria 2, 3, and 10 are primarily procedural in nature and are designated as Procedural Criteria. Criterion 11 is in a category of its own and is addressed separately.
The decision was made to select different teams to evaluate each set of criteria, since different expertise was required in each case. In fact, as described below, two Technical Evaluation Teams were selected that operated independently and without knowledge of each other in order to lend confidence to the final results. For reasons described below, ICANN's General Counsel evaluated the Procedural Criteria. Criterion 11 was also assigned to one of the Technical Evaluation teams for reasons described below.
The reports of the evaluation teams can be viewed at the links below:
ICANN staff analyzed these reports, and synthesized the results into the final staff recommendation to the ICANN Board as presented in this document. During the evaluation process, staff were available to answer questions of the evaluation teams and to clarify terms and definitions or any issues associated with the RFP. Staff also addressed any issues regarding potential conflicts of interest that arose in one or two cases (there were no actual conflicts of interest, in fact).
An attempt was also made to create an international panel of experts who could provide e-mail answers to technical questions regarding registry operations. This proved impractical in the short timeframe available, particularly given the small pool of large registry operators who were not themselves part of one or more of the proposals (and also given the influence of summer vacations!).
The proposals were all required to follow a particular format to ease comparison and evaluation. Section C of the responses consisted of answers to 36 questions (several with subparts) that were intended to elicit information regarding how well the bidder met one or more of each criterion. Staff also provided to the evaluation teams a mapping of these so-called "C-questions" into the criteria, that is, for each criterion this mapping indicated the list of relevant C-questions that should primarily be considered in the evaluation team's analysis. This mapping was intended to be helpful, but by no means binding, that is, an evaluation team could consider any part of each response and related posted material in reaching their conclusions.
The processes used by each evaluation team are summarized below. Prior to the posting of this report, the identity of each team and of its members has not been made publicly available. This was to ensure as far as possible that each team could work quietly through their evaluations based on the posted materials without any risk of being importuned by anyone who may have an interest in the outcome of the process.
ICANN should also be grateful to all the members of the evaluation teams listed in this Report who worked very hard under a very tight deadline to produce their recommendations. Many of these individuals are volunteers who graciously made their time available for this activity.
The evaluation reports can be directly accessed and speak for themselves. Although each evaluation took a different approach, both the technical and the usage evaluations resulted in classifying each proposal into one of three overall tiers. The exact definitions of these tiers varied among the evaluations, as did the methodologies for integrating the applicable criteria. In Table 2, however, we summarize the proposals according to these tiers, using A, B, C as indicators for the categorization, while again emphasizing that these indicators represent different metrics for each evaluation and can only be interpreted in detail by reference to the reports themselves. In fact, two separate columns are presented for the Usage Evaluation representing the results of two different approaches used in that evaluation.
Table 2: Summary of Rankings3 of Evaluation Teams
Again, it must be emphasized that these designations cannot be readily combined into a single result for each proposal.
Conclusions and Recommendation
Notwithstanding the caution just expressed, it is apparent from Table 2 that only one proposal, namely the ISOC proposal, was ranked top-tier in each evaluation, according to each evaluation's team definition of such a designation. That is the fundamental basis for a recommendation to the Board that the ISOC proposal be selected and that the Board authorize the President to proceed to negotiate an agreement based on that proposal.
However, further comments are in order. The ISOC proposal is also in the "A" category as evaluated by Gartner with respect to Criterion 11 addressing completeness of proposal and soundness of plans. ICANN Counsel has also validated that the ISOC proposal meets the requirements of Criterion 2 insofar as compliance with ICANN-developed policies is concerned: indeed, the ISOC "back-end" operator, Afilias, has already demonstrated this in its operation of the recently launched .info registry. Counsel has also opined that this proposal should qualify for the VeriSign endowment (Criterion 10).
The "back-end" operator contracted by ISOC, Afilias, is a consortium of eighteen gTLD registrars. VeriSign is a minority (5.6%) shareholder of Afilias as one of these registrars. Because the other Afilias shareholders are VeriSign's competitors, however, VeriSign's ability to exercise control over Afilias is effectively minimized and, indeed, no VeriSign employee has been elected to Afilias' Board of Trustees/Directors. In these circumstances, VeriSign's minority stake in Afilias does not materially implicate Criterion 3 (Enhancing Competition for Registry Services), particularly in view of the fact that the .org registry would be assigned to ISOC, not Afilias.
From Table 2, it can be seen that there are other strong proposals. The Board has stated that it gives primacy to consideration of continuing stability of operation of the .org registry as indicated through demonstrated experience. This gives special weight to the technical evaluation of Criterion 1 (need to preserve a stable, well-functioning .org registry) and to an extent Criterion 9 (transition considerations). In other words, proposals that rank very high with respect to these criteria should be given primacy of consideration above all others, and vice versa. The Gartner evaluation ranks the NeuStar proposal highest in these categories, followed by ISOC and then UIA (Criterion 1) or GNR (Criterion 9). However, NeuStar, UIA, and GNR all rank in the "B" category of the Usage Evaluations, and the last two also fall in the "B" category of the Academic CIO Technical Evaluation. The UIA proposal also falls short of the other proposals with respect to Criterion 3, in enhancing competition. In considering the GNR proposal, it is also appropriate to recognize that its experience in operating a shared registry is demonstrably less than that of Afilias (ISOC), NeuStar, or VeriSign (UIA), in that the .name registry is an order of magnitude smaller than the .info (Afilias), .biz/.us (NeuStar), or certainly any of the VeriSign operated gTLD registries; furthermore, as described in the General Counsel's evaluation of Criterion 3, GNR's shared (real-time) mode operation of the .name registry has used a VeriSign "back-end", that is, not using the technology that GNR proposes to use for .org.
Nevertheless, further consideration can be given to one of these proposals – particularly the NeuStar and GNR proposals (because of the registry diversification Criterion 3 issue with the UIA proposal) – if it proves impossible to negotiate an agreement with ISOC should the Board approve this recommendation.
Conversely, the Unity proposal ranks at the top of the Usage Evaluation, and the IMS/ISC proposal ranks second in one approach in the Usage Evaluation. Both of these proposals, however, fall short in the Technical Evaluations, although the Unity proposal does fall in the "B" category of the Gartner Technical Evaluation.
In summary, although there are several strong proposals among the eleven submitted, the staff's view is that the ISOC proposal is the strongest and most balanced proposal overall. We recommend that the Board authorize the President to enter into negotiations immediately on a schedule that would allow Board approval of the negotiated agreement and ultimate transition of the registry so that ISOC could commence operations on 1 January 2003.
Should negotiations with ISOC fail in the allowed timeframe, staff recommends that the President be authorized to enter into negotiations with NeuStar and then GNR, in that order. NeuStar's and GNR's ranking could be perceived as about equal overall, but weight should be given to the greater experience of NeuStar in actually and successfully operating a large registry and having completed a transition of an actual registry (.us) from VeriSign.
We conclude this report by thanking all the many individuals and institutions that worked so hard and who expended significant resources in formulating and submitting proposals; and in evaluating the proposals. A competitive process such as this can only have one successful award. But this result should not detract from the overall effort and thoroughness of all the submissions.
Technical Evaluation Teams
Comments concerning the layout, construction and functionality of this site
should be sent to firstname.lastname@example.org.