.org Reassignment: Final Staff Evaluation Report on the Proposals for Reassignment of the .Org Registry

Posted: 23 September 2002

Final Staff Evaluation Report on the Proposals for Reassignment of the .Org Registry

Contents

1. Summary

2. Background

3. Comments Received

4. Summary of Revised Evaluations

5. Leading Proposals

ISOC

NeuStar

GNR

DotOrg and RegistryOrg

---

Unity

UIA

6. Recommendation

Appendix 1: Analysis of additional comments

1. Summary

This report to the ICANN Board of Directors represents the final ICANN staff analysis and recommendations regarding the selection of a new registry to replace VeriSign as the operator of the .org registry. It builds upon the "Preliminary Staff Report on Evaluation of the Proposals for Reassignment of the .Org Registry" that was posted for community comment on 19 August 2002. The background, the solicitation process followed, and the evaluation process to date were detailed in the Preliminary Report, along with other relevant information; we do not repeat that material here.

This report is now posted for community comment (comments should be sent to <org-eval@icann.org> by 8 October 2002) to inform the Board's final decision.

The challenge for the ICANN Board is to determine the best proposal among 11 excellent submissions that were received, taking all the many complex factors – as specified in the Request for Proposal (RFP) – into account. Several of the proposals would, if selected, lead to satisfactory – even excellent – results, which makes it difficult to select a single best proposal. Nevertheless, there can only be one registry operator, and so a selection must be made.

The purpose of this Final Staff Evaluation Report is to assist the Board in its decision by providing guidance that would help to ensure that the decision is as neutral and objective as possible, grounded in independent evaluations.

Eleven high-quality proposals were submitted; only one can be chosen. Undoubtedly, there will be ten disappointed bidders who put substantial effort into their proposals. There are also significant commercial interests at stake. This underscores why the Board's decision must be neutral and objective based on the written, open, and transparent record, not on factors that lie outside the RFP development, solicitation, and evaluation processes.

Although the final decision will be made by the Board, staff believes that it is appropriate to make its own recommendation based on its own analysis of the material submitted. The Board, of course, is free to accept or reject that recommendation, and make its own decision. Staff believes, however, that it has made every practicable effort to arrive at its recommendation objectively, relying on the evaluation reports themselves, not on extraneous subjective factors.

ISOC Recommendation: After careful consideration of all the valuable comments received on its Preliminary Report, staff, in this Final Staff Evaluation Report, maintains its initial recommendation (see Section 6 ("Recommendation")) that the Board reassign the .org registry to ISOC. Although the comments contained much useful insight and, indeed, pointed out some errors in the original evaluations, none of these errors was sufficiently material to change the ultimate conclusion. The ISOC proposal is still clearly ranked the highest overall in the evaluation, as shown by the independent evaluations conducted by the teams¹ against the criteria specified in the RFP. Staff relies solely on these evaluations for its recommendation, Additional comments, however, are provided by way of clarification.

It should be noted that although we use the term "ISOC Proposal" since it was submitted by the Internet Society, the actual registry agreement would be with a new entity, the Public Interest Registry (PIR). ISOC proposes to form a not-for-profit corporation, known as "Public Interest Registry" or "PIR" to carry out the .org registry functions. PIR would operate as a fully separate corporation from ISOC, with ISOC having responsibility for appointing PIR's Board of Directors. PIR would enter into an agreement with Afilias, the operator of the .info registry, to perform the technical aspects of registry operations.

Certain criticisms have been made against the ISOC proposal, including some that were external to the RFP. These criticisms include the financial condition of ISOC; certain technical considerations regarding the choice of database system by ISOC's operating partner, Afilias (a concern that has been also raised by the Gartner evaluation); and the comment that the Board has a conflict of interest were it to select ISOC because many Directors are members of ISOC. We believe the concerns merit discussion and are addressed in the section on ISOC in Section 5 ("Leading Proposals"). For the reasons presented in that section, we believe that the first of these concerns is not applicable; the second at worst presents, in Gartner's words, "a manageable risk;" and the third does not, in reality, exist.

Our recommendation is based on the assumption, as required by the RFP's Criteria for Assessing Proposals, that stability of operation is the primary objective to be achieved, both in the steady state and during the transition of the registry, and that "demonstrated ability to operate a TLD of significant scale" (see Criterion 1) be a key indicator of this.

Acknowledgments: As we did in the Preliminary Report, we would express our extreme gratitude to the eleven bidders and to all the members of the evaluation teams. This acknowledgment is more fully expressed in Section 7 ("Acknowledgements and Closing Observations"). We believe the analysis of the evaluation teams has been fair and impartial based on the information available, acting within the constraints of the RFP.

There is no such thing as a perfect evaluation and assessment, or a perfect bidding process. It is quite possible to spend extraordinary time and resources in an attempt to achieve an unattainable end, and those that have a reason to do so will exploit that fact to what they perceive as their own advantage. Any process is open to criticism and comment in the absence of (unattainable) perfection. There comes a time, however, when the best management judgment has to be exercised based on the best interpretation of the information available. This Final Staff Evaluation Report represents the ICANN staff's best assessment of that information and of its implications based upon the impartial recommendations of the evaluation teams.

2. Background

This report summarizes the final ICANN staff analysis and recommendations regarding the selection of a new registry to replace VeriSign as the operator of .org (also termed "dot org" to avoid confusion). It builds upon the "Preliminary Staff Report on Evaluation of the Proposals for Reassignment of the .Org Registry" that was posted for community comment on 19 August 2002. The contents of that report (particularly its discussion of background, solicitation process, and evaluation process) are relevant to this report, but for brevity are not repeated here.

The Preliminary Evaluation Report described the background of the three evaluation teams that were used as the basis for the preliminary conclusions of that Report. There were two technical evaluation teams (the Gartner team and the Academic CIO team) that focused on Criteria 1, 7, 8, and 9 of the RFP²; and one team concerned with evaluating Criteria 4, 5, and 6 that was termed the "Usage Evaluation Team." (The Usage Evaluation Team was a task force formed by the ICANN Non-Commercial Domain Names Holders Constituency (NCDNHC).) The ICANN General Counsel also conducted an evaluation of Criteria 2, 3, and 10 and some miscellaneous matters.

As noted in Section 3 ("Comments Received"), many detailed applicant and public comments were received on each of the team reports. These comments were carefully reviewed and appropriate adjustments made, as described in Section 4 ("Summary of Revised Evaluations"). For the reasons described there, only the reports of the Gartner Evaluation Team, the Usage Evaluation Team, and the ICANN General Counsel are used as the basis for this Final Evaluation Report.

Although a tentative recommendation for a new operator was stated in the Preliminary Report, we emphasized there that the selection was subject to change in this Final Report depending on public comments and other information received during the comment period. We also emphasized that the evaluation reports reflect only recommendations that may or may not be accepted by the ICANN Board of Directors. The final decision will be made by the Board in the near future. This Report is being posted now for public comment so that the Board may be fully informed when it makes its decision. The public comment period will close on 8 October 2002. Comments should be submitted by e-mail to <org-eval@icann.org>.

We emphasize that this is the staff's Final Evaluation Report. Further comments received will not affect this Report. There will be no further versions of the Report itself, although there may be supplemental explanations should they be warranted by comments received during the public posting period. Comments received, however, will be carefully considered by the ICANN Board in making its final decision.

3. Comments Received

Many comments were submitted between August 19 and now following posting of the Preliminary Report. These comments were directed to the Preliminary Evaluation Report itself and to the underlying evaluations that were the basis for preliminary recommendations contained in that Report. These comments were submitted both by those entities that had submitted proposals and by the general public. For the most part, the comments were targeted and thoughtful.

We appreciate all of the effort and consideration that went into these comments. We particularly appreciate the extra work by bidders in answering the many detailed questions that were posed by staff to help clarify their earlier proposals. Broadly speaking, the comments fall into four main categories:

1. Comments by the public at large.

2. Comments by bidders that pointed to errors in the evaluation reports, highlighted material in their proposals that may have been overlooked or misunderstood in the evaluations; agreed with (when the evaluations favored the bidder's proposal) or took exception to (when the evaluations did not favor the bidder's proposal) material contained in the evaluations; commented adversely on proposals of other bidders; and generally argued their individual cases. These comments were robustly presented as would be expected in an open bid solicitation process where there can only be a single successful proposal out of eleven strong submissions. Some comments contradicted each other, another hallmark of robust competition.

3. Comments by bidders that contained new material to strengthen their original proposal. Since this is a single stage, open RFP solicitation and selection process, these comments could not be accepted and have not been considered – as is indeed standard in such processes.

4. Responses to additional questions that were posted to provide material that clarified the original proposals that may have been incomplete or unclear. The need for such additional material and for analysis of this material has delayed this Report somewhat, but it was felt that the benefits of correct understanding outweighed the downside of additional delay.

As noted, comments that fell into the third category have not been considered so as to be fair to all bidders. Otherwise, all other bidders would have had to be afforded the same opportunity to submit new material and the opportunity to comment on new material submitted by other bidders. This would have extended the bid process even further.

Comments in the second category in turn broadly fell into two categories:

• Comments that correctly pointed out errors, or lack of adequate explanation, in the original evaluations or in the Preliminary Report. The Gartner and Usage Evaluation Teams and the General Counsel have corrected and elaborated their reports to address these comments as detailed in their revised reports. (The Usage Evaluation Team³ was comprised of individuals active in the Non-Commercial Domain Name Holders Constituency (NCDNHC) who graciously volunteered their time). We comment on the Academic CIO Report later. We appreciate all the careful analysis that brought these errors to light, which is one of many benefits of an extraordinarily open process such as being employed for the .org reassignment.
• Comments that were based on misunderstandings of the evaluations or that were not relevant to the evaluations. Where possible, these misunderstandings have been pointed out in documents accompanying the revised evaluations. Some comments were purely speculative about motive and intent, without any basis in fact. These have not been given any weight.

It should be noted that many detailed comments were received, and it is not feasible to discuss every single one in detail, although every point has been carefully considered. Many comments, however forcefully worded, were either not germane or would have little if any effect on the final evaluations. The evaluations were the combined effect of many details, so that no single detail, even if in error, would be likely to have a substantial impact.

4. Summary of Revised Evaluations

The Academic CIO Report has not been revised. As was pointed out in the Preliminary Report, the Academic CIO Report was a brief, independent, and informal report done as a "reality check" on the Gartner Report, in the sense that wide differences between these two reports would have prompted a deeper investigation and would have raised serious questions about the validity of the Gartner Report. In fact, there were not any unusual or wide differences: the two reports tracked fairly closely. Any differences that occurred can be attributed to differences of methodology. What is important is that, in spite of those differences of methodology, the conclusions were surprisingly well-correlated.

The Academic CIO Evaluation Team did precisely what was asked of them for the Preliminary Report. We are extremely appreciative of the thought and effort that they put into their evaluation. Since the Gartner Evaluation has not changed substantially (see below) between the Preliminary Report and this Final Report, however, there is no need to ask the Academic Team to do further analysis. Their report served the purpose for which it was intended, and they did what they were asked to do. Their original report, therefore, will not be used as a basis for the final staff recommendation contained in this Report.

Based on comments received and further analysis, the Gartner and Usage and Evaluation Teams have modified their original evaluations. The ICANN General Counsel has also modified his evaluation. The following updated reports have been prepared to reflect these three analyses, as refined:

Gartner Technical Evaluation Team

• .org Reassignment – RFP Review (19 September 2002)
• Response to Comments (17 September 2002)
• Advisory Letter (17 September 2002)

NCDNHC Usage Evaluation Team

• Final Report (9 September 2002)
• Supplemental Report (12 September 2002)
• Final Scoring Spreadsheet
• Adjusted Public Support Spreadsheet

ICANN General Counsel

• General Counsel's Report (22 September 2002)

This Final Staff Evaluation Report also reflects other comments that have been received that are outside the scope of these evaluations. In supplementary material, the Gartner, Usage, and General Counsel Reports indicated how comments received have affected their final reports. In Appendix 1: Analysis of Additional Comments, we refer to how we have addressed comments received that refer to the Preliminary Report itself, but which have not otherwise been addressed in this Final Report or by the Evaluation Teams.

The Gartner and Usage Evaluations rank the proposals into tiers. The Gartner evaluation ranks the proposals into three tiers. The Usage Evaluation combines its methodologies into two different ranking systems, what it terms its Average Ranking (four tiers) and Normalized Ranking (three tiers). For ease of reference, we name these tiers A, B, C, and, in the case of the Usage Evaluation Normalized Ranking, D. Note that these tier assignments are not directly comparable but have their own respective definitions.

The revised Tiers assigned by the Gartner and Usage Reports are tabulated below:

See also Section 5 ("Leading Proposals") for comment on the negative effect on GNR's Usage Ranking of General Counsel's evaluation regarding the GNR's suitability to receive the VeriSign endowment.

These tier assignments result from various approaches to combining "sub-rankings" into a single result. Each team has reflected the words contained in the RFP Criteria themselves to determine these sub-rankings, and proposes an approach to combining these sub-rankings. Such approaches are not perfect and are subject to the different formulae used to arrive at them. Because of this, both teams also point out that minute numerical differences in the final results are themselves not meaningful – the rankings are determined from clustering the final results into tiers such that numerical differences between clusters are more significant than numerical differences within clusters. A proposal that is ultimately ranked as an "A" proposal by a given team is, in effect, being denoted as a proposal to which much more serious consideration should be given than one that is ranked as a "B" or a "C" according to the criteria for selection.

The Usage Team gives equal weight to each of its three assigned Criteria that leads to a given sub-ranking, although it presents two ways of combining the sub-rankings. The Gartner team gives considerably more weight to Criteria 1 and 9; together they comprise 70% of the result, justified by the primacy of consideration that the RFP gives to Criterion 1 in particular and stability overall. This is because these Criteria most reflect the first priority placed by the ICANN Board as reflected in the RFP, namely to preserve stability during both the transition and during continued operation. That is, the strong weight can be justified by the words of the RFP itself.

Since no such ranking system can be considered perfect, it is important to look beyond the final rankings at the evaluations of the individual criteria that underscore the sub-rankings, at least to understand how these affect any final evaluation. Thus, for example, a given proposal could rank high overall yet fail on some critical point or sub-ranking, such as its "demonstrated ability to operate a TLD of significant scale" (see Criterion 1). For this reason, we provide a closer look at some of the leading proposals as determined by the two evaluation teams. We also consider in those closer looks any effect that the General Counsel's evaluation might have on the final recommendation.

We note at this stage, however, that purely based on the rankings alone – while given all the above caveats – the ISOC proposal is still ranked highest by both evaluation teams, as it was in the Preliminary Report. Although in the revised Usage Evaluation it now receives a "B" in one of the two rankings provided (the Average Ranking), this is as a result of (1) clustering causing different boundaries and only one proposal (Unity) receiving an "A" with this method, and (2) there now being four tiers with this ranking method. The ISOC proposal is still the only proposal that receives an "A" ranking from both evaluation teams.

It is staff's view that, given the Board's express instructions that demonstrated ability in operating a TLD registry of significant scale and continuous stability of the .org registry must receive primacy of consideration, only those proposals that are ranked "A" by Gartner and that receive high scores under Criteria 1 (Need to preserve a stable, well-functioning registry) and 9 (Preserving a smooth transition) by Gartner should receive further consideration. These are ISOC, NeuStar, GNR, DotOrg and Register.org. Indeed, Gartner recommends in its covering letter that "ICANN select the next operator of the .org TLD from among" these "five candidates". We term these the "Leading Proposals". Although other proposals certainly might be able to fulfill the required objectives, there is higher risk of failure if one were to be selected by ICANN.

It is staff's view that one of these five Leading Proposals should be selected. That is why, although it is ranked first by the Usage Evaluation Team, the Unity proposal cannot be the recommended choice in this Final Staff Evaluation Report. This is not to discount this excellent proposal – that is the clear favorite of the Usage Evaluation team – but because it is simply outranked by the named proposals in the technical criteria that count the most. The Gartner evaluation suggests that a Board decision in favor of the Unity proposal would present greater risk than one of the other proposals. It would thus be harder to defend, although in practice it might nevertheless work well. Given the presence of several proposals that present less operational risk, it would not be appropriate, in staff's view, to choose the Unity proposal.

Choosing among the five proposals recommended by Gartner presents a more nuanced challenge. A key determinant of choice among these five is, of course, how they rank in the Usage Evaluation report. Among those five proposals, as noted the ISOC proposal ranks clearly the highest in the Usage Evaluation, and is the only one to receive an "A" and a "B" in the Usage Evaluation. The GNR proposal receives two "B"s.; the NeuStar proposal receives a "B" and a "C"; the DotOrg and Register proposals both receive two "C"s.

However, before reaching a final staff recommendation for the ISOC proposal, we comment further on these five proposals in the next section. For completeness, we also comment on the Unity and UIA proposals.

5. Leading Proposals

In this Section, we provide comments on the "leading proposals" (as defined in the previous Section). These are to supplement the Gartner, Usage, and General Counsel evaluations, on which we principally rely. The comments are intended to highlight some of the key differences that affect the staff final recommendation that follows. It also provides an opportunity to respond to some of the comments that have been raised following posting of the Preliminary Report.

Discussion of "Demonstrated Ability"

Criterion 1 of the RFP focuses on the "need to preserve a stable, well-functioning .org registry" as ICANN's "first priority". We take that to imply that Criterion 1 must be satisfactorily addressed for a proposal to be considered further. Among the provisions of Criterion 1 is the statement that "ICANN will place significant emphasis on the demonstrated ability of the applicant or a member of the proposing team to operate a TLD registry of significant scale in a manner that provides affordable services with a high degree of service responsiveness and reliability"⁴ (italics added). Coupled with Criterion 9 that focuses on the need to ensure a smooth transition, Criterion 1 assumes primacy of consideration, and the requirement for demonstrated ability as described assumes significant emphasis within that consideration. This is underscored by the words of the RFP itself.

We take as a given that "demonstrated ability" means that for a proposal to be given greater weight, the bidder (or member of the proposing team) must have actually operated a TLD of significant scale for some period of time, not just shown the potential for doing so or have contracted the function out to a third party that is not a member of the proposing team (and who, therefore, would have no role in operating the .org registry). The words "significant scale" also imply that experience at operating a larger operation should be given more weight than operating a smaller operation. Since the purpose is to demonstrate the ability to operate the .org registry, the closer the scale approaches that of the .org registry, the more relevant the experience, all other things being equal. Furthermore, the type of operation should be reflective of the complexities of operating a large, thin registry or something closely comparable.

Some bidders commented that they believed that it was incorrect to give added weight to experience (demonstrated ability) or significant scale: if so, they should have raised these objections when the draft RFP was posted for public comment. The RFP is quite clear on this point.

There are many parts to operating a large, thin or thick registry. For convenience, we split these parts into two: those operations, or SRS services, that interact real-time with the registrars' systems; and those operations that provide real-time services to the general public (including registrars), including nameserver and Whois services. Much, but not all, of the complexity that revolves around business rules resides in the SRS services and databases. The business considerations that are incorporated into these applications raise the level of complexity to a very high level. Weight, therefore, must be given (in choosing among the five leading proposals) to direct experience in operating SRS systems of significant scale.

We note, however, that registrars operate the "front end" of SRS services that in turn interact real-time with registrants, and also operate Whois and other realtime services in a thin registry model. These registrar operations should therefore be considered as demonstrating ability, and, when they are sufficiently large in terms of number of domain names housed, as being of "significant scale". However, the registrar operations do not typically include gTLD Nameserver services. Reliable operations of Nameserver services ultimately is essential to stable Internet services. Important as this is, however, these were not singled out in Criterion 1 for separate consideration.

Where does all of this lead? It takes us to the conclusion that Criterion 1 implies the following:

• Greater weight should be given to extended experience in operating a TLD of a size commensurate with .org (that is, roughly the same order of magnitude) than one that is an order of magnitude or more smaller.
• Greater weight should be given to extended experience in operating a TLD that includes SRS services (whether on the registry or registrar side of these services) than one where there has been little or no direct experience in operating SRS service by the bidder or a member of the proposing team. This statement does not penalize outsourcing, provided the outsourcing is to an entity that constitutes part of the proposed environment (else the "ability" has not been "demonstrated").
• Greater weight should be given to operating a registry over operating a registrar operation, since this more closely aligns with what is required.

These assertions are not new criteria. They are direct implications of the words of the RFP itself. To set the stage for a straightforward comparison of demonstrated experience, the following supplemental questions were posed to the applicants:

• Supplemental Question #1 – Registry and registrar experience relied upon
• Supplemental Question #10 – Timeline of experience (useful in determining length of experience)

The responses to these supplemental questions by the Leading Proposals, as well as the Gartner Second Tier UIA and Unity proposals, are summarized in the table below:

Notes:

*. Duration measured until 18 June 2002 (the application deadline).
**. Experience with EPP/RRP was only 1 Month.
***. This is VGRS's domain count as of 31 March 2002.
****. After the application date, Unity's partner AusRegistry brought the .au registry online with approximately 300,000 names. There was no real-time SRS experience with .au as of the time of its application. Unity's other partner, Poptel, had 6,300 names as of the application deadline, but they did not run in a shared registry environment.

These data are factored into the considerations below.

Turning to considerations regarding the five leading proposals:

ISOC

The ISOC proposal is ranked in the top tier (out of three tiers) in the Gartner evaluation. It is ranked in the top tier (out of three tiers) by the Usage Evaluation Team using its Normalized Ranking methodology. It is at the top of the second tier (out of four tiers) in the Usage Evaluation Team's Average Ranking methodology (only Unity ranks in the top tier). It is the only one of the proposals to rank that high overall. These rankings are scored by evaluation teams that have no reason to prefer any one proposal over another; the Gartner team in fact had no connection with ICANN prior to this assignment. This consistently highest ranking of ISOC was the basis for the staff's preliminary recommendation of the ISOC proposal in the Preliminary Report. That highest overall ranking has not changed with the corrected evaluations.

As is understandable, since the ISOC proposal was highest ranked in the preliminary evaluation, it became the primary target by other bidders in their comments posted since the staff Preliminary Report was posted. As such, in this Report we devote some attention to the most often repeated of these criticisms and comments by other bidders.

Most of these comments have been addressed as appropriate in the Gartner and Usage Evaluation Team revised reports. There are, however, several outstanding issues that have been raised on which it is important to comment.

• Database and other Technical Considerations: Most bidders use Oracle, a widely used commercial database, at the heart of their proposed operations. ISOC's partner, Afilias, relies (as does one other bidder, IMS/ISC) on PostgresSQL, probably far and away the leading open source database, quite widely used by many institutions and for which several entities provide third-party maintenance support (see ISOC response to posted Supplemental Question # 13). Afilias has operated its system (that it inherited from Tucows) in support of the .info registry for over a year (see ISOC response to posted Supplemental Question No. 10), albeit with some initial service problems in its first days of operation. It could be argued that a bidder should not be penalized for whatever database system – or for that matter any other significant component of operation – it chooses (open source or commercial) to use provided it has demonstrated its ability to operate a TLD of significant scale over an extended period of time. That is, Criterion 1 places greatest emphasis on the notion of "demonstrated ability" as described above. The ISOC team meets or exceeds this requirement.
  
  Gartner has, however, in its advisory letter, and its response to the ISOC comments on the Preliminary Report and original Gartner Evaluation, recommended the use of PostgreSQL be carefully reviewed – beyond what was possible in the Gartner Evaluation – with regard to potential risks (noted by Gartner) associated with PostgreSQL. Gartner notes, however, that these are manageable risks – risks "…that can be managed through frequent monitoring of database performance and mitigated – if required – through the deployment of alternative technology." In other words, should PostgreSQL in the future prove not to be viable, ISOC/Afilias could – at some expense – migrate to another database environment, something Afilias would most probably need to do for .info in any event. Furthermore, this migration could occur over time. Further, Gartner has already factored these concerns in its evaluation that resulted in the tiering of the proposals, but singles the concerns out for attention as an important issue for consideration beyond the rankings.
  
  Nevertheless, this is a point that the Board should consider in making its final decision.
  
  Another advisory point raised by Gartner is that ISOC did not provide any SRS fail-over times in its proposal in the event of system failure at the primary site. ISOC, however, was not alone in this among the bidders, nor were they alone in using adjectives like "instant", "brief", or "swift" to replace actual times. The question always arises, of course, as to whether such actual times materialize in practice under a plethora of potential disaster conditions (ISOC points out that in any event it takes time to alert registrars to point to backup sites). Gartner also points out that the ISOC proposal includes the statement that "the secondary data center may perform at a degraded level" because it does not replicate exactly the primary site capabilities. But that particular engineering choice has been factored into Gartner's evaluations and rankings.
  
  
• Demonstrated Ability: The ISOC partner, Afilias, has, as indicated above, operated a TLD of significant scale for an extended period of time. There are now nearly one million registrations in the .info database, a scale commensurate (as defined above) with the approximately three million .org registrations. Furthermore, this experience is with the same operating environments as is being proposed for .org. The fact that much of this operation is outsourced (to IBM for computer and network operations, and to UltraDNS for nameserver services) should not be a reason to penalize the ISOC proposal as some commentators have suggested, since Afilias has a proven track record operating .info in this manner. To the contrary, credit could be given for careful selection of operating partners and for actual operating experience with those partners.
  
  
• Financial Concerns: Many have commented on the financial condition of ISOC as a reason why ICANN should not award .org to ISOC. There are several points to be made:
  • No financial evaluations were made of any of the bidders. Demonstrated experience over extended periods was considered to be of greater significance since financial conditions can always change, positively or negatively.
  • Although throughout this report the subject proposal has been referenced as the "ISOC Proposal", the agreement will actually be with a new entity called "PIR" should the award be made to this bidder. PIR will be financed with start-up capital by Afilias (that as such would be contractually obligated should the ISOC proposal be selected). There will be no flow of funds either way between ISOC and PIR. Any surpluses generated by PIR from operating .org would be used as indicated in the ISOC proposal and as has been evaluated by the Usage Evaluation Team; these surpluses would not be used to subsidize ISOC, a requirement that would be clearly set forth in any agreement between ICANN and PIR should ISOC/PIR be selected. ISOC's role with respect to PIR would be limited to appointing the PIR Board of Directors (whose members would not overlap the ISOC Board of Directors, and at least a majority of which would differ from ISOC's directors). It is feasible to insulate PIR from any possible ISOC financial difficulties by appropriate contractual provisions.
  • Thus, although it is entirely reasonable to raise the concern, ISOC's financial condition, one way or the other, is not a factor to be considered, just as no consideration was given to the financial status of any other bidder.
    
    
• Conflict of Interest: Some detractors have suggested that the ICANN Board would have a conflict of interest in accepting the ISOC proposal, since many Directors are also members of ISOC. The Chair of the ICANN Board is a former director and chair of the board of ISOC; and the Vice-Chair of the ICANN Board is President of ISOC Mexico.
  
  The General Counsel's revised evaluation indicates that there is in fact no conflict of interest since no financial advantage whatsoever accrues to any member of the ICANN Board were the Board to approve this selection. There is no possibility of financial gain, a prerequisite for there to be any conflict of interest. Furthermore, the Chair of the ICANN Board is no longer a director of ISOC. There is also no legal or financial relationship between ISOC and ISOC Mexico other than ISOC Mexico is permitted to use the name "ISOC" in exchange for agreeing to abide by certain bylaws, and that ISOC may or may not provide speakers or similar services for certain ISOC Mexico meetings.
  
  In short, there is no conflict of interest because there is no financial interest, and thus there would be no basis in ICANN policy to deny the award to ISOC on this basis. For the same reason, there is no reason why any ICANN Director should recuse his or herself from a vote to approve ISOC, simply because he or she a member of ISOC. As noted in the General Counsel's report, ICANN's conflicts-of-interest policy does not prohibit Directors from voting on matters merely because those matters involve a philanthropic society which the Directors may philosophically support. Conflicts-of-interest principles are intended to prevent improper personal financial gain, not to negate the philosophies of ICANN's Directors. Directors are not expected to discard their philosophies and past experiences when they undertake their service at ICANN, but instead are expected to incorporate those philosophies and experiences in formulating well-reasoned decisions.

In summary, staff do not believe that any of the foregoing present any reason not to award the bid to ISOC should the ICANN Board wish to do so.

NeuStar

The NeuStar proposal was ranked the strongest overall by the Gartner team in its highest tier. Furthermore it scored highest in 4 of the 5 criteria considered by Gartner, including the important Criterion 1 and Criterion 9. Gartner had no management advisories regarding NeuStar in Gartner's letter of transmittal. If the technical evaluation were the sole determinant, staff would recommend NeuStar to the ICANN Board to be the successful bidder.

However, this would not comply with the RFP. The RFP required that first priority (but not the only priority) be given to Criterion 1. All proposals that ranked in the highest tier by Gartner are considered as eligible for further consideration for purposes of this Section and capable of operating the .org registry, that is, they are Leading Proposals. The question is: which is most suitable based on factors exogeneous to the technical evaluation? Gartner emphasizes that small numerical differences among the top-ranked proposals should not be determinant.

Although the NeuStar proposal is evidently extremely strong technically, and although NeuStar now has a long history (demonstrated ability) of operating .biz, a gTLD containing about 735,000 domain names (that is, of significant scale) and a shorter history of operating .us that contains about 300,000 domain names, it is ranked significantly lower than ISOC by the Usage Evaluation Team. NeuStar is ranked as a "C" and a low "B" (in the two Usage Evaluation ranking methods), compared with a "B" and an "A", respectively. Reasons for these differences are detailed in the Usage Evaluation Team report.

Both ISOC (Afilias) and NeuStar clearly have demonstrated ability in running a registry of significant scale, as well as meeting other requirements of Criteria 1, 7, 8,and 9. The ISOC proposal, however, as indicated by the Usage Team Evaluation, ranks considerably higher with respect to Criteria 4, 5, and 6.

Nevertheless, the NeuStar proposal is a very strong proposal that must be given serious consideration should the ICANN Board choose not to accept the ISOC proposal. This is primarily because of the evident technical strength of the NeuStar proposal.

GNR

The GNR proposal is given further consideration in this Section because it also ranked in the top tier in the Gartner Evaluation, albeit lower than NeuStar and ISOC. It also ranked higher than Neustar in the Usage Evaluation (a "B" and a "B", compared with a "C" and a "B"), although, again, lower than ISOC ("B", "A"). There are, however, two additional significant concerns regarding GNR, both of which relate to its "demonstrated ability to operate a registry of significant scale" under Criterion 1, and both of which have been raised in the Gartner transmittal letter:

• Demonstrated Experience: GNR has operated the public services for the .name registry since January 2002. At the time of its proposal, however, GNR had no experience with the significantly more challenging real-time SRS services; those services were not launched until shortly after GNR submitted its .org proposals. This length of experience is a significantly less impressive demonstration of ability to operate a large TLD registry comparable to .org than the experience of NeuStar or ISOC/Afilias. GNR made a strong point in its proposal that it has achieved 100% uptime during this period of operation, obviously a creditable accomplishment, but this accomplishment related only to the public (nameserver and Whois) services, since real-time SRS services had not been implemented at the time of the proposal.
  
  Moreover, GNR's post-proposal SRS experience is with a different operating environment than what is proposed in the GNR bid. As indicated in the GNR response to Supplemental Question No. 6, during this period the real-time SRS operations (as indicated earlier, for purposes of this discussion we simplify the precise line-up of services) have been outsourced to VeriSign who have arguably, therefore, played a key role in the stability accomplishments.
  
  Using its own expertise (rather than VeriSign's), GNR has operated the NameServer, Whois, and other related real-time services, but not the real-time SRS services with which registrars interact. As indicated in the introduction to this Section, direct experience in operating real-time SRS services is given greater weight than not. As such, for example, GNR's experience must be given significantly less weight than that of ISOC/Afilias or of NeuStar.
  
  
• Scale: The .names registry operated by GNR contains about 83,000 domain names. In addition, the registry handles mail forwarding for about another 61,000 names. Collectively, however, this cannot be considered as being of "significant" scale (in the sense described in the introduction to this Section), that is, this size is not of the same order of magnitude as .org, and of the registries operated by ISOC/Afilias and NeuStar.

Were the GNR proposal the only proposal ranked in the top tier by Gartner, these concerns could be set aside if necessary. GNR has obviously – and to its credit – demonstrated its ability to manage an important gTLD registry. But it is not the only proposal so ranked. There are other proposals more clearly indicating demonstrated ability to operate a registry of significant scale than the GNR proposal. And the GNR proposal is ranked significantly lower than the ISOC proposal by the Usage Evaluation Team.

An additional problem with the GNR proposal is that the General Counsel ranks it as a "C" regarding its likelihood to be eligible for the VeriSign $5 million endowment. If, as seems quite likely, GNR were not eligible for this endowment, the ranking of GNR in the Usage Evaluation Report would likely have been diminished, since this ranking is in part predicated on GNR receiving this endowment. Other Leading Proposals either are ranked as an "A" in this regard in the General Counsel's report (that is, there is a very high probability that the sponsoring organization would be eligible) or the proposal has no dependency on the endowment.

Gartner also raised certain other advisories related to GNR in its transmittal letter, including a concern about the capacity of backup data centers.

DotOrg and RegistryOrg

We consider these two proposals together. They are essentially identical proposals from a technical standpoint insofar as they both employ the same Register.com/Registry Advantage "back-end", The difference between them is that the Dot Org proposal uses the DotOrg Foundation (not to be confused with the .Org Foundation) as a "front-end", presumably because of the influence this might have on Criteria 4, 5, and 6. However, both proposals are ranked approximately equally by Gartner ("A"), and by the Usage Evaluation Team ("C", "C"). The latter rankings are significantly lower than those of ISOC or GNR, and are lower than those of NeuStar.

Register.com has considerable demonstrated experience in operating as a SRS registrar to the .org registry, as well as to many other registries. Its registrar operations are indeed of a significant scale, housing over 3.4 million domain names. Through its Registry Advantage operation, Register.com also has relevant experience in operating registries, albeit on a very small scale (a total of 15,000 domain names). Nonetheless, its registrar operations should be considered to constitute significant experience, which counts toward demonstrated ability.

However, the lower rankings by the Usage Evaluation Team mitigate against these two proposals compared with the other leading proposals. They are the lowest of all five proposals.

Gartner does raise certain other concerns in its letter of transmittal that would have to be considered by the Board were it to consider these proposals further.

The remaining proposals mentioned in this Section are included not for consideration – since they are not in the highest ranking of the Gartner Evaluation – but to make comments for completeness to explain what might seem to be (but are not) discrepancies.

Unity

The Unity proposal is a strong proposal that received clearly the highest ranking by the Usage Evaluation Team ("A", "A"). It is a tribute to the proposers that they apparently recognized the implications of Criteria 4, 5, and 6, and, according to the Usage Evaluation Team, were indisputably the most responsive to those criteria, whatever method is used to combine the separate rankings with regard to the individual criteria.

Nevertheless, ICANN staff excluded this proposal from further consideration because it is not ranked in the top tier of the Gartner Evaluation which, for the reasons described, is a prerequisite for further consideration. The Unity proposal was ranked as a "B" by Gartner and as such is not included in the five candidates recommended by Gartner. However one considers fine numerical differences between the "scores" of different bidders, there is a demonstrable gap between, say, the two bidders ranked at the top of the Gartner ranking and the seventh-ranked, and this gap reflects serious differences in the quality of the proposals from a technical perspective.

This does not mean that Unity demonstrably could not operate the .org registry satisfactorily, but it does mean that five other proposals demonstrated that they could operate the registry reliably and stably with greater conviction and hence with lower risk of failure.

Regardless of this concern, one could argue that Unity should be included, were it true that every proposal that ranked in the top Gartner tier made a poor showing in the Usage Team Evaluation. But that is simply not the case, as described in the Usage Team Evaluation and above.

The scale of operations experience demonstrated by Unity's partners also does not compare with that of ISOC/Afilias, NeuStar, DotOrg or Register.com, or even with that of GNR.

As such, Unity is not considered further in this Report.

UIA

The UIA proposal employs VeriSign as its registry operations provider, at least for the first three years of operation. As such, as detailed in the General Counsel's report, it is the only proposal that ranks low on Criterion 3: Enhancement of Competition for Registration Services. Since this is the overarching goal for the entire undertaking of re-assignment of the .org registry, we believe that UIA/VeriSign should not be favorably considered, unless there was no other satisfactory proposal of sufficient merit – which is clearly not the case.

The UIA proposal did not rank in the top tier of the Gartner evaluation, thereby also removing it from further consideration in this Report for the stated reasons. And it received lower rankings than other proposals in the Usage Team evaluations ("C", "C"). Therefore, there is no compelling reason for considering this proposal further.

It might seem surprising on the surface that the incumbent operator of .org – that has successfully operated the registry for almost a decade – should not rank high in the technical evaluation. In particular, the UIA proposal was ranked 3rd in Criterion 1 and tied for 5th in Criterion 9. On the face of it, it would seem that it has demonstrated the ability to operate a TLD of significant scale (indeed the "most" significant scale). And it would seem that it has no transition to undergo, but can just continue business as usual.

The reasons for its lower overall rankings in the Gartner Evaluation are:

• Criteria 1 and 9 are only two of the four criteria used in arriving at the overall tiered classification. There were other considerations even though these two criteria constituted 70% of the weights used and UIA did not score as high as other proposals on these criteria. Within Criteria 1 and 9, there were also a range of considerations, on some of which UIA did not score as high as other proposals.
• Little information, in support of Criterion 9, is provided in the UIA proposal regarding the transition. Presumably the bidder relied on the assumption that there is no transition for the incumbent (VeriSign). However, there is a transition. There is a transition of overall responsibility and policy management from VeriSign to UIA and, compared with other bidders, little or no information was provided on this, raising the question as to how much control UIA would really have over the proposed .org arrangement.
• With regard to Criterion 1, the Gartner team evaluated the technology environment being proposed by two other bidders as being of higher quality than that now provided by VeriSign. Nevertheless, UIA ranked among the top three in this Criterion and small numerical differences are probably not significant.

In addition, the UIA proposal ranked lower than many other proposals in the Usage Team Evaluation. For all these reasons, it is not considered further.

6. Recommendation

Considering the reports of the Gartner and Usage Evaluation teams, considering the supplemental information kindly provided by all applicants, and considering the comments of the previous section, the staff recommends to the Board that it award the .org registry to the ISOC proposal and instruct the ICANN President to finalize an agreement within ten days of the Board's decision.

The ISOC proposal clearly receives the highest ranking overall of those teams that are ranked in the top tier in the Gartner evaluation, that is, taking both the Gartner and the Usage Team evaluations into account. We have addressed in the previous Section the main concerns and comments that have been expressed about the ISOC proposal since we issued our Preliminary Report and Recommendation; and other comments made have been factored into the revised Gartner and Usage Evaluation Team reports and rankings, without significantly changing the latter.

We note that most of the comments received that point out errors in the original evaluations and the Preliminary Report have little effect on the final results, even when corrected. This is because the evaluations are compendiums of many factors, and adjusting a few factors does not generally have a substantive effect even if given prominent voice by the commentators (sometimes in the most strident of language, which is understandable given that there may be much at stake!).

Should the Board for some reason decide not to accept the ISOC recommendation based on a different assessment of the relative merit of the various proposals according to the stated criteria and the showings documented in the submitted proposals, or should ICANN be unable to consummate a timely agreement with ISOC, then staff would recommend that the Board authorize the ICANN President to negotiate an agreement with NeuStar or, in the event an agreement with NeuStar cannot be consummated, GNR. NeuStar has considerably more demonstrated ability than GNR in operating a TLD of significant scale and scores higher on Criteria 1, 7, 8, and 9; GNR (see comments in previous Section) does not, in the opinion of staff, meet the test in this regard. On the other hand, GNR ranks higher than NeuStar in the Usage Team evaluations. If the Board gives first priority to stability (as indicated in the priority accorded to Criterion 1), then NeuStar should rank ahead of GNR.

7. Acknowledgments and Closing Observations

The staff would like to thank all of the applicants for the extraordinary efforts in submitting eleven thoughtful and excellent proposals; in making valuable comments on the Preliminary Report and associated evaluations; for providing answers to the supplementary questions in such a timely manner; and for their patience, good heartedness, and responsiveness as we have worked through this difficult process.

The staff would like to thank all of the many individuals on the evaluation teams (Gartner, Academic CIO, Usage) for all of their significant contributions and incredible hard work in a short space of time. Most of these individuals contributed their time voluntarily (and we thank their institutions, where applicable, for making this possible), prompted only by what was good for the Internet – they deserve a special accolade of recognition. And Gartner Inc. deserves special thanks for working far beyond the terms of their engagement in fulfilling their professional obligation to "get it right". This, after all, is not a normal procurement: the special nature of a global and totally open ICANN places unusual burdens.

There can only be one successful bidder, and that means that there will be ten disappointed applicants each of whom will surely see flaws in the process, and many of whom may continue to voice their concerns. Regardless of these perceptions and statements, both the staff and the evaluators have made every attempt to be open, transparent, and, above all, fair. No one involved has had any reason, personal or institutional, to favor one proposal over another, and there is nothing to be gained in any event from playing favorites. The overall concern is what is best for the Internet community, using a fair – albeit inevitably not perfect – bidding, evaluation, and selection process. And even to the extent there are inevitable imperfections, these are subjugated by the whole weight of the evaluation and recommendation process that involved many individuals from many different parts of the world.

There are those who will believe that only large registries could be competitive in this bidding process. Given the primacy of transferring a complex and large registry, .org, in a demonstrably stable manner, there must be some truth to this. This was well articulated in the RFP itself. In fact, staff was surprised by the number of proposals received, a number that exceeded initial expectations (hence the subsequent refund) given the focus of the RFP on "demonstrated ability to operate a TLD of significant scale".

This does not mean that all future ICANN gTLD awards will contain the same requirement. After all, none of those featured in this Report for further consideration started "large": ISOC's partner Afilias, NeuStar, and GNR all started from scratch last year. But in this case, the Board is considering transfer of a large, functioning registry, .org, containing well over 2 million registrants, and cannot responsibly risk jeopardizing stability through venturing too far into the unknown and the unknowable. That is the compelling fact for this reassignment. But there will likely be other opportunities for those who did not succeed in this particular award.

Appendix 1: Analysis of additional comments

In this Appendix, we address comments that were generally addressed to the Preliminary Staff Report and not directly to any of the evaluation teams or the any of the evaluation reports. Those comments have been addressed in the responses by the evaluation teams. Some of the comments have been addressed in the text. All other comments we believe are addressed in either the Technical or Usage Evaluation Teams' revised reports or in the General Counsel's revised report.

Comments by Bidders:

DotOrg Foundation:

(No comments directed to the Preliminary Staff Report, although comments were directed to the individual evaluation team reports (particularly the Usage Evaluation Report). As note above, necessary adjustments have been made to the team evaluations, as explained in their reports. See also comments made by RegisterOrg, a "sibling" proposal).

RegisterOrg:

• Comment: Both the Gartner Report and the Academic CIO Report should consider the financial stability of the bidders.
  
  Response: Gartner considered evidence of commitment of resources as part of its evaluation of Criterion 1. In the Criteria and other materials for the RFP for the .org reassignment, proven track record (demonstrated ability of operating TLD of significant scale), and commitment of resources to assume responsibility for the obligations of the .org operator, were considered to be superior measures of overall stability than a specific assessment of the financial stability of applicants. The criteria stated, and the information sought to evaluate proposals under those criteria, is the basis of this evaluation. Financial information other than evidence of commitment of resources was not required as part of the RFP responses and was not specified as part of Criterion 1 or any other criterion. Moreover, it is extremely hard to assess reliably a particular applicant's financial stability directly. The spectacular failures of the past few years of large enterprises indicate the difficulties in predicting the financial future of any institution. In addition, RegisterOrg scored well in the Gartner Report on Criterion 1.
  
  Allegations concerning the financial condition of other bidders are not considered as evidence of strength of RegisterOrg's proposal.
  
  
• Comment: The Academic CIO Team Evaluation is highly subjective…..and therefore must be disregarded.
  
  Response: This Final Staff Evaluation Report does not rely on the CIO Team Evaluation to ascertain the rank of any particular application. See Section 4 ("Summary of Revised Evaluations").
  
  
• Comment: The Noncommercial Domain Name Holders Constituency should reconsider its ranking of RegisterOrg.
  
  Response: The Usage (NCDNHC) Evaluation Team has carefully considered the RegisterOrg comments and has modified its report to address the comments.

GNR:

• Comment: Financial status of bidders not adequately considered (this comment is made in several places). Makes allegations regarding comparative financial condition of Neustar and ISOC.
  
  Response: Gartner considered evidence of commitment of resources as part of its evaluation of Criterion 1. In the Criteria and other materials for the RFP for the .org reassignment, proven track record (demonstrated ability of operating TLD of significant scale), and commitment of resources to assume responsibility for the obligations of the .org operator, were considered to be superior measures of overall stability than a specific assessment of the financial stability of applicants. The criteria stated, and the information sought to evaluate proposals under those criteria, is the basis of this evaluation. Financial information other than evidence of commitment of resources was not required as part of the RFP responses and was not specified as part of Criterion 1 or any other criterion. Moreover, it is extremely hard to reliably assess a particular applicant's financial stability directly. The spectacular failures of the past few years of large enterprises indicate the difficulties in predicting the financial future of any institution. In addition, RegisterOrg scored well in the Gartner Report on Criterion 1.
  
  Allegations concerning the financial condition of other bidders are not considered as evidence of strength of GNR's proposal.
  
  
• Comment: It is less pro-competition to assign a third TLD to NeuStar (which operates .biz and .us) and ISOC/Afilias (which operates .info and .vc), than a second gTLD to European-based GNR.
  
  Response: There was nothing in the RFP that suggested that any current operator of two TLDs should be "marked down" over one that only operates one TLD. (Moreover, it should be noted that the .vc TLD operated by Afilias is quite small and just started up in July 2002.) Extending GNR's argument would suggest that it would be even more pro-competition to allocate the .org registry to an institution that does not currently operate any large TLDs. In any event, there are many arguments (some of them raised by other applicants) that could be made that suggest that a larger institution could more readily compete in the current marketplace than a smaller institution.
  
  The RFP gave no credit to consideration of any geographic region over another. This was not a consideration in any of the evaluations or in the Preliminary Report. Therefore, no advantage should be given to GNR's European headquarters. Besides, geographic affiliation is a very amorphous concept. ISOC, for example, while headquartered in the United States draws 65% of its members and 60% of its board of directors from outside of North America; and Afilias is an Irish corporation.

NeuStar:

• Comment: Report tainted by flawed nature of evaluation (introduction of new criteria, reintroduction of previously rejected criteria, misapplication of published criteria, severe internal inconsistencies, mathematical errors, biased interpretations etc.).
  
  Response: We appreciate the specific comments that NeuStar has made with respect to some or all of these categories.These specific comments have been appropriately addressed in the place where those comments occur. Corrections have been made where appropriate, and explanations given where not. Specific comments, whether correctly founded or not, are helpful to the process. General statements are not very useful.
  
  
• Comment: Preliminary Staff Evaluation Report ignores Board-established priorities of the criteria, particular noting the primacy of the requirement for demonstrated ability to operate a TLD of significant scale.
  
  Response: The essence of NeuStar's argument is that the applicant that scores the highest on Criterion 1 or on the technical evaluation overall should be awarded the bid, regardless of how well the applicant scores on Criteria 4, 5, and 6 as evaluated by the Usage Evaluation Team, or on Criteria 2, 3, or 10 reviewed by the General Counsel. This is clearly incorrect, otherwise the non-technical criteria would not have been included in the Statement of Criteria issued as part of the RFP. When the draft RFP was posted, NeuStar did not object to the inclusion of these criteria; and indeed they responded to the RFP knowing that these criteria were included.
  
  As Gartner points out, any of the five proposals ranked in the top tier of the technical evaluation should be considered as candidates for consideration. The staff agrees with Gartner that the Board can be comfortable with any of those five from a technical and stability perspective. The Usage Team evaluation of Criteria 4, 5, and 6 can and is used to further refine and select from among those five proposals. The General Counsel's review is also factored in.
  
  
• Comment: The staff in their report should summarize the strengths and weaknesses of each proposal, not make a specific recommendation.
  
  Response: We disagree. The strengths and weaknesses are summarized in the evaluation reports. The staff's recommendation merely synthesizes what is in those reports. The relative overall ranking of the ISOC proposal and the NeuStar proposal in those reports is quite clear, and that is what the staff is pointing out through its recommendation.

IMS/ISC:

• Comment: No technical due diligence performed. Past performance not documented. No examination of codes, logs, or configurations.
  
  Response: This was not within the scope of the RFP process. IMS/ISC did not comment to our knowledge that this should be part of the process when the draft RFP was posted for comment. IMS/ISC bid knowing that these factors were not part of the process. We cannot introduce into the evaluation requirements that were not part of the RFP.
  
  Within available resources and timeframe, it is not likely that such a requirement could be performed. With regard to past performance, information was sought and obtained both in the application process and by supplemental questions. The necessity for, or even real value of, examination of codes, logs, or configurations is not clear. Moreover, such a highly intrusive exercise was not contemplated by the RFP and would risk constraining the range of reasonable engineering judgments.

Organic:

• Comment: The ICANN Board (Organic includes specific quotations from Board members) indicated that no credit should be given to any proposal that plan to use surplus funds for "good works". Yet the Organic proposal was penalized precisely because it did not include any "good works."
  
  Response: As discussed in detail in the Usage Evaluation team's Supplemental Report, the issue of whether it was consistent with the RFP to include the "good works" score was considered at length by that team in response to the comments. Several members of the team initially felt the objections to its inclusion were well founded, but after extensive analysis and discussion the team reached the conclusion that good works are an appropriate element of the "supportiveness" called for by the RFP, provided the good works are targeted at the noncommercial Internet user community. The Usage Evaluation team notes that the "good works" score was lightly weighted in the analysis, but notes that it could be excluded from consideration if the Board felt that appropriate.
  
  Although the discussion of the Board on "good works" is subject to differing interpretations, the staff believes there is some merit to Organic's comment, particularly as to good works unrelated to the noncommercial Internet community. We have recalculated the NDNHC evaluation of Responsiveness and Governance (Table 2) without the "Good Works" column, and the consequent effect on the overall rankings, and found that it makes no significant difference (because "Good Works" only had a weight of .5 out of 6.25).
  
  
• Comment: Organic raises the concern that the ICANN Board would have a conflict of interest were it to select the ISOC proposal. Organic is also concerned with a potential conflict of interest for Harold Feld with the Usage Team Evaluation.
  
  Response: Staff disagrees with Organic that the ICANN Board has a conflict of interest based on some Directors' membership in ISOC. See the General Counsel's report and the comments under ISOC in Section 5 ("Leading Proposals") above. Harold Feld resigned as Chair of the Usage Evaluation Team once he discovered circumstances the might affect, or appear to affect his objectivity with respect to RegisterOrg. (In view of the relatively low ranking of RegisterOrg in the work that Mr. Feld had already completed, any claim that he was biased in its favor is not credible.)
  
  
• Comment: Organic asserts that the leading bidders (they cite .info as an example) have shown inability to comply with ICANN agreements in the past, and therefore cannot be counted to do so in the future.
  
  Response: Organic provides no specific data to support their assertion. ICANN is aware of both the ICANN agreements and the leading bidders' conduct under them, and knows of no grounds to support the assertion that the leading bidders have been shown to be unable to comply with those agreements.
  
  
• Comment: Organic asserts that other bidders have shown weaknesses that should be considered. They cite NeuStar's recent restructuring as an example.
  
  Response: Organic provides no evidence to support their assertion that restructuring or resizing should be considered evidence of weakness. Indeed, taking the necessary steps to ensure efficiency and continuity in the face of evolving challenges is an attribute of sound management.
  
  
• Comment: Organic asserts that the size of an organization should not be taken as a predictor of future stability, and that this superseded past experience.
  
  Response: Criterion 1 gave strong consideration to the demonstrated ability to operate a TLD of significant scale. It is not the size of the organization that is the operative consideration; it is the significant scale of the registry experience. The implications of this are presented in Section 5 ("Leading Proposals").
  
  
• Comment: Organic asserts that the award should not be given to any organization whose survival is predicated on winning the .org bid.
  
  Response: This was not a factor (one way or the other) that was included in Statement of Criteria or elsewhere in the RFP. We have no knowledge of which proposals fit or do not fit with Organic's concern (including, for that matter, Organic's own proposal).
  
  
• Comment: The U.S. Department of Commerce should certify the fairness of the bid and evaluation processes.
  
  Response: A certification by the U.S. Department of Commerce is not contemplated in the RFP. To our knowledge, Organic made no such suggestion during the RFP comment process, and submitted its proposal with no understanding that such would be part of the process.

SWITCH

• Comment: Asserts that that proposals that "looked the most impressive got the best rankings."
  
  Response: SWITCH provides no specific evidence to support that assertion. The evaluations were neutral and based on the information provided, not how they were displayed. SWITCH, however, did not make the task easier insofar as its proposal did not directly answer most of the questions posed, as did other bidders, but combined those answers into appendices. This led to some missed answers by Gartner in its original evaluation. Gartner has corrected its evaluation now that it has located required information distributed across the appendices.
  
  
• Comment: Asserts strong imbalance in evaluation teams. "It was an All-American play."
  
  Response: 6 out of the 8 members (6 out of 7 once Harold Feld resigned) of the Usage (NCDNHC) Evaluation Team were from outside of the U.S.. 2 out of 5 members of the Academic CIO Evaluation Team (whose report was part of the Preliminary Staff Report on which SWITCH was commenting) were from outside of the U.S.. Furthermore, 2 out of the top 4 ranked proposals by the Usage Team, including the top-ranked proposal, were from outside of the U.S.. The third-ranked Gartner proposal was from outside of the U.S.. This hardly correlates well with SWITCH's assertion.
  
  In a world, particularly an Internet world, where national boundaries and economic boundaries are often blurred, it is hard to draw hard and fast national or regional lines. In addition to what was noted in the previous paragraph, although ISOC is headquartered in the United States, 65% of its members and 60% of its board of directors are from outside of North America. Furthermore, Afilias is an Irish corporation, even though much of its operations are in the U.S. and Canada.
  
  We are sensitive to SWITCH's concern, but geographic location was not a factor in any of the criteria (that is, none of the criteria made any assertion as to whether the successful bidder should come from one part of the world or another), and we are confident that the evaluation teams focused on the facts and the material, not on exogenous factors.
  
  
• Comment: Financial stability seems to have been a non-issue.
  
  Response: Gartner considered evidence of commitment of resources as part of its evaluation of Criterion 1. In the Criteria and other materials for the RFP for the .org reassignment, proven track record (demonstrated ability of operating TLD of significant scale), and commitment of resources to assume responsibility for the obligations of the .org operator, were considered to be superior measures of overall stability than a specific assessment of the financial stability of applicants. The criteria stated, and the information sought to evaluate proposals under those criteria, is the basis of this evaluation. Financial information other than evidence of commitment of resources was not required as part of the RFP responses and was not specified as part of Criterion 1 or any other criterion. Moreover, it is extremely hard to reliably assess a particular applicant's financial stability directly. The spectacular failures of the past few years of large enterprises indicate the difficulties in predicting the financial future of any institution. In addition, RegisterOrg scored well in the Gartner Report on Criterion 1.
  
  Allegations concerning the financial condition of other bidders are not considered as evidence of strength of SWITCH's proposal.
  
  
• Comment: A non-commercial bidder could better suit the non-commercial community. This was treated as a non-issue.
  
  Response: The ICANN Board of Directors explicitly stated that the commercial or non-commercial status of the bidder was not to be considered one way or the other, and the RFP reflected that neutrality. SWITCH chose to submit a proposal, which we appreciate, knowing this to be the case. This factor was indeed not considered in the RFP and the evaluation. Moreover, many bidders that are non-commercial were scored highly by the teams and overall. This Final Staff Evaluation Report, for example, recommends ISOC/PIR, which is noncommercial.
  
  
• Comment: SWITCH asserts that their "bid was considered as stemming from an "outsider" with no real experience and therefore was treated as negligible (quatité négligeable)".
  
  Response: SWITCH provides no specific evidence to support that assertion. Their bid was as seriously considered as any other proposal, SWITCH was as well-known or not to the evaluation teams as any other bidder. The reasons for SWITCH's comparatively low rankings are documented in the Gartner and Usage Evaluation Team reports.
  
  
• Comment: Academic CIO Team did not provide any detail. Therefore unable to comment.
  
  Response: This Final Staff Evaluation Report does not rely on the CIO Team Evaluation to ascertain the rank of any particular application. See Section 4 ("Summary of Revised Evaluations").

Unity:

• Comment: The Academic CIO Report did not provide detail and should be disregarded.
  
  Response: This Final Staff Evaluation Report does not rely on the CIO Team Evaluation to ascertain the rank of any particular application. See Section 4 ("Summary of Revised Evaluations").

.Org Foundation:

• Comment: The Academic CIO Report did not provide detail and should be disregarded.
  
  Response: This Final Staff Evaluation Report does not rely on the CIO Team Evaluation to ascertain the rank of any particular application. See Section 4 ("Summary of Revised Evaluations").

UIA:

• Comment: To ensure fairness and transparency in the process, ICANN should have indicated up front to the bidders how the criteria would be weighted so that bidders could all have prepared bids with this knowledge in mind.
  
  Response: The weights being given to the criteria were derived from the words of the criteria themselves. The Usage Evaluation Team weighted them equally because there was nothing in the words to suggest otherwise. The Gartner team gave 70% of the weight to Criteria 1 and 9 because the words clearly stated that primacy of consideration had to be given to stability of operation and transition.
  
  No weights were indicated in the draft RFP that was posted, and neither UIA nor any of its partners commented that this was a deficiency. UIA submitted a proposal with full knowledge that no weights were assigned a priori.
  
  
• Comment: Full details of the bid process were not laid out up front by ICANN. Specifically, it was not known that endorsements received after the proposal deadline would be accepted in the Usage Evaluation Team proposal. The bidder that ranked top in this category had received no endorsements at the time of the proposal deadline.
  
  Response: Perhaps it would have been better had that it been made more explicit in the RFP that endorsements could be received after the deadline. However, the instructions posted as part of the final RFP materials included a schedule specifically providing for extensive public comment on the proposals once they were submitted. The self-evident purpose of this post-proposal public comment period was to allow members of the community to review the various proposals and express support (or opposition) to the various proposals. Thus, it was apparent from the RFP that endorsements from the public made after the application deadline would be considered. We also note that the proposal which UIA claims received no endorsements until after the deadline was IMS/ISC, which is not one of the Leading Proposals being considered.
  
  
• Comment: Regret that more attention was not given to confidentiality of the bid process and its disassociation from ICANN politics. Would have preferred that sealed bids be handled by an independent body rather than give rise to situations in which ICANN is seen to be so intimately involved with the lead candidate.
  
  Response: ICANN is an open and transparent organization. This required that all bids be posted as soon as possible after the proposal deadline. Sealed bids were not contemplated in the RFP process for this reason. UIA submitted its proposal knowing this to be the case.
  
  Both technical evaluation teams (Gartner, Academic CIO) performed their evaluations in confidential settings. Their identities were not made known until the staff Preliminary Evaluation Report was issued. Neither team or any of their members had any prior association with ICANN or ISOC (the "lead candidate").
  
  The proposer's comment about ICANN's involvement with the "lead candidate has no merit. ICANN, as it happens, is involved with all of the candidates or their partners, and receives significant funding from many of them, particularly from UIA's partner, VeriSign. The inference that the staff's preliminary recommendation was affected by the fact that many members of ICANN's Board of Directors are members of ISOC is entirely without merit, and no specific evidence was presented by UIA to substantiate this inference.
  
  
• Comment: Two bids were posted after the deadline. Why?
  
  Response: The bids were received by the deadline. The delay in posting was because of the need to work out certain confidentiality issues in accordance with the RFP. It was deemed to be more open and transparent to post all bids as soon as possible even if that resulted in two bids being posted later for the reason mentioned.
  
  
• Comment: The Names Council passed a resolution advising ICANN that the .org award should go to a non-profit organization. This preference was not included as a factor.
  
  Response: That recommendation was specifically not accepted by the ICANN Board of Directors. As such it was not reflected in the RFP. We note that the applicant submitted its proposal knowing this to be the case.
  
  
• Comment: Comment expressed that applicant cannot understand low rankings by Gartner of applicant's proposal with regard to stability and transition, given that the incumbent operator was their partner.
  
  Response: Gartner comments on this in their responses to comments received. See also subsection on UIA proposal in Section 5 ("Leading Proposals").

Notes:

1. For the reasons discussed in Section 4 ("Summary of Revised Evaluations"), the report of the Academic CIO evaluation team is not relied upon in this Final Report.

2. The Gartner team was also asked to evaluate the proposals with respect to the technical aspects of Criterion 11. Although the result of this evaluation is contained in Gartner’s final report, it was not used as part of their ranking of overall technical capability.

3. The Usage Evaluation Team was originally co-chaired by Harold Feld and Milton Mueller. As noted in the Fifth Update on .org Reassignment Process, however, Mr. Feld disclosed that he now understood that he had a conflict of interest that had not been apparent before, and resigned from the Team. We are grateful to Mr. Feld for his forthright actions and to Mr. Mueller for taking on the burden as chair on his own.

4. In general, the pricing of services is addressed as part of Criterion 7. For the purposes of Criterion 1, we assume that any proposal that complies with the requirement that basic services to registrars be priced at no more than the required $6 level be considered “affordable”, since this is the price that has been charged by the existing registry operator that has nurtured successful growth of the .org registry.

Page Updated 23-Jan-2008
©2002  The Internet Corporation for Assigned Names and Numbers. All rights reserved.