.org Reassignment: Academic CIO Team Evaluation Under Technical Criteria (#1, #7, #8, and #9)

Posted: 19 August 2002

.org Reassignment: Academic CIO Team Evaluation Under Technical Criteria (#1, #7, #8, and #9)

August 16, 2002

Mr. Stuart Lynn
President and Chief Executive Officer
Internet Corporation for Assigned Names and Numbers
4676 Admiralty Way, Suite 330
Marina del Rey, CA

Re: Evaluation of .ORG Registry Applicants

This report summarizes the technical review team's evaluation of the eleven proposals submitted for operation of the .ORG registry. Included is a description of the process that we followed, a listing of those proposals that we believe are most worthy of further consideration and a description of the factors on which we based our recommendation.

Process

Our technical review team was asked to evaluate the proposals based on four specific criteria in the Request for Proposals:

1. Need to preserve a stable, well-functioning .org registry (Criteria 1);
2. The type, quality, and cost of the registry services proposed (Criteria 7);
3. Ability and commitment to support, function in, and adapt protocol changes in the shared registry system (Criteria 8); and
4. Transition considerations (Criteria 9).

Our work was completed primarily in two sessions. The first session was used to assure that all members of our team had the same level of understanding of the domain name system, the functions of registries and registrars, and an understanding of the RFP process.

The second session was conducted in person over a day and one half. During this session, each team member independently reviewed each proposal to enable an impartial assessment of the four criteria above. After the individual reviews of each proposal were completed, all members then discussed the proposal's strengths and weaknesses against the criteria. Should one member's evaluation vary significantly from others', the basis of this difference was researched based on further review of the proposal. This process occurred successively until the review team agreed on the assessment of each proposal. The result of this process was the assignment to each proposal of one of the following three categories:

• high ranking proposal, could definitely operate the registry successfully;
• acceptable proposal, could probably operate the registry reliably; and
• marginal proposal, could possibly operate the registry but less evidence of this and less experience than other proposals.

It should be noted that the team weighted the first criteria – preserving a stable, well-functioning .org registry – as the most important. Therefore, our joint assessment of proposals on the ability to operate the registry, upon transition and without major risk during the six year term of the anticipated agreement, was given most weight.

Evaluation

The team was impressed with all eleven proposals and found its task of identifying the strongest proposals to be more difficult than originally anticipated. All proposals exhibited strengths and evidenced considerable preparation. The technical review team would like to pass on to all proposers the team's recognition for their high quality submissions.

A summary of the team's categorization of the proposals is presented in the following table.

The team feels that two proposals, NeuStar and ISOC, were the strongest, given the four criteria against which we were to complete our assessment. This assessment is based on our analysis that these proposals were strong on all of the factors on which the second group (Acceptable Proposal) was also judged at least satisfactory. Additionally, these two proposals were also very strong on the following:

• Very solid technology, technology plan, and technology staff with strong likelihood of stable registry operation during the life of the agreement. A partner in the bidding organization has a demonstrated ability to operate a large, global registry.
• A strong business and organizational model. Organization appears well prepared to operate registry without significant organization risks.

A second category of proposals ranked high, but were not seen as a whole as being as strong as those given the highest endorsement. Proposals falling in this category include Global Name Registry, Union of International Associations, Organic Names Ltd., and Register ORGanization. These proposals evidenced one or more of the following that resulted in a lower ranking than that applied to the group above:

• More than one organizational or technical transition during the six years of the anticipated agreement.
• A new and/or complicated organizational or technical plan that was judged to carry less than 100% chance of execution.
• A technology plan that was judged to be less strong than that of other bidders.
• A transition plan that was judged to be weaker than others or to take too long.
• A proposal that was not as complete as the two in the first category.
• Less or no experience operating a registry.

The third category of Marginal Proposals had a range of strengths and weaknesses. However, across all the four criteria, these proposals were not as comprehensive, complete or convincing when compared with the proposals in the other two groups.

We appreciate the opportunity to be able to contribute to the Internet community by participating in this evaluation. If you have any questions, do not hesitate to contact us.

Appendix A: Scope of Work

Date: Mon, 19 Aug 2002
To: Jim Dolgonas
From: M. Stuart Lynn
Subject: Confirmation of changed scope of work

Dear Jim:

This note is confirmation of our verbal discussion on July 26, 2002 redefining the scope of work of the Academic CIO evaluation team for the dot org registry proposals. As you recall, at that time I asked the team to group the proposals into three tiers (High, Acceptable, Marginal) according to the evaluation team's best judgment of how each proposal fits into this overall ranking, instead of rank ordering the top 3 to 5 proposals. It was my understanding from our conversation that you still planned essentially to follow the methodology outline in your proposal whereby members of the team made their own individual assessment to be followed by a group consensus process in which the team as a whole would come to a group conclusion by reconciling the individual opinions.

This change was made as a result of our discussions that tiering was more appropriate given the time available to the team, and because because both I and the team felt that tiering would give a more realistic picture than rank ordering given the expected close nature of several of the proposals and the difficulty of placing too much reliance on marginal numeric differences

Stuart

Page Updated 19-Aug-2002
©2002  The Internet Corporation for Assigned Names and Numbers. All rights reserved.