31 July 2002

Final Report of the New TLD Evaluation Process Planning Task Force

Table of Contents

1 Overview
2 Background
3 Approach
4 Parallel Processing: Opportunities and Risks
5 Structure
6 Areas
7 Questions to be Addressed
8 Monitoring Program
9 Evaluation Methodology
10 Evaluation Timetable
11 Funding and Priorities
12 Recommendations
Appendix 1: Task Force Comments on Questions
Appendix 2: Comments Received on Interim Report
Appendix 3: Comments Received on Draft of Final Report

1 Overview

This is the final Report of the New TLD Evaluation Process Planning Task Force (NTEPPTF). This Report is now submitted to the ICANN Board of Directors for its further consideration. It follows posting for community feedback and comments of the Task Force’s Interim Report in December 2001, and of a draft of this Final Report on June 15, 2002. Neither of the on-topic comments received has caused the Task Force to make any changes to this Report (see Appendix 3).

This Report poses Questions (see Section 7) in each of four Areas (Technical, Business, Legal, and Process) that the Task Force suggests should be addressed in any evaluation of the new gTLDs, and provides guidance to any Evaluation Team in the form of Comments on each Question (see Appendix 1). The Report also proposes an on-going Monitoring Program (see Section 8).

A complete evaluation of the new gTLDs is a formidable undertaking that could stretch out indefinitely and could be extraordinarily expensive. The Task Force has already significantly pared down its initial list of questions and concerns, but there remains a considerable body of work. In its entirety, this may well be beyond the resources of ICANN to carry out.

This lack of resources compared with the magnitude of the work to be done could also extend the timescale of the evaluation beyond what the community can reasonably be expected to tolerate. The Board, therefore, may need to balance the need to make decisions sooner against the risks of not waiting for completion of the evaluation (see Section 4).

To mitigate the implications of delay, the Task Force, therefore, has established priorities indicating those Questions that, in its view, must be addressed early and most importantly as a prerequisite to embarking on a another round of proposals for new gTLDs. These priorities are established by assigning a criticality factor to each Question; this criticality factor indicates when the Question must be answered within reason as a prerequisite to either launching a new request for proposals or to entering a successful proposal into the root zone file. The answers to the Questions are also segregated according to the stage of activity to which they apply (pre-contract, start-up, or steady-state stage); or whether the Question applies to sponsored or unsponsored new gTLDs. See Section 5 for a more detailed discussion of the Structure used to classify Questions.

The Report proposes possible alternatives for the Evaluation Team and how it might proceed with its work (Section 9). It also lays out a schedule (Section 10) for accomplishing at least what the Task Force considers to be the most important parts of the evaluation. Section 11 focuses on funding issues and the relationship of funding to priorities. Finally, Section 12 of the Report lays out a series of recommendations to the ICANN Board on how to proceed from here.

At this point the work of the Task Force is complete. As indicated in Section 9, however, members of the Task Force stand willing to assist in whatever way the Board considers to be appropriate if and when the Board proceeds with the evaluation itself.

2 Background

This is the final report of the New TLD Evaluation Process Planning Task Force (NTEPPTF). The NTEPPTF was chartered at the June 4, 2001 meeting of the ICANN Board of Directors in Stockholm, Sweden under the following resolution:

Whereas, in resolution 01.60, the Board directed "the President to prepare and present to the Board at its Stockholm meeting in June 2001 a proposal to form a committee to recommend processes for monitoring the implementation of the new TLDs and evaluating the new TLD program, including any ongoing adjustments of agreements with operators or sponsors of new TLDs;"

Whereas, the President has recommended to the Board the formation of a New TLD Evaluation Process Planning Task Force chaired by the President and consisting of members selected with the advice of the Names and Protocol Councils and the Chairs of the IETF, IAB, and ICANN DNS Root Server System Advisory Committee;

Resolved [01.74], the Board directs the President to form and chair a New TLD Evaluation Process Planning Task Force, for the purpose of recommending to the Board and the broader Internet community, by means of a report to be discussed at ICANN's Montevideo meeting in September 2001:

(a) a plan for monitoring the introduction of new TLDs and for evaluating their performance and their impact on the performance of the DNS. This assessment should focus in technical, business, and legal perspectives and rely on data gathered as part of the contractual arrangements with the new TLDs as well as other data inputs that can be readily secured; and

(b) a schedule on which a plan should be executed.

The President of ICANN carried out the directive of the Board to form the Task Force. Following consultations with the Councils and constituencies designated in the resolution the following individuals were appointed (primary affiliations shown in parentheses):

• Jaap Akkerhuis (ccTLD Constituency)
• Sebastien Bachollet (Business Constituency)
• Marilyn Cade (Business Constituency)
• David Conrad (Root Server Systems Advisory Committee)
• Michael Heltzer (Intellectual Property Constituency)
• Geoff Huston (Protocol Support Organization)
• Roberto Laorden (Protocol Support Organization)
• Stuart Lynn (Chair)
• Vany Martinez (Non-Commercial Domain Name Holders Constituency)
• Y J Park (Non-Commercial Domain Name Holders Constituency)
• Adrian Pinder (Government Advisory Committee Liaison)

This Final Report follows the posting on an Interim Report in December 2001. That Interim Report requested community comment and feedback on the approach being followed. Although the responses on the ICANN Forum were traditionally spirited and frank, almost all were off-topic insofar as their authors were intent on conducting their own evaluation of ICANN, of the new gTLD registry operators and registrars, and of the start-up processes, and did not comment on the Interim Report itself. Nevertheless, they did serve to reflect the general areas of concern associated with new gTLDs that were at least of interest to those segments of the community that participate in the ICANN on-line forums. There were three on-topic comments on the Interim Report itself. The Task Force carefully considered these comments, but for various reasons either did not accept the comment, or believe the essence of the comment was also incorporated elsewhere in the Report. More detail is given in Appendix 2.

A draft of this Final Report was posted for public comment on June 15, 2002. Among considerable numbers of off-topic comments and spam e-mail, two on-topic comments were received. These are included in Appendix 3. Neither of these comments has caused the Task Force to change substantially the contents of this Report.

It is important to re-emphasize is that the Task Force was not chartered with conducting the actual evaluation itself, but rather developing a plan for the Board's considerations as to how such an evaluation should be conducted. This Final Report should be read with this limited charter in mind.

3 Approach

In the Interim Report, we stated that the Task Force decided to divide its work into three parts:

1. Define the "Areas" for which the NTEPPTF recommends that an evaluation be undertaken (see "Areas").

2. Define a short list of "Key Questions") that the NTEPPTF recommends be posed for the evaluation of each Area (see "Questions to be Addressed").

3. Define "Criteria" that the NTEPPTF recommends be used for answering each Question.

The third step proved to be a difficult challenge. Useful – particularly quantitative – criteria are hard, if not impossible, to develop as benchmarks for asking questions about processes that have never been undertaken before. What are too many of something? What are too few? The Task Force instead has chosen to provide narrative guidelines to any future Evaluation Team that should prove useful in addressing the questions raised. These guidelines appear in the final analysis in the form of comments to the questions that are included in "Appendix 1: Task Force Comments on Questions". The Board will need to follow its own judgment following community consultation on how best to interpret what will necessarily be incomplete answers to many of the questions.

The Task Force has, in fact, decided to adopt a structured approach that is a slight departure from what was presented in the Preliminary Report. This is described more fully in Section 5 on "Structure". The basic idea, however, is that the questions to be asked will be classified along three axes according to (a) the Area into which they fall, as indicated above; (b) the Stage to which they are most relevant, where "Stage" refers to whether the question addresses the pre-contract stage, the start-up stage, or the steady-state stage; and (c) how Critical is the question to determining how and whether new proposals may be considered for another round of new gTLDs or for entry of any newly selected gTLD into the root zone file. Particularly this latter classification may create some different answers to the questions between sponsored and unsponsored TLDs. This classification helps to identify the most important questions to be addressed first as a prerequisite to launching a new round of proposals for new gTLDs.

As stated in the Interim Report, evaluation can be an almost limitless undertaking. It could continue almost indefinitely if every concern or question were pursued in the minutest detail. However, there are practical decisions that must be made by ICANN in the near future if it is to be responsive to community demands for clear indications as to how ICANN plans to move forward with the introduction of new TLDs. This consideration is discussed more fully in the following section on Parallel Processing.

The Task Force has therefore adopted as a guiding principle that its recommendations should be limited to answering only those questions that are likely to have a significant bearing on decisions that would affect those plans now and in the immediate future. Furthermore, questions should be limited to those that are answerable to the extent possible with reasonable certitude using data that is readily accessible (including such data as the new TLD registry operators are willing to provide beyond that which they are contractually obligated), and preferably as quantitatively as possible. This will be a challenge for the Evaluation Team since, beyond that specified in the agreements with the new gTLDs, necessary data may be difficult to acquire, although the objective of many of the questions can be addressed using appropriate sampling techniques.

4 Parallel Processing: Opportunities and Risks

Even given the foregoing guiding principle limiting the recommended scope of the evaluation, the approach being followed is still serial in nature. Some of the new gTLDs have only recently been launched at the time of writing of this report, while contractual agreements have only just been signed with one new gTLD registry, and it may yet be several months before the new gTLD is actually launched. Important issues have already come to the fore – particularly in connection with the implementation of "sunrise" and "landrush" domain name allocation methods – but it will be some time before any kind of steady state can be reached allowing for a full evaluation of performance.

Yet segments of the community are concerned about the delays in moving forward (it must be recognized that this sentiment is not universally shared – indeed, weaknesses in the domain name market that have become manifest over the past several months may have cooled some of the ardor for launching new gTLDs, particularly unsponsored new gTLDs, compared with the enthusiasm of the days when the market was growing at a frenetic pace). Those most directly concerned, of course, are those organizations whose proposals were not accepted during the first round of submissions in the year 2000 and those who are anxious to submit new proposals. The ICANN Board may find that it needs to move forward faster than can be accommodated by a serial and somewhat lengthy process. There may be a need to move forward in parallel.

The Board, therefore, may be tempted to consider the extent to which it can move forward on certain fronts – such as planning for the next round of gTLD applications – in parallel with the evaluation process. The Board may (or may not) find that the problems and risks associated with moving forward with new sponsored gTLDs are less than those associated with new unsponsored gTLDs, and therefore may wish to move ahead faster with introducing requests for proposals for new sponsored gTLDs.

Certainly much of the planning for new gTLDs can be done in parallel with the evaluation, as can much of the proposal solicitation and selection – provided that proposers understand the risks of submitting proposals with no guarantee that any will be selected. The key checkpoint is the actual entry of any newly selected gTLD into the root zone file, assuming proper procedures and understandings are implemented. Much, however, can be accomplished in parallel on the way to that point of no return.

The potential for moving ahead faster does emphasize the importance for starting to gather data as soon as possible. Appendix U (for unsponsored gTLDs) and Attachment 21 (for sponsored gTLDs) of the various agreements establishes requirements on the new registries for acquiring certain data. It is important that ICANN monitor the new gTLDs to ensure that the data is indeed being collected as provided for in the agreements. This may be problematic unless ICANN can devote resources to the task.

Risks:

Moving ahead faster, however, entails certain risks. This suggests that a timetable needs to be developed that carefully synchronizes the launching of any new gTLDs with the pace of the evaluation, ensuring that such launches do not occur unless there is reasonable certainty that downstream problems will not arise. This is one of the purposes of the three-axis structure introduced in the next section on "Structure".

Some of the risks associated with moving ahead too fast include:

• Some key item may emerge later in the evaluation that, had it been known sooner, would have affected the decision to launch one or more new gTLDs earlier in the process – that is, insufficient time may have passed to evaluate the current round of new TLDs so that not all potential problems have come to light. Clearly, the launching of several of the new unsponsored TLDs has already raised significant problems, particularly as part of their start-up stage. There has as yet been less extensive experience with the sponsored TLDs.
  
  
• The effect on DNS performance of adding new TLDs is still unknown. Conventional wisdom seems to be that although there are potential future risks in adding significant numbers of new TLDs, there may well be little or no risk¹ in adding tens or even hundreds of relatively small (in terms of numbers of domain names) provided they are not too "flat" in the shape of their hierarchy – insofar as the distributed architecture of the DNS presumes a hierarchical namespace for effective performance. (Relatively large numbers of domain names at the second level – as, for example, occurs with .com – may jeopardize performance regardless of how few or how many top level domains have been authorized.) The PSO and the IETF should be asked for their views on whether this conventional wisdom is substantially correct.
  
  
• If there are indeed practical limits on how many new gTLDs could be launched without destabilizing the DNS, utilizing available capacity now for certain purposes could preclude future options that may be of higher priority. Thus, for example, the addition of a large number of new TLDs could preclude launching multilingual TLDs before any interested community had time to develop satisfactory proposals in this area.
  
  
• DNS stability involves more than technical stability. There are also business or commercial issues that could affect stable performance and operation. Ultimately, both technical and commercial stability gauge the effect on consumers, both registrants and end users. They include such notions as predictable and understandable behaviors. There may not have been time to assess the commercial issues associated with DNS stability. Experience to date with the launching of unsponsored TLDs has certainly indicated that there are significant commercial issues to be considered.

On the other hand, there are risks in not moving forward in parallel. Although ICANN should certainly do what is right for the stability of the DNS, by not taking any action until after the evaluation is complete, ICANN becomes a pointed target of criticism to those who have strongly held beliefs about the need for more choice of domain names and hence for more gTLDs This is not a problem in itself provided that ICANN has clearly articulated reasons for not opening up a new round of gTLD proposals. In the absence of such reasons, there is danger that the potential registry operators who would want to support ICANN processes may look to other alternatives.

The first comment in Appendix 3 speaks to this issue and argues that ICANN should proceed as serially as possible.

5 Structure

The NTEPPTF has decided to classify the Questions that it believes should be addressed by the Evaluation Team along three axes as described below: Area, Stage, and Criticality. Besides ensuring a more orderly approach, there are three driving reasons for such a taxonomy:

• To allow the Board to establish priorities for the evaluation consistent with resources that might be available;
• To recognize that not all Questions can be answered until sufficient time has elapsed; and
• To provide a decision framework that addresses the possible need for parallel processing for new gTLDs.

Each Question (see Section 7) has been classified according to this taxonomy.

Area

The Area axis classifies the Questions according to whether they are pertinent to technical, business, legal, or process issues. These are defined more precisely in the following Section.

Stage

The Stages axis classifies the Questions according to their relevance to different time points associated with the introduction of new gTLDs. In particular, the following definitions have been adopted:

Stage S1	Pre-contract stage: from mid-2000 until signing of agreement(s).
Stage S2	Start-up stage: covering the period for the first year after signing of agreement, or the first 6 months of actual operation, whichever is later.
Stage S3	Steady-state stage: the first year following Stage S2.

Each stage can be and should be evaluated separately. Stage S1 is now complete for all the seven new gTLDs and all are partially or well into Stage S2 and two are now in Stage S3. Although the key issues for each stage may be distinct, it may not be possible to evaluate each stage until sufficient time has passed so that it can be assessed with some understanding of the effects decisions made at one stage may have on future stages.

Criticality

Criticality addresses the sensitive issues as to at what point and to what extent a Question must be answered to consider new proposals for new gTLDs, or to consider entry of an accepted new gTLD into the root zone file. The Task Force recognizes that the degree of criticality may differ between sponsored and unsponsored gTLDs, but rather than complicate the picture with a fourth axis these differences are noted in the Comments to the Questions contained in Appendix 1. The Criticality definitions are:

Criticality C1	The question must be convincingly answered as a prerequisite to a decision to allow new proposals at all in the applicable category (Sponsored or Unsponsored).
Criticality C2	The question must be convincingly answered as a prerequisite to entering a new gTLD into the root, but not necessarily as a prerequisite to allowing submission and evaluation of new proposals in the applicable category (Sponsored or Unsponsored).
Criticality C3	Reasonable attempts should be made to answer the question, but it is recognized that only partial or subjective answers may be obtainable. Incomplete answers are not of themselves a reason not to allow new proposals in this category (Sponsored or Unsponsored) or entry into the root of a new gTLD into the root.

With respect to both C1 and C2, the proper procedures and understandings must be implemented to ensure that there is good faith involved in the solicitation or later selection of proposals so that proposers clearly understand the issues affecting ICANN's ability or intent to select among submitted proposals, or to request U.S. Department of Commerce approval to enter selected TLDs, if any, into the root zone file, or the timing of any decisions.

6 Areas

In this Section, we give greater precision to the definitions of each Area.

The chartering resolution of the Board stated that the Task Force's work should focus on evaluation in the areas of Business, Technical and Legal. The Task Force, however, did not see this as an exclusive list. As such, the Task Force decided that a fourth area needed to be added to ensure a complete evaluation, namely, Process, including the processes followed by ICANN in selecting and negotiating the new TLDs.

The topics included in the proposed four areas can be summarized as follows:

Technical

• Effects on DNS stability and performance.
• Operational performance of the registry operators.

Business

• Compliance of new registry operators with signed agreements and with original proposals.
• Business processes followed by new registry operators in offering services.
• Scope of markets (registrants) attracted by new TLDs.
• Practices of new TLDs for regarding trademark concerns, cybersquatting, and dispute resolution.
• Effects on consumers and end users of new domain names.
• Provision of accurate and up-to-date Whois contact information.

Legal

• Scope and effectiveness of legal agreements.
• Monitoring of agreements for compliance.

Process

• Selection of new TLDs.
• Communication of information during the pre start-up and start-up stages (Stages I and II).
• Monitoring of performance of new TLDs.

7 Questions to be Addressed

This Section defines the key Questions that the Task Force believes should be addressed within the evaluation. In developing these questions, the Task Force has borne in mind three principles:

• The question should have significant bearing on an ICANN decision as to whether, when, and how to launch additional new TLDs or to shape the character of such new TLDs.
• The question should be answerable in a determinable timeframe (the longer the timeframe the less influence the result might have on any decision process).
• The question should be answerable to the extent possible either through analysis of data being gathered as part of the agreements, or through some other reasonably objective, albeit qualitative, process (requiring new data sources might present practical difficulties).

Each Question is classified according to the three-axis scheme described in Section 5: "Structure". A few Questions may need to be answered in different ways for different Stages and, similarly, may be Critical in different ways at each of these Stages. Thus, answers to a given question could affect both how new requests for proposals are designed as well as any future decision for entering a new gTLD into the root zone file – the latter may require a more detailed and thorough answer than the former.

As stated earlier, the Task Force recognizes that for reasons of lack of resources and time it may not be possible to answer all of the Questions formulated below. In Section 11: "Funding and Priorities" we recommend that priority be given to addressing the questions as follows:

• Highest priority should be given to those Questions indicated as Criticality C1, regardless of Stage. These should be addressed essentially right away. These are the question that must be convincingly answered as a prerequisite to a decision to allow new proposals at all in the applicable category (Sponsored or Unsponsored).
• Medium priority should be given to those Questions that are Criticality C2, Stages S1 or S2 or Criticality C3, Stage S1. These should be addressed within 6-9 months.
• Low priority should be given to all other Questions. Answers to these Questions can be deferred longer than 9 months until resources allow.

These priorities can be graphically depicted as follows:

In the following, the twelve highest priority Questions are indicated in boldface.

The Questions that the Task Force proposes are:

Technical

1. Has there been any measurable or otherwise determinable effect on DNS performance, security, and stability with the introduction of the new gTLDs, including any impact on the root server system?
[Criticality C1, Stage S2: Criticality C2, Stage S3]

2. Have new TLD registries incorporated technologies, including new technologies, that can adversely affect the performance of the DNS, violate DNS technical standards, or cause existing applications to fail?
[Criticality C1, Stage S2: Criticality C2, Stage S3]

3. Have there been any serious operational failures during the start-up and steady-state stages of operation that have caused, for example, serious interruptions of service, delays, or loading problems?
[Criticality C3: Stages S2 and S3]

4. To what extent have the registries implemented adequate protections against operational failure and performance problems?
[Criticality C3: Stage S2]

5. What metrics and other methodologies have been implemented (such as service-level agreements) to measure and ensure continuous and reliable service, and to what extent are they monitored for performance?
[Criticality C3: Stage S3]

Business

1. How effective have startup mechanisms been in protecting trademark owners against cybersquatting and other abusive registrations?
[Criticality C1, Stage S2: Criticality C2, Stage S3]

2. How often and how successfully have advance filtering and other the mechanisms for enforcement of registration restrictions been used, both in sponsored gTLDs and in restricted unsponsored gTLDs?
[Criticality C1: Stage S2]

3. To what extent and in what timeframe have the registry operators provided free, realtime access to a fully searchable Whois database?
[Criticality C1, Stage S2: Criticality C2, Stage S3]

4. What effect have the new gTLDs had on the scope and competitiveness of the domain name market, in terms of opening new markets, and in their effect on existing TLDs and registrants?
(Criticality C1: Stages S2 and S3]

5. Are adequate management policies and safeguards in place to ensure protection against accidental or malicious acts that could substantially interfere with continuity of service?
[Criticality C1, Stage S2: Criticality C2 Stage S3]

6. How effective were the different start-up mechanisms employed, from both a functional and an operational perspective? To what extent did they achieve their objectives or, conversely, cause consumer confusion, delays, legal issues, operational problems, or other impediments to smooth implementation?
[Criticality C1: Stage S2]

7. To what extent did the proposer comply with the terms of the original proposal in operating the registry? Are departures from the original proposal justifiable in terms of changed business conditions between the time the proposal was submitted and the registry went into operation?
[Criticality C2: Stages 2 and S3]

8. Were adequate resources devoted to support auditing, monitoring, and reporting or were these needs sacrificed to the business needs of marketplace entry?
[Criticality C3: Stages S1 and S2]

9. How viable were the business plans from an economic perspective? Did they achieve their goals during the first year of operation?
[Criticality C3: Stages S2 and S3]

10. Have the new gTLDs increased consumer interest in the Internet or provided more opportunities for existing consumers?
[Criticality C3: Stages S2 and S3]

Legal

The following Legal questions need to be considered in the context of the over-arching goals for the legal framework that was established. These goals are described more fully in the introduction to the Comments on the questions in Appendix 1.

1. How well do the agreements provide a framework for the addition of future TLDs?
[Criticality C1: Stage S1]

2. Have the new gTLDs encountered any legal or regulatory problems that were not considered at the outset, and, if so, how could they have been avoided?
[Criticality C1: Stage S2]

3. Have there been any unusual number of legal disputes during the startup period and how well have they been addressed?
[Criticality C1: Stage S2]

4. How well did the implemented agreements reflect the submitted proposals and ICANN policies?
[Criticality C2: Stage S1]

5. How well did the selection and implementation process take into account the Internet's international and uncoordinated legal framework?
[Criticality C3: Stage S1]

6. How well have the provisions of the agreements been complied with by either party?
[Criticality C3: Stages S2 and S3]

Process

1. To what extent were the Board's original objectives met through the processes that were used for selection, approval, negotiation, and implementation? How could these processes have been streamlined?
[Criticality C1, Stage S1: Criticality C2, Stage S2]

2. Were identified public policy issues addressed in the proposal and selection process; and were any unanticipated public policy issues identified during the selection and implementation processes that should be taken into consideration in any future new round of proposals?
[Criticality C3: Stage S1]

3. Did the selection process reflect sufficiently the international nature and diversity of the domain name system?
[Criticality C3: Stage S1]

4. Taking into account the primary need to ensure stability of the Domain Name System, did the fee charged for submitting proposals for new gTLDs strike a reasonable balance between affordability and the need to recover ICANN's costs of preparation, selection, negotiation, implementation, litigation, and evaluation?
[Criticality C3: Stages S1 and S2]

8 Monitoring Program

The Task Force highly recommends that ICANN institute an ongoing monitoring program of the new gTLDs immediately as a prerequisite to conducting an evaluation. This monitoring program should focus on the following key areas:

• The effect on the performance of the root of the addition of the new gTLDs as the number of registrants grows. The Task Force recognizes that it may be difficult to distinguish among the effect of the addition of new gTLDs and any number of other possible factors that may be at work. The Task Force also recognizes that measuring tools may not be sufficiently precise. Cause and effect cannot always be correlated or indeed separated. This implies that any monitoring program should only attempt to detect significant changes and effects that are demonstrable over time. The advice of the IETF should be sought in this regard.
• Operational performance problems that affect the stability of the DNS, particularly the ability to enter new registrants into the various registries, and any failures associated with "access" to those registrants.
• Accuracy and completeness of the Whois data. This may need to be accomplished through statistical sampling.
• The degree to which registries are conforming with their charters and other key contractual provisions, particularly with regard to including only registrants that conform to charter specifications.
• Any particular start-up issues, particularly with regard to sunrise or landrush problems.
• Compliance with contractual provisions regarding acquiring data and making it available to ICANN, and compliance with other reporting requirements.

The Task Force recognizes that such a monitoring program may be costly and require sources of funding that may or may not be available. We urge, however, the Board to budget funds for such an undertaking. Early detection of problems is important, particularly with regard to development of recommendations that might affect future decisions for new gTLDs.

9 Evaluation Methodology

Any evaluation is likely to be an on-going and costly undertaking. There is a definitive tradeoff between expense and time on the one hand, and the point of diminishing returns on the other. The Task Force recommends that the evaluation not become bogged down in extensive detail but be conducted at a "high-level". It should depend on:

• Data available as a result of Appendix U (for unsponsored gTLDs) and Attachment 21 (for sponsored gTLDs) of the new gTLD agreements, or such other data as may be readily obtained within the budgetary limits of the evaluation;
• Selected interviews with new gTLD personnel, sampled registrants, and other relevant individuals or organizations; and
• Analysis of published reports and comments.

To the extent that data obtained from the first of these sources is not confidential, every effort should be made to publish it to encourage other studies to occur. The second of these sources may only be accessible to the extent that new gTLD operators are prepared to share data that may or may not be confidential. Every effort, however, should be made to encourage new gTLD operators to share their experiences with the Evaluation Team, to the extent this does not jeopardize their competitive position.

To the extent possible, the evaluation should not only focus on problem identification and assessment but also consider possible solutions to the problems. The Task Force recognizes that proposed solutions may be subjective and not necessarily conform to community expectations, but it would be useful nevertheless to obtain any ideas that immediately jump to the minds of the evaluation team(s). By the time the evaluation is complete (or partially complete according to how the evaluation is phased), the Evaluation Team should have some useful insights into these problems.

The evaluation may be broken into separate activities corresponding to the priorities defined in Section 7, that is, first addressing those Questions that are Criticality C1. In parallel, the Monitoring Program outlined in Section 8 should be undertaken since the data obtained from this Monitoring Program bears on the answers to these highest priority Questions.

The Task Force recommends that ICANN initially solicit bids for an Evaluation Team to conduct the evaluation of those Questions listed in Section 7 as being of the highest priority, that is, those that are Criticality C1 indicated by boldface in Section 7. Within this priority, the Stage S1 Questions should be addressed first since this Stage is now complete; Stage S2 Questions can be addressed next. All Criticality C2 and C3 Questions can be addressed later. Reasonable latitude should be given to the Evaluation Team as to how it proposes to answer any of the questions.

An ICANN committee or task force (which we designate for convenience the TLD Evaluation Advisory Committee or TEAC) should be appointed to provide overall coordination and guidance to the Evaluation Team, but direction should be left to the ICANN staff. The TEAC should be required to provide quarterly reports containing findings to date. Members of the NTEPPTF stand willing to assist the TEAC in whatever way would be helpful.

Given the different kinds of expertise required to address the questions raised by the evaluation, the ICANN Board will need to decide whether to have a single Evaluation Team coordinated by the TEAC; or whether to appoint a separate evaluation team for each area all coordinated by the TEAC with staff assistance.

10 Evaluation Timetable

The NTEPPTF suggests the following schedule of events for the next steps in the evaluation process. It urges that the process be streamlined as much as possible to ensure that data can be collected and evaluated in a meaningful timeframe. Given the complexity of the process, lack of dedicated staff, and the sheer inertia of ICANN processes, this evaluation can stretch out indefinitely. Given this, therefore, at this point the Task Force only proposes an evaluation schedule for addressing those eleven questions designated as being of the highest priority in Section 7.

Consideration should be given by the Board to initiating another round of new gTLD proposals, perhaps appropriately limited in scope, in parallel with the evaluation process.

This timescale obviously stretches into next year. Shortening this timescale will depend on how much the Board and the community wish to inject themselves into the process and how much is delegated to staff to complete. The Board and the community will also need to decide when and how to launch a new request for proposals and what should be the form of such a request.

11 Funding and Priorities

The complete evaluation indicated by the above may require substantial funding, more than may be available by allocating the remaining balance of funds derived from the new gTLD application fees (these fees may also be required in part to fund unanticipated litigation that may arise). This presents a serious challenge. ICANN will either need to secure alternate sources of funds or forego part or all of the evaluation.

If choices have to be made, the Task Force recommends that priority be given to establishing the Monitoring Program outlined in Section 7. Ongoing monitoring in the areas described will at least provide valuable material to inform decisions about launching new gTLD programs.

If additional funds are available, the Task Force recommends that priority be given to addressing those questions labeled Criticality C1, then to Criticality C2 in that order of preference and that are associated with Stages S1 and S2. Consideration of other questions can be deferred.

The Board in any event will likely need to make choices and decisions faced with uncertain or incomplete answers to many of the questions. Judgment may need to take precedence over precision.

12 Recommendations

In summarizing the above, the NTEPPTF makes the following recommendations to the Board:

1. The Board should immediately initiate an Evaluation of all Questions indicated as Criticality 1 in Section 8 of this Report.

2. The Evaluation should adhere approximately to the schedule laid out in Section 10 of this report. This schedule can be accelerated to the extent the Board and the community is prepared to delegate more of the decision process to ICANN staff.

3. The Evaluation should address the questions defined in Section 7 of this Report as are applicable to Stages S1 and S2. To the extent that insufficient funds are available in the ICANN budget, priority should be given to answering those questions labeled as Criticality C1.

4. The Evaluation should follow the methodologies outlined in Section 9 of this Report.

5. A Monitoring Program as outlined in Section 8 of this Report should be launched as soon as possible, in any event no later than September 2002.

6. The Board should consider budgeting adequate funds for the above activities. In the event insufficient funds are available, priorities should be established as described in Section 11.

7. The Board should consider to what extent it can initiate planning of and proposal solicitations for new rounds of gTLDs in parallel with the Evaluation and Monitoring projects recommended above.

Notes

1. Within the context of the DNS as used today, the scaling property of the system makes extensive use of cached information to avoid continual repetition of queries being directed towards higher levels of the DNS hierarchy. On the assumption that caching continues to be used extensively within existing and new TLDs, then the assertion regarding levels of risk associated with tens or hundreds of TLDs is, from a technical perspective, a relatively conservative assertion. On the other hand, there may be intended applications associated with new TLDs where caching of any information within the hierarchy of the new TLD may be counter to the intended behavior of the application. In such a situation the non-cacheable TLD may exert significant load pressures on the DNS system that may, in turn, affect the DNS's scaling properties. Given this possibility of non-cacheable name domains, this assertion regarding levels of risk is not highly conservative in nature and encompasses a reasonable estimate of the scaling properties of the TLD domain.

Appendix 1: Task Force Comments on Questions

In this Appendix, we re-list the questions raised in Section 7, but provide guidance to evaluation in the form of comments on each question. Such comments indicate the kinds of issues or criteria that may be important in developing answers to each question.

In the Task Force's Interim Report, we had stated that questions should not be included unless precise and determinable criteria could be addressed in answering each question. The Task Force has changed its thinking in this regard. The questions may still be on the minds of many even if precise answers cannot be obtained, and it still may be important to develop qualitative answers doing the best possible. Besides, it is not even clear that scientifically precise answers can be obtained to any of the questions, at least not in any reasonable timeframe or not without significant expenditure of funds.

It is clear that answering any or all of these questions with any great precision constitutes an enormous undertaking. The resources required could be considerable. As indicated in the body of this Report, the NTEPPTF believes that it is critical that ICANN focus on the most important questions to be answered within the most reasonable time frame. The Criticality and Stage axes described in the body of this Report can provide valuable guidance. But even those questions that are the most Critical may not be easily answerable within a short timeframe or limited resources. The Board may have to settle for incomplete answers. As such, the Board may need to balance the desires of some members of the community in moving ahead faster with the introduction of new gTLDs against the risks of moving forward with incomplete information (see Section 4).

General Comment:

Many, if not most questions can be answered from different perspectives: those of the registry operator, of registrars, of actual or would-be registrants, perhaps of end users, of ICANN, and so forth. Undoubtedly there will be many kinds of perspectives on these questions, and it will be important to sort out answers that are factually based to the extent possible, from those that are entirely subjective. Any Evaluation Team will need to sort through these different perspectives. This may best be accomplished through a series of interviews with individuals, with organizations, and with focus groups. Indeed, this approach may be the most productive tool that can be used.

What can be seen from the comments to these questions is that, in most cases, obtaining an answer to the question may be a significant study in itself. The challenge for the Evaluation Team will be to pare down to the essential within the resources available. The classification scheme proposed in Section 5 of this Report can be helpful in this regard. See also Sections 10 and 11.

Besides providing answers to the Questions as far as possible, the Evaluation Team should feel free to provide suggestions on possible solutions to some of the problems encountered. By the time the evaluation is complete (or partially complete according to how the evaluation is phased), the Evaluation Team should have some useful insights into these problems, and any recommendations for solutions could be very helpful.

We do not repeat here the classification of each question by Criticality and by Stage. Please see Section 7: "Questions to be Addressed."

Technical

1. Has there been any measurable or otherwise determinable effect on DNS performance, security, and stability with the introduction of the new gTLDs, including any impact on the root server system?

Comment:

This is a critical question from a technical perspective. One potential approach would be to take a baseline data set of measurements that were undertaken prior to the introduction of each new domain and then take the same set of measurements at regular intervals thereafter and attempt to correlate changes in the measurement readings with changes in the DNS. The baseline measurements, however, are not practical since all the new gTLDs, with the exception of .pro, have been launched. Nevertheless, perhaps the baseline data could be approximated by extrapolating back from continuing measurements, normalizing for the known rate of growth of domain names.

One measurement technique that was used with the original introduction of the DNS into the Internet was that of comparing the total of DNS traffic with all other traffic carried on the Internet. A rapid rise in the proportion of DNS packets or DNS volumes in relation to all other applications that correlates with the introduction of new domains into the DNS root zone would tend to indicate some negative effect.

A somewhat different approach is to use a single site, and analyze all DNS packets within the site over a period of some days, and undertake this exercise at regular intervals. Such a detailed packet analysis can reveal issues in the implementation of new DNS gTLD s, as well as a number of other observations on the robustness of the implementation and operation of the DNS. This approach can match queries to responses, allowing the analysis to obtain an overall measure of DNS resolution times, the rate of successful resolution of requests. Flow tracking network monitoring tools have been used in the context of monitoring DNS queries and responses, and such an approach can provide a useful overall metric of the performance of the DNS from the perspective of end application performance.

A complementary approach is to measure the behavior of the root servers. The base set of relevant measurements can include packet and volume rates of delivered responses to queries, relative rates of invalid queries as compared to resolvable queries. Related measures regarding availability of the root server platform and availability of the DNS server process are also relevant, as are the CPU and memory load imposed on the root server platform by the DNS server application.

The overall objective is to determine is there is a consistent and observable incremental load added to the Root DNS Servers, and if there is a consistent and observable incremental performance penalty imposed on end applications that is attributable to the introduction of additional entries into the root DNS zone.

The evaluation team will need to work closely with the Root Servers Operators' Forum and a number of network measurement research groups in order to devise a consistent measurement and analysis framework, and in order to engage the participation of a number of data collection agents.

2. Have new TLD registries incorporated technologies, including new technologies, that can adversely affect the performance of the DNS, violate DNS technical standards, or cause existing applications to fail?

Comment:

The objective here is to understand whether the implementation of new TLDs has been undertaken such that it has a negative effect on the performance of the DNS, or such that it violates technical standards for the DNS or uses the DNS in novel ways such that existing applications fail to operate correctly.

It is expected that this question would be the topic of an evaluation rather than the collation of material gathered from the registries themselves.

At least an early approximate answer to this question would be important to shape any new requests for proposals for new gTLDs to determine whether it is important to incorporate language into proposals and agreements that would restrict certain kinds of technical implementations. A firmer understanding may need to be obtained before any new gTLD is entered into the root.

One part of this exercise is to evaluate if the new registries use DNS zonefile parameters that substantively alter the caching properties of the domain, or require constant zone refreshes – in other words if the TLD zone uses SOA record values that have a potential for affecting cache performance, zone update performance or the frequency of zone updates.

The second part of the exercise is to evaluate if the intended use of the TLD requires processes that are not part of the DNS technical standards, and are not available in current applications.

3. Have there been any serious operational failures during the start-up and steady-state stages of operation that have caused, for example, serious interruptions of service, delays, or loading problems?

Comment:

This evaluation question is focused on operational failures of individual components of the DNS service that have, in turn affected the delivered service. There are three parts to this question; the first refers to the operation of the root servers, the second part refers to the operation of the new TLD servers that are the particular subject of this evaluation and the third refers to the operation of the associated registries.

The focus should only be on serious failures, the kinds of failures that are publicly reported and are apparent. These are typically serious outages where it is impossible to resolve domain names in a zone for several hours or to register domain names for extended periods measured in days. Or the kinds of failures that can propagate across the DNS. Or extended failure in the ability to access Whois data. Serious failures should not include the normal kinds of operational failures that are routinely addressed, and most of which are not apparent externally because backup systems kick into operation or for other reasons.

Each server operator (root server and new TLD zone server operator) would be expected to lodge a response to this query at regular intervals to the ICANN evaluation group. Responses are anticipated to be either a negative response, indicating that across the most recent interval no operational failures were noted for the server, or, in the event of a failure of any component of the server, a report indicting the nature of the failure, its time and duration, anticipated impacts and the nature of the remedial actions taken.

In terms of evaluation of operational measures undertaken by the server operators to ensure operational integrity, the operating requirements for root servers, as documented in the IETF document Best Current Practice 40 (RFC 2870) serves as a useful resource. This document should be reviewed by the Evaluation Team. The compliance of the server operators to the accepted operating requirements should be periodically audited and the outcomes of this audit matched to the incidence of service component failures. Evaluation of registry operation is envisaged to follow a similar process of filing of a periodic report concerning the operating status of the registry. Where some service failure is noted, the time and duration of the service failure should be recorded, together with the nature of the failure and the method of service restoration that was used.

4. To what extent have the registries implemented adequate protections against operational failure and performance problems?

Comment:

This is a question where the evaluation is intended to take the form of an audit of protection measures. The process envisaged is one where the registry is to be requested to enumerate to the evaluation team the specific measures taken by the registry to protect against failure of operational systems and to safeguard against performance problems.

While this comment is not intended to be prescriptive of the form of protection mechanisms that should be used by a registry, the measures that would be anticipated here include the use of off-site duplication of critical data and critical service delivery elements, the consideration of individual component or subsystem failure in the design of the registry infrastructure, and the operational processes used by the registry to monitor the operational state of the registry service delivery systems.

The evaluation of the responses is intended to allow the evaluation exercise to comment as to the level of attention this aspect of a registry operation has received by the registries, and an assessment as to whether there are residual exposures in the described measures.

5. What metrics and other methodologies have been implemented (such as service-level agreements) to measure and ensure continuous and reliable service, and to what extent are they monitored for performance?

Comment:

Service Level Agreements are one approach intended to set agreed expectations relating to the performance of a service, where the agreement is between the service provider and the service client. They are not the only approach and, indeed, may not be the best approach depending on the circumstances. Nevertheless, they are one example of metrics and methodologies used to define service expectations and to achieve agreed levels of service.

The first part of this question is intended to establish whether the registry has considered what metrics and methodologies should be used to describe the baseline parameters of the performance of the delivered service, and what values of these metrics is considered to be acceptable to the service's clients. The evaluation exercise here is to make a judgment as to whether the choice of service level metrics are an appropriate and adequate set of metrics that encompass the registry operation, and, secondly, whether the baseline values of these metrics represent a realistic assessment of a minimum adequate service.

The second part is intended to establish whether these metrics and methodologies are monitored and if so, how they are used. This is envisaged to be a registry response that describes how the values of each service metric are gathered, the frequency of this collection of data, how the data is analyzed for conformance to agreed methodology, and the process used to initiate remedial action if the service level is not being achieved.

This question can be addressed by surveying registry operators, sampling associated registrars, and reviewing and reporting on the responses.

Business

1. How effective have startup mechanisms been in protecting trademark owners against cybersquatting and other abusive registrations?

Comment:

There is likely to be some degree of overlap between Questions 1 and 6, particularly with regard to reports of legal questions or complaints received.

However, in addition to the information received in answer to Question 6, there is more information that should be pursued with respect to this Question. In particular, data that should be obtained include reports of:

• The percentage of domain names currently registered in each of the new gTLDs that correspond to trademarks for which the registrant was seeking protection (a sampling approach to obtaining data may be required).
• For each new gTLD, the percentage of the total number of registrants in that gTLD that were awarded a domain name for which they were ineligible under the charter or restrictions of that gTLD.
• For each new gTLD, cases that have been filed under its start-up challenge procedures, and the percentage of successful challenges. Further analysis might be made at some point in the future if any of the results of these challenges were subsequently appealed under a national legal system.
• Difficulties faced by trademark owners in using each of the various start-up systems, the nature for these difficulties, and the reasons behind them.
• Quantitative data on the number of sunrise applications filed in .info and .pro, the number of trademark claim forms purchased in .biz, and the number of defensive registrations in .name.

2. How often and how successfully have advance filtering and other the mechanisms for enforcement of registration restrictions been used, both in sponsored gTLDs and in restricted unsponsored gTLDs?

Comment:

For sponsored gTLDs (.museum, .aero, and .coop), the key data to be obtained (perhaps using sampling techniques) is the percentage of registrations that did not comply with the terms of their charters. This can be done by comparing (on a sampled basis) actual registrations with membership or other lists pertinent to the registry's charters, to the extent this is feasible.

For unsponsored restricted gTLDs (.biz, .name, .pro), the process is similar, but, except possibly for .pro, membership lists may not be applicable. Again, however, a random sample of registrants could be selected and a review of any websites associated with that sample (while recognizing that domain names are not just used to construct websites) could be reviewed for compliance with the imposed restrictions. In the case of .names, a review of the Whois data could be undertaken (again, by sampling) to determine whether the registered name is indeed authentic (except where allowed otherwise by terms of the agreement).

3. To what extent and in what timeframe have the registry operators provided free, realtime access to a fully searchable Whois database?

Comment:

A reasonable assessment could be made based on obtaining the following data (some of which may need to be obtained using sampling techniques):

• The times that elapsed between the startup of the Whois database compared with the launch dates of the registry service.
• Frequency of update of the Whois databases.
• Percentage of required fields that are missing.
• Percentage of false contact data obtained through random sampling of the database.
• The degree to which those registries that promised enhanced Whois services (such as Boolean search capabilities) have followed through on their commitments.
• The costs, if any, for access to the Whois databases.

4. What effect have the new gTLDs had on the scope and competitiveness of the domain name market, in terms of opening new markets, and in their effect on existing TLDs and registrants?

Comment:

This question can best be addressed by comparing with plan the number and character of domain names in each registry classified according to whether they are:

• Entirely new registrants or existing registrants (that is, in some other registry)
• Have they established websites or are otherwise using the domain names for "productive" purposes.
• If they are existing registrants that are using the new domain name for productive purposes, is that use for new purposes that add to what the original domain name(s) is (are) used for, or, for example, are the new domain names just being used for complementary purposes such as a website that merely points to the old website. Or is it being used to substitute for the old purpose?

5. Are adequate management policies and safeguards in place to ensure protection against accidental or malicious acts that could substantially interfere with continuity of service?

Comment:

Policies are needed to protect against major risks ranging from malicious virus-like attacks to damage that can be caused by disgruntled employees to major physical damage.

Examples of such management policies should be sought from registries to the extent they are willing to share such examples. Standard policies that are in widespread use in conjunction with standard audits could be used as benchmarks, and registries requested to indicate to what extent they comply with or extend such benchmarks.

Since answers to this question depend on information that would not routinely be provided to ICANN, it should be recognized that it may be difficult to obtain meaningful answers to this question except to the extent that registries are voluntarily willing to provide answers.

6. How effective were the different start-up mechanisms employed, from both a functional and an operational perspective? To what extent did they achieve their objectives or, conversely, cause consumer confusion, delays, legal issues, operational problems, or other impediments to smooth implementation?

Comment:

This will be one of the most difficult, although one of the most important, questions to answer objectively. Any second-level domain name can only in the end be awarded to a single registrant. Those who wanted to register the domain names but failed to obtain it are likely to be critical of the outcome. Different start-up mechanisms were employed among the various registry operators making direct comparisons difficult. Nevertheless, one of the purposes of the "proof of concept" was to stimulate different approaches to gain some understanding of what does and what does not work.

The issues are particularly taxing to address from a functional perspective. Key data that should be obtained include reports of any:

• Significant potential registrant confusion concerning the nature and manner of any start-up mechanism, that is, of how they were expected to apply, when they were expected to apply or receive decisions, and the nature of the ground rules:
• Actual registrations that did not conform to the stated ground rules;
• Legal disputes that arose regarding the start-up methodologies that resulted in changes to the start-up processes;
• Significant numbers of complaints received by the registries or other ICANN constituent bodies from actual or would-be registrants (as distinct from non-participating observers of the scene). These complaints can be analyzed according to, for example:
  • Who is launching the complaint
  • The type of complaint
  • The effect of the complaint
  • The responsiveness of the proper authority in addressing the complaint

It may be necessary to approach this issue through sampling a selected number of complaints and analyzing them as case studies.

From an operational perspective, the issues are somewhat simpler. Key data that should be obtained from the registry operators and elsewhere include reports of any:

• Need to extend start-up deadlines and the reasons behind these extensions;
• Operational failures during the start-up period;
• Performance congestion during the start-up period.

Unexpected technical challenges faced by the registry operator and by the registrars during the start-up period that required any significant operational changes.

7. To what extent did the proposer comply with the terms of the original proposal in operating the registry? Are departures from the original proposal justifiable in terms of changed business conditions between the time the proposal was submitted and the registry went into operation?

Comment:

Proposals were accepted because, presumably, the ICANN Board found those proposals superior in the context of what was being attempted in this "Proof of Concept" than other proposals. There was a passage of time between selection and actual signed agreements, during which it is possible that changed business or other conditions perhaps justifiably warranted departure form the original proposals. Too many changes could imply that ICANN was too restrictive with its selection criteria and was not giving sufficient flexibility to those awarded new gTLDs. This question is directed at helping to understand whether or not this is the case.

To answer this question, the following kinds of data need to be provided for each new gTLD:

• The number and nature of significant requests received by ICANN staff for departures from the proposal;
• The number and nature of such requests that were accepted into the final agreements;
• The number and nature of reports or complaints received by ICANN staff following posting and signing of the final agreements based on departures from original proposals.
• The number and nature of changes that may have been initiated by ICANN staff to the original proposals.

8. Were adequate resources devoted to support auditing, monitoring, and reporting or were these needs sacrificed to the business needs of marketplace entry?

Comment:

This is another question for which it may be difficult to obtain meaningful answers since it depends on data that may not be readily accessible to ICANN. In effect, it depends on data and answers to questions that may not be readily volunteered by registries.

One measure, however, is the extent to which each registry complies with its agreement with ICANN regarding provision of data: Appendix U in the case of unsponsored gTLDs, and Attachment 21 in the case of sponsored gTLDs.

Another indication may be the extent to which the registry experienced unusual incidents of operational failures (see Question 2). As a follow-up to Question 2, it may be possible to determine whether investments in quality of service, auditing, and reporting requirements were unduly sacrificed to the product incentive.

On the other hand, it must be recognized that the incentive to build a product to enter the market is an imperative that cannot be ignored. No amount of auditing or reporting is of any value if the registry cannot commence operations. This question, however, is driven by attempting to understand to what extent proposers tend to underestimate the costs of entering the marketplace in a fully responsible manner, and whether there are lessons to be learned for future proposal cycles.

9. How viable were the business plans from an economic perspective? Did they achieve their goals during the first year of operation?

Comment:

This question is difficult to answer in the short-term. Long-term viability can only be assessed to the extent that the gTLD "stays in business" and there is no significant financial or operational collapse. Even rates of growth of registered domain names are not a reliable indicator. Any new business may expect losses in the short term during its start-up phase.

Paradoxically, long-term viability cannot be the determining factor in ICANN's decision to move forward with new gTLDs. The first-year viability is what is key to this decision. Furthermore, business plans change along with changing market and other conditions. A registry may not achieve its original goals, but still remain viable in the long run if it has successfully adapted to changing conditions. Indeed, a business may not be viable in the long-term if it doggedly sticks to its original goals and does not adapt.

Obviously, the question can be answered if there is a significant business failure or possibly if the gTLD is unable to function effectively from an operational perspective. But such obvious failures are not likely during the first year of full operation. Other indicators that might reveal problems regarding longer-term viability may be obtained from:

• Review of disclosed financial reports compared with business plans.
• Rates of growth or attrition of staff.
• Disclosed investments made in developing new features or technologies versus what was originally planned.
• Disclosed new calls for capital or new loans that were not foreseen in the original business plan.
• Stock values (for publicly held corporations).
• Expenditures during start-up operations to the extent these were more than planned.

The key question to be answered for future decisions is whether ICANN should change any of its requirements for financial information in assessing future proposals. Whereas ICANN should expect well-thought out business plans from proposers, it is not clear that ICANN can expect any guarantees of viability beyond the first year of operation. On the other hand, of course, such longer-term viability is essential if registrants are not to be left stranded by registry failures. Or at the very least, contingency plans need to be in place to address such potential failures, including the appropriate escrow of registry data.

10. Have the new gTLDs increased consumer interest in the Internet or provided more opportunities for existing consumers?

Comment:

This question may be difficult to answer, particularly in the short-term because of its somewhat subjective nature, and because of the rapid growth if the Internet with or without new gTLDs. It will be difficult to separate the effects of new gTLDs form what would have happened in any event. It may also be ambiguous as to who is the real consumer.

Some possible indicators are:

• The rates of growth of registrants in each of the new gTLDs.
• The rates of growth of "hits" received by a random sample of registrant websites (where relevant) tracked over a period of time.
• The percentage of registrants (again, achieved through sampling) that launch "active" websites (that is, those that are not merely advertised for sale or which just point to other websites) during, say, the first year of operation.
• The percentage of websites that are active, as just defined.
• Global sampling of individuals to assess whether they have heard of or use any websites that are registered to the new gTLDs.

The foregoing is very website-centric, and may not apply to, say, .pro or .name where e-mail could be an important if not dominant application. In the case of both these registries, rates of growth of registrants would be important indicators, since normally in these registries the registrants may well be the ultimate consumers. Does an attorney register in .pro to have an attractive website for the benefit of clients, or to have a convenient e-mail address?

Legal

One of ICANN's principal mechanisms for implementing its policies and achieving its mission is entering agreements with other entities responsible for coordination or operation of key elements of the DNS and the Internet architecture. These agreements include agreements between ICANN and registry operators (in the case of unsponsored TLDs) and TLD sponsors (in the case of sponsored TLDs).

One of the key questions is whether ICANN introduced unnecessary complexity into the agreements over and above what is required. This criticism has been made, often without significant examination of the issue. It is important that this broad question be addressed.

At the time seven new TLDs were selected in November 2000, ICANN had a registry agreement with Network Solutions, Inc. (now VeriSign) for the .com, .net, and .org TLDs. That agreement was part of a three-way set of agreements that included not only ICANN and Network Solutions, but also the U.S. Department of Commerce based on its continuing role under its 1993 Cooperative Agreement with Network Solutions. Because of this difference in legal background, and because the selected new TLDs (particularly the sponsored TLDs) had new features, the Board provided for negotiations of agreements for the new TLDs between ICANN and the registry operators and sponsors.

In negotiating the agreements, ICANN sought to achieve several overall goals:

• The agreements should assist in implementation of existing ICANN policies to the extent the TLDs operation is relevant to those policies.
• The agreements should facilitate the implementation of future ICANN policies.
• The agreements should require adherence to the material provisions of the proposals that were selected by the Board.
• The agreements should be as uniform as feasible given the above considerations, so that similarly positioned registry operators and sponsors are treated in similar ways.
• The agreements should be enforceable by ICANN, and avoid inappropriate risks of liabilities on ICANN's part.
• The agreements should avoid requirements not justified by the above considerations.

The comments on the following legal questions have these goals in mind. They are directed towards the overarching questions as to (a) Are these the appropriate goals for such agreements, (b) Should any be added, modified, or dropped, (c) Is the separation between Sponsored and Unsponsored, Restricted and Unrestricted, the most appropriate way to differentiate between different classes of TLDs, and (d) How well do these goals tie in to ICANN's overall mission?

1. How well do the agreements provide a framework for the addition of future TLDs?

Comment:

This question is inherently tied to the introductory statement at the start of this section on Legal questions and comments. More specifically, it is tied to the goals for the agreements stated there and the questions in the paragraph following that statement of goals.

The question is also inherently tied to answers to Question 3 below. If it is found that the answers to this Question 3 indicate that the agreements reasonably conformed with both the proposals and with ICANN policies, then a reasonable conjecture is that the agreements provide a good framework for going forward. If the answer is that they did not conform in significant ways, then there are serious grounds for a reexamination.

But there are also three other serious grounds for reexamination.

• First, although the agreements may have conformed with ICANN policies, those policies may need to be revised. This is a subject, however, that may be outside of the scope of the evaluation, although the opinions of the Evaluation Team would be of interest. These opinions, however, should be firmly grounded on substantive evidence of dysfunction, and not just be subjective opinion.
• Second, although the agreements may or may not conform with the initial proposals, the specifications for the proposals may have been insufficiently precise, causing some confusion in the minds of the proposers or some arbitrariness on the part of ICANN in evaluating the proposals. Again, the opinions of the Evaluation Team would be of interest, and, again, these opinions should be grounded on solid evidence of serious dysfunction.
• Third, the two-dimensional framework of (a) sponsored vs unsponsored, and (b) restricted versus unrestricted, may not be the right framework to carry forward. The Evaluation Team should be required to provide convincing argument should that be the case. Are there types of gTLDs for which the existing agreements are not suitable?

There is likely to be much that is subjective in the evaluation of these issues. Every effort should be made to minimize such subjectivity.

2. Have the new gTLDs encountered any legal or regulatory problems that were not considered at the outset, and, if so, how could they have been avoided?

Comment:

The key indicator here is whether lawsuits have been launched or threats of lawsuits have been made that caused major changes in behavior on the part of either the gTLD registry operator or of ICANN. An analysis should be made of major changes that each registry was obliged to make, if any, as a result of lawsuits or other legal threats, complaints received, or to comply with regulatory or other unforeseen requirements. A survey of the registries would be useful in this regard to the extent they are willing to share information not obligated by their Agreement with ICANN.

3. Have there been any unusual number of disputes during the startup period and how well have they been addressed?

Comment:

Each registry should be asked to provide a count of dispute requests received and acted upon in connection with their different start-up processes and start-up dispute resolution procedures. A report should be provided by the Evaluation Team that analyzes the extent to which these disputes substantially impaired compliance with the stated objectives of the gTLD.

4. How well did the implemented agreements reflect the submitted proposals and ICANN policies?

Comment:

In fulfilling its negotiations, ICANN staff had a responsibility to ensure that there were not significant and unjustifiable gaps between what was proposed and what was implemented in the agreements. If a putative registry were permitted to depart significantly from its proposal, ICANN would open itself up to cries of "foul" and potential litigation from those whose proposals were not accepted and who could argue that they, too, would have welcomed the post-award opportunity to refine their proposals.

On the other hand, particularly given the length of time between announcement of the award and signing of the agreement, and given rapid changes that were occurring in the business climate for domain names in the interim, certain changes must be desirable. ICANN should not require an awardee's proposal to fail by holding its feet to the fire of its proposal unreasonably, if there are legitimate business reasons for change. ICANN should also not be interfering unnecessarily in the business of the registry.

To gain understanding for future rounds of new gTLDs, therefore, it is important to assess the differences between the submitted proposals and the final agreements, and the reasons for those discrepancies. This should be a relatively straightforward analysis aided by ICANN General Counsel.

In addition, ICANN staff are responsible for ensuring that the final agreements conform with established ICANN policies, such as, for example, the registrar competitive model for ICANN-accredited registrars. The agreements should also be analyzed – in collaboration with ICANN General Counsel – to determine whether they conform to those policies or go beyond the intent of those policies. The opinions of counsel from the affected registries should also be sought. This analysis should shed light not just on the extent to which those agreements did or did not conform with those policies but also whether there are aspects of those policies that may need to be modified in the future.

5. How well did the selection and implementation process take into account the Internet's international and uncoordinated legal framework?

Comment:

The agreements with ICANN are not subject to any particular legal system. Disputes, however, are to be resolved through arbitration by the International Court of Arbitration of the International Chamber of Commerce (ICC), such dispute resolution to occur in Los Angeles, California. The notion is that the ICC would consider whichever law is most applicable to the issue at hand (see the answer to Question 7 in Reassignment of .org Top-Level Domain: Responses to Questions).

Clearly ICANN cannot be expected to resolve disputes in different legal systems across the world. The key issues underlying this question are whether (a) the system of arbitration used in these agreements allowed for sufficient differences of international legal concerns, that is, the differences in laws and legal systems across the world (see also the answer to Question 13 in Reassignment of .org Top-Level Domain: Responses to Questions); (b) whether the fact that such a system of arbitration is used could demonstrably inhibit proposals from different parts of the world or demonstrably biases the selection process; (c) whether there is anything in the agreements that would prevent the registry operator from complying with local laws wherever they operate, and (d) whether there is sufficient reflection in the agreements of the need for registry operators to comply with local laws.

The Evaluation Team may wish to engage a team of international lawyers (perhaps acting pro bono) to provide broad answers to these questions.

6. How well have the provisions of the agreements been complied with by either party?

Comment:

Interviews with ICANN staff and a sampling of complaints can be used to determine a candidate set of possible violations of agreements. Analysis of data from the required reports can also provide indications of possible non-compliance (the absence of such data is of itself a possible indicator). The selected candidate set should focus on the most serious of such potential violations. Each candidate should be investigated by the Evaluation Team (subject, of course, to available resources) to determine whether there is in fact a prima facie reason to believe that such a violation did, in fact, occur. The affected gTLD should be given every opportunity to respond, and staff comment should also be sought.

Process

1. To what extent were the Board's original objectives met through the processes that were used for selection, approval, negotiation, and implementation? How could these processes have been streamlined?

Comment:

The Board laid out its overall goals and expectations for the new gTLD program in a resolution passed in July 2000. It subsequently selected the seven new gTLDs at its meeting in Marina del Rey. These resolutions embodied the Board's selection philosophy (including the "proof of concept" notion), its selection guidelines, and its anticipated schedule, along with the selections themselves.

Clearly certain expectations were not met, particularly the negotiation schedule that was extraordinarily optimistic given that these were essentially the first new gTLDs since the inception of the DNS, and certainly the first to be launched with the expectation of hundreds of thousands if not millions of new registrants right "out of the box".

A comparison should be made between the Board's original stated expectations and what actually transpired subsequently. Any differences should be analyzed to determine the reasons for such differences, with a view to understanding how processes can be improved for the future. This task can mostly be accomplished through consultation with ICANN staff and the gTLD registries themselves.

2. Were identified public policy issues addressed in the proposal and selection process and were any unanticipated public policy issues identified during the selection and implementation processes that should be taken into consideration in any future new round of proposals?

Comment:

Several objectives that had public policy implications were identified in the ICANN Board resolution that authorized solicitation of new gTLD proposals. Paraphrasing, these include:

• Enhancing registry competition;
• Protection of rights of others, particularly intellectual property rights;
• Enhancing geographic diversity of gTLD registry ownership and locations;
• Enhancing utility of the Internet, including by addressing unmet needs.

An analysis should be made of the extent to which these public policy objectives were met by integrating the answers to other questions posed in this evaluation (Business 1, 2, 3, 4, 6, 10; Legal 2, 4: Process 3).

Interviews should be conducted with ICANN staff, new gTLD registry operators and others to determine if any other public policy issues came to the fore during the process. For example, the issue of whether and how to reserve country names in .info arose during the process raising the public policy issue of whether such reservation of country names is in the best public interest.

3. Did the selection process reflect sufficiently the international nature and diversity of the domain name system?

Comment:

An analysis should be made of the geographic diversity of proposals and of those actually selected. The latter should also distinguish between location of ownership and location of operations. The analysis should present (a) by region of the world (b) the number of applications received, (c) the characteristics of those proposals in terms of diversity, and (d) the number of proposals accepted.

4. Taking into account the primary need to ensure stability of the Domain Name System, did the fee charged for submitting proposals for new gTLDs strike a reasonable balance between affordability and the need to recover ICANN's costs of preparation, selection, negotiation, implementation, litigation, and evaluation?

Comment

A financial report should be prepared, with the assistance of ICANN staff, of actual revenues and expenditures-to-date through the implementation stage of the process. The balance of revenues less expenditures is presumably what is left to cover the costs of evaluation and future litigation, if any. To the extent possible, expenditures should be classified according to whether they were to support (a) the pre-proposal stage (b) processing, evaluation, and selection of proposals (c) negotiations leading to agreements with selected registries, and (d) any-post implementation costs not covered by other budgetary sources, such as litigation.

Appendix 2: Comments Received on Interim Report

Of the approximately 100 comments posted on the Public Comment Forum following posting of the Interim Report, 3 were on topic, that is, they were commenting on the Interim Report itself not providing feedback on how well the new gTLD process went or on the new gTLDs themselves. The Task Force appreciates these comments.

These three comments are listed below along with Task Force comments on whether or how the comments were addressed (or not) in the Final Report.

1. From Ray Fassett. December 3, 2001:

If I understand parts of the NTEPPTF evaluation process correctly, it appears that a parallel system can be implemented for prospective TLD applicants while data from the recently admitted TLDs are being properly analyzed.

If this is accurate, I think it is a sound approach towards further TLD expansion.

Mr. Fassett then quoted from the Interim Report: "Certainly much of the planning for new TLDs can be done in parallel with the evaluation, as can much of the proposal solicitation and selection – provided that proposers understand the risks of submitting proposals with no guarantee that any will be selected.

The key checkpoint is the actual entry of any newly selected TLD into the root zone file. Much, however, can be accomplished in parallel on the way to that point of no return."

Task Force comment: The Task Force comments on the opportunities and risks associated with parallel processing in Section 4 of the Final Report.

2. From Bret Fausett. January 15, 2002

At the outset, I would like to thank and compliment the Task Force for putting together a document that should allow the evaluation of new TLDs to begin quickly and with a clear focus once the final report is approved in Ghana.

I have only three comments:

1. Several aspects of the draft report are just right, and I would like to express my strong support for their inclusion in the final version, specifically: (a) that the focus of the review should be only on those aspects of new TLD operations likely to impact the Board's future decision-making; (b) that the review should focus on those questions answerable with reasonable certitude using readily available data; and (c) that many aspects of the testbed review and a second-round of selections can proceed along parallel tracks.

2. I'd like to see the final report more explicitly state that in addition to identifying problems and concerns arising from the implementation of new TLDs, the evaluation also will focus on possible solutions. A problem arising in the testbed is only a deterrent to the launch of additional TLDs if it is unsolvable or can't be minimized. As a consequence, without focusing on possible solutions, we won't be able to accurately gauge the impact of identified problems on future TLD launches. Data gathering would obviously precede a discussion about possible solutions, but the problem-solving stage is an important aspect of the process.

3. I appreciate that a number of participants in the ICANN process are concerned about the expansion of the DNS, but Section 4, "Other Issues of Concern," seems too speculative.

Specifically, on the second and third bullet points, the discussion of possible lower limits on the number of new TLDs and technical preference for small, hierarchical TLDs are the kind of assertions that should be accompanied by a technical reference. I'd suggest collapsing the second and third bullet points into a single statement: "The effect on DNS performance of adding new TLDs is still unknown. The PSO should be asked for its views on whether (a) expansion of the number of TLDs by tens, hundreds, thousands, or tens of thousands would have a significant negative impact on the performance of the DNS; and (b) if so, whether that impact would be minimized by the addition of relatively small TLDs instead of large, "flat" TLDs (such as .com)."

Thank you again for your work, and I look forward to seeing the final draft.

Task Force comment: We appreciate Mr. Fausett's kind comments and support in his first point. With regard to the second point, we have made it clearer in the Final Report that the Evaluation Team should propose solutions to problems encountered wherever this is feasible. With regard to Mr. Fausett's third point, the Task Force prefers its original language as being a better expression of the Task Force's intent.

3. From Edward Hasbrouck. January 16, 2002.

I agree with all of the "Areas for Evaluation" (Section 5) and "Questions to be Addressed" (Section 6) in the Interim Report of the New TLD Evaluation Process Planning Task Force as posted 3 December 2001.

However, I recommend to the Task Force that an additional area of evaluation, and a related set of questions to be addressed, should be added to those in the Interim Report.

The Task Force was directed "to recommend processes for monitoring the implementation of the new TLDs and evaluating the new TLD program, including any ongoing adjustments of agreements with operators or sponsors of new TLDs". This raises issues of (1) how operators and sponsors of new TLDs have exercised the authority delegated to them by ICANN under new TLD agreements; (2) how ICANN has or has not exercised, or should exercise, oversight over delegated decison making; and (3) what, if any, adjustments in TLD agreements should be recommended with respect to delegations of authority and ICANN oversight over delegated authority.

I believe that these issues are of special importance because many of the most widespread and serious criticisms of the new TLD process have related to how operators and sponsors of new TLDs have exercised the authority delegated to them by ICANN. In particular, many stakeholders have complained that the process of decision-making by new TLD operators and sponsors has been neither open nor transparent; that new TLD operators and sponsors have not made decisions in a bottom-up, participatory, or consensus based manner; that the results of many of the decisions made by operators and sponsors of new TLDs have not in fact reflected a consensus of opinion by stakeholders; that ICANN has failed to exercise effective, if any, oversight over decisions by those to whom it has delegated authority; and that the new TLD agreements should be adjusted to include stronger mandates for openness and transparency and stronger provisions for ICANN oversight over the exercise of delegated authority.

An evaluation of the new TLD process which fails to investigate the basis, or lack thereof, for these complaints will fail to address many stakeholders' concerns, and would thus be unlikely – regardless of its conclusions on other issues – to be able to contribute to any consensus among stakeholders as to the validity of these complaints, or what, if anything, should be done in response to them.

Accordingly, I recommend that the additional area of "Delegation of Authority" be added to the evaluation. Within that area, I recommend that following key questions be investigated:

(1) Have operators and sponsors of new TLDs made decisions, under delegations of authority by ICANN, in accordance with ICANN's principles of maximum feasible openness and transparency?

(2) What, if any, changes should be made in new TLD agreements to facilitate or ensure maximum feasible openness and transparency in delegated decision-making?

(3) Have decisions by operators and sponsors of new TLDs, under delegations of authority by ICANN, been made in a bottom-up manner?

(4) What, if any, changes should be made in new TLD agreements to facilitate or ensure that decisions by TLD operators under delegations of authority are made in a bottom-up manner?

(5) Have decisions by operators and sponsors of new TLDs, under delegations of authority by ICANN, actually reflected a consensus of opinion by affected stakeholders?

(6) What, if any, changes should be made in new TLD agreements to facilitate or ensure that decisions by TLD operators, under delegations of authority by ICANN, actually reflect a consensus of opinion by affected stakeholders?

(7) Have adequate and effective mechanisms been available under the new TLD agreements for oversight by ICANN of the exercise of delegated decision-making authority by new TLD operators and sponsors?

(8) What, if any, changes should be made in new TLD agreements to facilitate or ensure effective oversight by ICANN over the exercise of delegated decision-making authority?

Task Force comment: The Task Force believes that this subject is well covered by the proposed Questions, particularly Business 2 and 6, Legal 6, and Process 1.

Appendix 3: Comments Received on Draft of Final Report

Of the several hundred comments posted on the Public Comment Forum or e-mailed to ICANN following posting of the Draft of this Final Report on June 15, 2002, two were on topic, that is, they were commenting on the Draft Report itself. The Task Force appreciates these comments. The remaining comments were providing feedback on the TLDs themselves, or were just spam e-mail. The on-topic comments received were:

From Steve Metalitz. July 15, 2002:

The Copyright Coalition on Domain Names (CCDN), founded in 1999 to advocate the interests of copyright owners in the domain name system, and currently comprising the associations and companies listed at the end of this message, offers the following comments on the draft final report of the NTEPPTF ("draft final report").

The organizations in CCDN participate actively in ICANN processes, primarily through the Intellectual Property Constituency of the Domain Name Supporting Organization. CCDN has also made submissions in its own name within various ICANN public comment fora, including those dealing with the introduction of new gTLDs. CCDN participants are strongly supportive of ICANN's efforts to manage the domain name system, and commend its many accomplishments in this sphere.

CCDN also appreciates the thorough and comprehensive work reflected in the draft final report. We believe that the Task Force has done a good job of identifying and prioritizing the key questions that should be answered during the evaluation of the new TLDs. Its report provides an excellent road map for this critical process.

In general, with regard to parallel processing of applications for a new round of TLDs (see section 4 of the draft final report), CCDN believes the risks outweigh the benefits. As a matter of common sense, ICANN should complete and digest the results of the evaluation process - at least as to the most critical questions identified--before beginning a new round.

We agree that many of the risks of most concern to copyright owners may be less with sponsored TLDs, in which the sponsoring organization plays a gatekeeper function, than in unsponsored TLDs. Even here, though, it makes sense to have at least some idea of how the existing sponsored TLDs are functioning before kicking off a new round.

CCDN agrees with the draft final report that "planning for new gTLDs can be done in parallel with the evaluation," but urges caution about parallel processing of the solicitation or consideration of proposals. While the entry of a new TLD into the root may be a technical "point of no return," as a practical matter that point is reached much earlier in the process with regard to the expectations of new registry applicants. Asking "proposers [to] understand the risks of submitting proposals with no guarantee that any will be selected" may not be realistic.

Nevertheless, we agree that ICANN should "start to gather data as soon as possible," and should initiate and fund the monitoring program (aimed at gathering such data) as a top priority. The evaluation process should be carried out as expeditiously as possible precisely because it is not advisable to roll out any new TLDs until the evaluation process has been completed and the results digested by the community.

Thank you for the opportunity to comment.

Steven J. Metalitz
Smith & Metalitz LLP
Counsel to CCDN

CCDN Participant Organizations

American Society of Composers, Authors and Publishers (ASCAP)
AOL Time Warner
Broadcast Music, Inc. (BMI)
Business Software Alliance (BSA)
Motion Picture Association of America (MPAA)
Recording Industry Association of America (RIAA)
Software and Information Industry Association (SIIA)
The Walt Disney Company

From Jeff Williams, June 18, 2002 (this note was addressed to the GA List but the Forum comment mailbox was copied on the comment):

There are a few problems that I have with the final recommendations
of this Task force see: http://www.icann.org/committees/ntepptf/draft-final-report-15jun02.htm#12. Which are as follows:

…..[Text omitted. The author repeats the recommendations contained in Section 12 of the Report]

INEGroup recommends that the ICANN Board reject most of these recommendations for the following reasons:

1. Actual stakeholders were not allowed to participate in this task forces activities or even be aware of such activities in any way.

2. Members of this Task Force were "Selected" not Elected by the stakeholders as required in the White Paper and the MoU.

3. Recommendation #1, references the criticality as evaluated in Section 8 of this Task Forces report. It is our strong belief that the criticality determination in section 8 of this task forces report is greatly skewed and cannot accurately reflect the interests of the Stakeholder community as this Task force did not adequately or in any meaningful way conduct solicitation from the stakeholders so as to make such a determination as to criticality as put forth in Section 8 of this Task forces Report.

4. We have certain procedural problems with recommendation #2 to the extent that the this task force is seemingly desires of leaving more of the determination as to the need and desire of additional gTLD's to the ICANN staff for purposes of excelerating the recommendations in section 10. We believe and has been expressed by many other stakeholders in various other DNSO Constituency's that such determinations should remain firmly in the hands of the stakeholders, not just the ICANN staff. We believe that at this juncture, following the June 12th Senate hearings that it is clear that the ICANN staff has yet to show adequately it's ability to perform sufficiently, openly, transparently with the stakeholders in accordance with it's mandate's in the White Paper and/or the MoU.

5. We believe that Section 7 as referenced in recommendation #3, are not sufficient, and lack proper phrasing as to address the actual problems with the already introduced new gTLD. It is also in our considered opinion that the priority of the improperly phrased or couched questions in section 7 are only partly in order of importance or priority.

6. INEGroup agrees that monitoring needs to be ongoing and be done as efficiently as possible, but find the recommendation in #5 to be insufficient on several fronts. First, such monitoring should be conducted in an open and above board manner by independent organizations along with a elected group from the DNSO constituencies or the DNSO GA members. Second, recommendation #5 suggests a start date of no later than September 2002. We believe that this process should have already been ongoing currently and as a result the start date should be moved up to as soon as possible, and earlier than September 2002.

7. Recommendation #6 seems to suggest indirectly that perhaps funding or this task forces recommendations will fall short of the required amount and therefore provide a fall back position the task force outlined in section #11. We therefore believe that all other measures, including the ICANN staff and Board should give up a large portion of their salaries before section 11 is invoked.

Regards,

Jeffrey A. Williams
Spokesman for INEGroup - (Over 124k members/stakeholders strong!)