These reports will be made available for download by the registrar.

C17.4 Zone file generation. Procedures for changes, editing by registrars, updates. Address frequency, security, process, interface, user authentication, logging, data back-up.

On behalf of RegisterOrg, Registry Advantage will operate a world-class DNS infrastructure to support the .org registry operations. In order to overcome the limitations of BIND, RegisterOrg will use the proprietary DNS server application developed and deployed by Registry Advantage. The important features of this technology include:

The design of this DNS infrastructure allows per-domain updates to the Registry database from authorized parties. These updates are initiated through the registrar’s SRS interfaces or through the registrar secure web based account management interface. Once a registrar makes a successful change to a domain involving DNS-related information, the database will insert an update message into a queue for the cluster master. The cluster master then uses a secure network protocol to distribute the updates throughout all other clusters of DNS servers. In this manner, changes can be propagated from the database to all DNS servers with a frequency of under five minutes. The process of distributing these updates to satellite DNS sites is described in greater detail in the attachment, Section XI, Database.

This per-domain update mechanism (see diagram below) eliminates the need for traditional zone file generation to actively manage the DNS.

All updates will be logged by both the database and the DNS application; these logs will periodically be crosschecked to verify consistency. These logs will also be backed up on a daily basis (see C17.7) and can be used to restore the DNS system to its state at a previous moment in time, and provide an audit trail. In addition, traditional BIND compatible zone files will be generated on a daily basis and archived to tape as part of the data escrow process. This also allows for high level inter-day changes to be easily identified, in addition to the SRS audit logging.

Since the DNS information is automatically updated in near real time, no access to the DNS zone information is allowed outside of the SRS repository. All changes to the DNS data must be made through SRS to ensure access security and proper change tracking by registrars. The daily zone files generated are for backup and referential purposes only (see below) and are stored in a restricted file system that only Registry Advantage authorized personnel have access to.

In order to ensure the stability of its DNS infrastructure, the Registry will also deploy several backup DNS servers running BIND. Although these systems will not generally be used to answer DNS queries, in the event of a catastrophic failure of the principal .org DNS infrastructure, Registry Advantage will use the BIND-based DNS servers in a failover mode. Traditional zone file generation will be required to keep these servers up to date, and this zone file generation will occur on a once daily basis as described above. These zone files are validated using secure cryptographic hashes and digital signatures before they are loaded into the backup BIND servers. Once the ongoing stability of the Registry infrastructure has been established, these backup servers may be phased out.

Once the ongoing stability of the Registry infrastructure has been established, these backup servers may be phased out.

C17.5 Zone file distribution and publication. Locations of nameservers, procedures for and means of distributing zone files to them. If you propose to employ the VeriSign global resolution and distribution facilities described in subsection 5.1.5 of the current .org registry agreement, please provide details of this aspect of your proposal.

As described in C17.4 and the attachment in Section IX, Software, Registry Advantage will utilize a DNS infrastructure that does not rely on zone files as used in BIND and other traditional DNS server software. In order to provide end-users with a reliable and responsive DNS infrastructure, Registry Advantage will deploy DNS servers in a number of locations, including those referenced in C17.1:

Additionally, Registry Advantage will deploy DNS servers at its secondary site and will use an RFP process to select at least three additional locations for DNS servers. Sites would be selected in order to improve the overall geographic and network diversity for the .org DNS service. It is currently anticipated that one of these locations will be deployed in each of the following regions:

At any given time, the DNS cluster at either the primary site in New York or the secondary site will act as the “master” cluster. (The master cluster is the one adjacent to the active database and shared registry service, so if the New York site is providing the active shared registry service, the DNS cluster in New York will be the master.)

The master will be responsible for keeping distributed logs, issuing warnings to operators, initializing new servers in the other clusters, and zone data propagation. In the event of catastrophic master failure, the satellite clusters will still continue to operate. It will be possible to retrieve configuration information from them once the master is rebuilt.

The master will also distribute updated zone information to the satellite clusters. All communications between the master and satellite clusters will be encrypted for security. Updates received by an individual node within a satellite cluster will be distributed to the rest of the servers within that cluster. As indicated in C17.4, the Registry will tune communications parameters so that changes generally propagate to all clusters within five minutes.

Registry Advantage’s system employs advanced DNS software developed at Register.com, which achieves accelerated response times, rapid zone information updates, and improved fault tolerance. (Please refer to C17.1 for detail.) In developing this software, Register.com leveraged its considerable experience in providing authoritative DNS services for over three million domain names and specifically designed this software to create a highly scalable and available DNS infrastructure. Registry Advantage has built on this base of knowledge in creating its infrastructure and software to support the .org registry on behalf of RegisterOrg.

As indicated in C17.4, this entire DNS infrastructure will be backed up by a parallel system of DNS servers running BIND. Once a day, zone files will be generated (from the same location as the master DNS cluster) and distributed to all BIND servers via a secure, authenticated network layer protocol such as SSL. After performing integrity checks on the received data, BIND will reload on these remote systems in a staggered manner so that no fewer than half are active and ready to respond to queries at any given moment.

In addition to distributing zone data to its own DNS servers, Registry Advantage will also make limited use of VeriSign’s DNS constellation for several weeks immediately following the transition of registry operators. Initially, VeriSign will continue to use the last zone file generated during its tenure, so no zone file distribution will be required, but once the Registry begins to reliably generate zone file data, it will become necessary to propagate this data to VeriSign.

Registry Advantage proposes to periodically transmit zone file data to VeriSign either through the DNS protocol itself using such mechanisms as AXFR or IXFR, or by periodically using a file transmission mechanism such as FTP or SSH/SCP. The specific details for transferring this zone file data will be negotiated with VeriSign in the event the .org registry is redelegated to RegisterOrg.

For more details regarding the use of the VeriSign DNS constellation, see C18.1 and C18.5.

C17.6 Billing and collection systems. Technical characteristics, system security, accessibility.

Billing of registrars is critical to the financial stability of the registry. Strict processes are in place to ensure the least amount of disruption to registrars and the maximum security to the registry.

RegisterOrg will subcontract with Registry Advantage to provide billing and collection services. The decision to subcontract this service to Registry Advantage was made due to Registry Advantage’s extensive experience operating billing and collection systems for existing registries and registrars. RegisterOrg will audit its subcontractor to ensure appropriate billing practices.

To minimize the potential for disruptions caused by changes to billing procedures during the transition, RegisterOrg will adopt a billing procedures that are similar to those used by VeriSign, albeit with some improvements to accessibility and reporting capabilities. As is the case for the existing .org registry, registrars will receive monthly invoices, have secure access to reports through an Account Management Interface (AMI) and maintain letters of credit on file with the registry. There will be no requirement for registrars to pay using credit or debit cards and hence risk incurring additional processing fees or interest charges from credit card companies and banks.

Billing Process

In some cases, registrars may make partial payments pending minor disputed amounts relating to a small number of names. In these situations, the Registry would coordinate with the registrar to answer questions and obtain payment for outstanding invoice amounts.

If a registrar were not to pay, the Registry would shut off such registrar’s ability to register names. The Registry could draw on the letter of credit to pay outstanding amounts; in extreme delinquencies, the Registry may delete the names in question.

Account Management Interface (AMI)
AMI will provide a secure web-based portal with the following functions:

Subcontract
The Registry is subcontracting with Registry Advantage to carry out invoicing and collections responsibilities. Registry Advantage will be responsible for invoicing registrars, collecting from them, and sending the collections to RegisterOrg’s account.

The letter of credit from each registrar would be held by RegisterOrg.

C17.7 Data escrow and backup. Frequency and procedures for backup of data. Describe hardware and systems used, data format, identity of escrow agents, procedures for retrieval of data/rebuild of database, etc.

All Registry data is continuously replicated to two standby database instances. Registry Advantage also stores backups at off-site locations, including a Data Escrow provider.

Back Up
The .org registry will be protected by a variety of data backup and escrow mechanisms, ranging from the real-time replication of critical data to a comprehensive traditional tape backup scheme, and also including full third party data escrow.

The Registry will replicate most important data in real time. Using Veritas Global Cluster Manager and the Global Application Data Synchronization Framework, the Oracle database will be replicated from the active site (see the definition in C17.3) to the backup site. Application data will be copied to both the active and backup sites before moving from development to production systems. This duplication of data allows for failover procedures to provide a mechanism to resume the operation of the registry at the alternate site in the event of a failure at the active site.

At the primary site in New York, a secondary storage array will also back up the active database. As described in C17.3, redo logs will be copied from the active database server to the separate storage array used by the standby database server. These logs are used to bring the standby database instance up-to-date at one-minute intervals. In the event of a failure of the active storage array, the standby database server can become active and resume all database functions.

Two additional mechanisms will be employed to back up database information to tape, using AIT3 tape libraries as described in Attachment 10. First, logical backups will be made of the complete database daily. Second, complete physical backups of all .org database information will be performed on a daily basis at the primary site; at the secondary site in Asia, complete physical backups will be performed on a weekly basis complimented by incremental backups on a daily basis.

Application and log data will be stored to tape on a similar schedule to physical data backups for the database. This information will be completely backed up to tape on a daily basis at the primary site. At the secondary site, complete weekly backups will be complimented by daily incremental backups. The Registry will use Veritas Net Backup software to facilitate the restoration of data in accordance with disaster recovery procedures. Sets of tapes will be rotated to secure off-site locations on a regular basis.

Escrow Services
As required by Attachment S of the registry agreement between ICANN and RegisterOrg, the Registry shall deposit into escrow all registry data on a schedule to be agreed with ICANN ((i) full data sets for one day of each week (the day to be designated by ICANN) and (ii) incremental data sets for all seven days of each week) and in an electronic format mutually approved by the Registry and ICANN. The escrow shall be maintained by a reputable independent escrow agent. Registry Advantage’s expense of the escrow arrangement is included in its sub-contract fee to RegisterOrg. Registry Advantage has already identified an independent escrow agent to provide similar services for the .pro TLD; this same escrow agent can also be used for the .org data.

Per Attachment S, the use of the escrow data will be limited to verification that the deposited data is complete and in proper format to protect the privacy of the data. The data would be released to ICANN upon the occurrence of one of the events and per the terms described in Section 9 of the Registry Data Escrow Agreement (Attachment S of the Unsponsored TLD Agreement (“model registry agreement”), which provides the template for the ICANN-Registry Agreement) termination of the registry agreement or in the event of an applicable court order or subpoena. Further details about data escrow would be modeled on Attachments R and S to the model registry agreement.

C17.8 Publicly accessible look up/Whois service. Address software and hardware, connection speed, search capabilities, coordination with other Whois systems, etc.

Registry Advantage will provide a robust Whois service supporting a variety of query types. The Whois service supports over 800 queries per second, compared to an expected peak requirement of 350 queries per second.

Through Registry Advantage, RegisterOrg will make available a centralized, shared and publicly available Whois service at its site in New York as well as at the secondary location, with access to the Internet connectivity and infrastructure built to support all of the .org registry functions. This service will utilize a custom Whois server application based on the Whois application developed by Register.com. This robust application provides Whois service for over three million domain names and answers approximately 500,000 queries each day. The Registry, through Registry Advantage, will deploy load-balanced clusters of Intel x86-based hardware (described in C17.1) able to handle tens of millions of queries per day. While only a small number of servers are needed to accommodate anticipated load, the Registry will deploy additional hardware to insure complete fault-tolerance within the system. Updates will be propagated from the database to the Whois service in near real time when domains are registered or modified.

Initially the Registry will operate in a “thin” mode, meaning that only a limited set of data will be stored in the Registry’s database and made available through the Whois. RegisterOrg proposes a gradual transition to a “thick” registry in which a complete set of Whois data is stored by the Registry in one central location, allowing for more effective and efficient search capability. For further details regarding the transition from thin to thick registry operations, please refer to Question C22.

For queries relating to domain names, the publicly available .org Whois service will respond to requests for a single domain name and will generally provide back the following information:

* These fields are only available in the “thick” Whois mode.

Queries may also be performed on name server, registrar, and contact (when applicable) objects. Details regarding these queries are provided below.

Once the transition from thin to thick registry operations has been completed, data can be presented in a standard format for domains registered by any registrar. However, in order to accommodate local privacy laws, the registry will provide a mechanism for certain data from individual domains, or possibly across entire registrars, to be “masked” and made unavailable to the public. The specific fields that can be masked will be determined by RegisterOrg and Registry Advantage in collaboration with ICANN, based on current ICANN policy.

Protocol
Registry Advantage’s Whois service is made available through the port 43 Whois protocol. The Whois protocol is defined in RFC 954 and Registry Advantage will ensure that its Whois server operates according to the protocol defined therein on behalf of RegisterOrg. RFC 954 describes the Whois protocol as follows (note that while the protocol specification references SRI-NIC specifically, it may be applied to other registries, such as the .org):

To access the NICNAME/WHOIS server:

Connect to the SRI-NIC service host at TCP service port 43 (decimal).

Send a single “command line”, ending with (ASCII CR and LF).

Receive information in response to the command line. The server closes its connection as soon as the output is finished.

Query Syntax
The command line transmitted by the client must be in the following format:

[type =] query

In the syntax above, arguments indicated in italics are keywords that must be replaced by particular values as described below. Portions of the syntax contained within [square brackets] are optional.

The following type keywords are used to indicate that the query applies to a specific object type:

Domain	Search only domain objects. The query string is searched in the Name field. The format of the results of this type of query is described below.
Nameserver	Search only nameserver objects. The query string is searched in the Name field and in the IP address field. The results of this type of query will be consistent regardless of the parent domain and are described below.
Contact	Search only contact objects. This type of query may be performed only after thick registry operations have begun. The query string is searched in the ID field. The results of this type of query will be consistent regardless of the associated domain and are described below.
Registrar	Search only Registrar objects. The query string is searched in the ID field. This type of object can only be associated with domains in the extended .org name space. The format of the results of this type of query is described below.

If the optional type keyword is not provided, the default behavior is to search only domain objects as if the “Domain” keyword had been specified.

The query provided must be an exact match for the Name (fully qualified), ID, or IP address field of the object being searched for (as specified above). The query is considered to be case insensitive. All queries will return at most a single result, except nameserver queries based on IP address, which may return multiple results if the same IP address is associated with multiple name server names.

The server will return two varieties of responses: errors, and successful query results. All lines in an error result will be preceded by the string “%%”. The rest of the line will contain freeform text describing the error condition. Each line will be terminated by . The number of lines and length of each line is variable.

An example of an error result is below:

%% No match.

The other type of response is a successful query result. Three types of lines may be contained within a query result: comment lines, blank lines, and data lines. The first character in a comment line is the character “#”. These lines contain information that is not intended to be parsed by machines and may contain statements of registry policy, status information regarding the Whois service, or other freeform messages. These lines should always be displayed by Whois client software. Blank lines contain no data or only white space and are presented only for visual formatting purposes; they should also be ignored by parsers. Data lines are formatted in key/value pairs to facilitate machine readability. The format of data lines is the key, followed by the character “:”, followed by a space, followed by the data. All lines are terminated by .

Domain Query Response
Domain queries will return different types of responses based on whether a thin or thick set of data is stored for that domain name. Fields in boldface below may appear more than once. Fields in italics are optional and may not appear. For those domain names with only a thin set of data, the following fields will be returned in response to a Whois query:

For those domain names with a thick set of data, additional fields will be returned in response to a Whois query, and the “Sponsor Whois Server” field will not be returned. The total fields present in a response to a domain with a thick set of registry data are as follows:

Sample input and output for a successful query for each type of domain is provided below. The first example is a domain with a thin set of registry data

Input:	whois “domain = thinexample.org”
Output:	Domain Name: thinexample.org Sponsor ID: ORG-SAMPLEREGISTRAR Sponsor URL: http://www.sampleregistrar.org/ Nameserver: NS1.MYISP.ORG Nameserver: NS2.MYISP.ORG Updated On: 22-Feb-2003

The second example demonstrates sample output for a successful query on a domain with a thick set of registry data:

Input:

whois “domain = thickexample.org”

Output:

Domain Name: thickexample.org Sponsor ID: ORG-SAMPLEREGISTRAR Sponsor URL: http://www.sampleregistrar.org/ Registrant ID: JD1325 Registrant Name: Jane Doe Registrant Address: 345 Storybook Lane Registrant City: Littletown Registrant State/Province: KS Registrant Country: US Registrant Postal Code: 66645 Registrant Phone Number: (800) 555-5555 Registrant Email: janedoe@littleisp.org Admin ID: JD1324 Admin Name: John Doe Admin Address: 123 Anyplace Street Admin City: Anytown Admin State/Province: NY Admin Country: US Admin Postal Code: 12345 Admin Phone Number: +1.845.234.5678 Admin Email: johndoe@someisp.org Tech ID: JD1324 Tech Name: John Doe Tech Address: 123 Anyplace Street Tech City: Anytown Tech State/Province: NY Tech Country: US Tech Postal Code: 12345 Tech Phone Number: +1.845.234.5678 Tech Email: johndoe@someisp.org Nameserver: NS1.MYISP.ORG Nameserver: NS2.MYISP.ORG Updated On: 22-Feb-2003

Contact Queries
The following lines may be present in the response to a query for a contact object:
· Contact ID
· Sponsor ID
· Sponsor URL
· Name
· Organization
· Address
· City
· State/Province
· Country
· Postal Code
· Phone Number
· Fax Number
· E-mail
· Updated Date

Sample input and output for a successful query is provided below:

Input:	whois “contact = JD1234”
Output:	Contact ID: ORG-CNT-JD1324 Sponsor ID: ORG-REG-FICTIONAL Sponsor URL: http://www.fictionalRegistrar.org/ Name: John Doe Address: 123 Anyplace Street City: Anytown State/Province: NY Country: US Postal Code: 12345 Phone Number: +1.845.234.5678 Email: johndoe@someisp.org Updated On: 22-Feb-2002

Name Server Queries
The following lines may be present in the response to a query for a name server object:
· Nameserver ID
· Sponsor ID
· Registrar URL
· Server Name
· IP address
· Updated Date

Sample input and output for a successful query is provided below:

Input:	whois “nameserver = ns1.someisp.org”
Output:	Host ID: US-HST-123 Sponsor ID: US-REG-FICTIONAL Sponsor URL: http://www.fictionalRegistrar.org/ Server Name: NS1.SOMEISP.org IP Address: 192.168.1.2 Updated On: 22-Feb-2002

Registrar Queries
The following lines will be present in the response to a query for a Registrar object:
· Registrar ID
· Registrar URL
· Registrar Whois Server
· Address
· City
· State/Province
· Country
· Postal Code
· Phone Number
· Fax Number
· E-mail
· Updated Date

The “Registrar Whois Server” field will be phased out once the Registry includes only domains that have a thick set of registry data.

Sample input and output for a successful query is provided below:

Input:	whois “Registrar = ORG-REG-FICTIONAL”
Output:	Registrar ID: ORG-REG-FICTIONAL Registrar URL: http://www.fictionalregistrar.org/ Registrar Whois Server: whois.fictionalregistrar.org Address: 678 Mark Twain Lane City: Florida State/Province: MO Country: US Postal Code: 24760 Phone Number: (573) 333-3333 Email: info@fictionalRegistrar.org Updated On: 22-Feb-2002

Abuse Prevention
To prevent the abuse of the Whois database, the IP addresses of incoming requests will be logged. In the event that an excessive number of requests from a specific IP or network are made, the Whois servers will temporarily block additional requests from the same IP address or range.

Bulk Access
In addition to the publicly available Whois information, the Registry will provide bulk access to a limited subset of this information to individuals or organizations that agree to specific terms of use. This bulk access is governed by the registry agreement between ICANN and RegisterOrg (see attachment, C50.6, RegisterOrg) thereto. Bulk access at the registry level will include only the following information:
· Domain name;
· Sponsor ID; and
· Associated nameservers (if any).

Bulk access to additional information may be provided by individual registrars, as per ICANN policies and in accordance with applicable law.

C17.9 System security. Technical and physical capabilities and procedures to prevent system hacks, break-ins, data tampering, and other disruptions to operations. Physical security.

Robust security is an element in every part of the Registry’s system design. Each user’s access to the system is strictly limited.

Due to the critical nature of .org registry data, Registry Advantage will conduct all of its operations with significant attention to security, in conjunction with existing security practices. All Registry Advantage systems are designed to provide the RegisterOrg with world-class security at every level.

Physical Security
All registry database and shared registration services are operated in extremely secure hosting facilities. The following physical protections are used throughout data center facilities: use of 24-7 guard service, alarm systems, several layers of physical barriers, and use of biometric identification devices.

Generally, positive identification is required of all personnel entering and exiting the facility. Data center floors are isolated from areas in which visitors are commonly allowed. Each data center tenant is separated from others by steel cages, and movements within the data centers are carefully monitored.

In addition to preventing unauthorized persons from accessing the registry systems, the robust infrastructure has significant fire suppression, flood, earthquake, and even radiation resistance. The robust physical plant is intended to minimize the chance of unexpected events from impacting the registry’s operations.

Network Security
Registry Advantage employs a number of measures to prevent unauthorized access to its network and internal systems. Before reaching the registry’s internal network, all traffic passes through a firewall system. Packets passing to or from the Internet are inspected, and unauthorized or unexpected attempts to connect to the registry’s servers are denied and logged. The registry’s routers also act as the first line of defense against any denial of service attacks, with the ability to instantly deny traffic from a specific host, network, or even an entire Internet Service Provider. Refer to the network diagrams (Question C17.1) for details of Registry Advantage’s network infrastructure.

Front-end registry servers (such as the SRS, Whois and name servers) sit behind a second layer of network security provided by load balancing equipment. The hosts use RFC 1918 reserved address space that is not routable on the public Internet. Packets destined for specific services such as DNS or the registry’s SSL-protected applications are translated only after being subjected to additional security checks; suspicious traffic is dropped before being translated and will not reach the servers. Additionally, front-end systems are arranged into load balanced clusters; even if an attack is successful on an individual host, subsequent requests by the attacker will reach other servers, making it extremely difficult to take advantage of any weakness. Critical internal systems, such as database and file servers, also have non-routable IP addresses, and absolutely no traffic is permitted to reach them from the public Internet.

Registry Advantage also operates a network-based intrusion detection system (NIDS) at both the primary and secondary site. The NIDS monitors the network for suspicious activity. If possible malicious activity is detected, appropriate personnel will be notified immediately.

In addition to traditional stateful inspection and border ACL levels of security, Registry Advantage utilizes state-of-the-art Denial of Service (DoS) and Distributed Denial of Service (DDoS) prevention techniques, including advanced flow setup throttling and aggressive session table management. Especially susceptible to DDoS attacks are UDP based services such as DNS. However, Registry Advantage has worked extensively with its firewall vendor, NetScreen, to provide unprecedented protection against these attacks on DNS services specifically by helping to design UDP state building rules using layer 7 DNS information. With the NetScreen advanced state building rules, DDoS attack packets targeting the Registry Advantage DNS UDP service are dropped at the firewall, before they reach the internal access network.

Server Security
Registry Advantage employs a set of security precautions to ensure maximum security on each of its servers. Although specific procedures vary based on operating system, specific function of the host, and other factors, minimum standards on each server include:
· Disabling all unnecessary services and processes;
· Regular application of security-related patches to the operating system or critical system applications;
· Installation of tools such as SYN cookies to prevent denial of service attacks;
· Use of host based intrusion detection tools and log file analysis to ensure the ongoing integrity of each system; and
· Accounts on all production systems are only assigned to a limited set of administrative personnel. Developers, customer service employees, and others will not have login accounts on these servers.

Application Security
In addition to following commonly accepted security standards for application design and operation, Registry Advantage subjects its code to rigorous internal security audits on a periodic basis. Additional security for the infrastructure as a whole is increased by Registry Advantage's custom DNS and WHOIS server implementations. These remove the possibility of being susceptible to security problems that may exist in the freely-available software products. The IETF has already noted that heterogeneity within the global DNS architecture is a benefit - RFC 2870, section 2.1 states, "It would be short-sighted of this document to specify particular hardware, operating systems, or name serving software. Variations in these areas would actually add overall robustness".

Protocol Security
The security mechanisms used to protect registry-registrar functions are discussed in Question C17.2. These measures are intended to prevent unauthorized access, and in the future, the Registry may also introduce a unique digital signature for each database object, making it extremely difficult for an attacker to tamper with the registry’s data.

Access to Systems
The Registry will provide highly differentiated system access to different classes of users. Each user will be provided with access to only those functions and portions of the registry data that are required by that user. Although specific policies will be implemented for each class of user and each service or machine, a general outline of access restrictions follows:

• Registrars
  • Read/write access through both the SRS and Account Management Interface to all domains and other objects sponsored by the registrar.
  • A limited set of information (similar to that provided to the general public) will also be accessible for those domains sponsored by other registrars.
    
• Registry Advantage employees
  • Customer Service representatives: Read-only access to the entire registry database; ability to perform certain pre-defined functions such as undeleting domains that are within the delete pending period.
  • Software developers: Access only to development and staging environments. No access to production systems.
  • System administrators: These are the only employees with logins directly on the registry’s production servers.
  • Accounting and other business functions: Access to reporting capabilities similar to those provided to registrars, but with the ability to access the data for the entire registry.
• RegisterOrg employees
  • Access to reporting capabilities similar to those provided to registrars, but with the ability to access the data for the entire registry.
• Zone file and Bulk Whois agreement signees
  • Access only to the appropriate distribution server. The only function that these users will be able to perform will be to initiate the download of the appropriate file.

C17.10 Peak capacities. Technical capability for handling a larger-than-projected demand for registration. Effects on load on servers, databases, back-up systems, support systems, escrow systems, maintenance, personnel.

All of Registry Advantage’s systems are capable of handling significant surges in demand. Based on extensive tests and comparison with historical public data, Registry Advantage’s systems can handle three times the expected peak loads on SRS, DNS and Whois services.

Registry Advantage has taken a number of steps and precautions to ensure that its systems are capable of dealing with larger than expected surges in demand, even though it is unlikely the transition of the .org registry will be accompanied by the same kind of initial surge in demand that became commonplace with the opening of new gTLDs.

Registry Advantage has already designed and built systems to accommodate a registry of the size and importance of the .org TLD. To validate the current capabilities of its systems, Registry Advantage performed a series of load tests on its SRS, DNS and Whois services. With the goal of enhancing the external validity of the tests, the entire .org dataset was loaded into the registry database and test queries were generated from this dataset. The hardware, network and software configurations used for the testing matched those already in place at Registry Advantage’s primary data center, and the configurations also matched those proposed for the .org registry.

Expected test results were established prior to running the tests, based on several triangulated inputs. The purpose of this was to extrapolate and predict the expected typical and expected peak loads on the SRS, DNS and Whois services for the .org registry. Information from ICANN advisories, SnapNames’ State of the Domain reports and data supplied by VeriSign to the North American Network Operator's Group were combined with data drawn from Registry Advantage’s experience as a registry outsourcing provider to analyze and predict the expected volumes. For example, the expected peak for SRS checks per second was estimated using data from the ICANN “add storm” advisory , which related to the full com/net/org dataset. These numbers were scaled based on the portion estimated by Registry Advantage as related to the .org domain . A full description of the test results and methodology is attached in Section XI, Test Results.

The SRS tests included queries of all the major types (adds, checks, infos, deletes) combined into test sets that were representative of ‘typical’ loads as well as atypical ‘add storm’ loads. The Whois and DNS tests covered a range of .org query mixes (successful queries, failed queries, malformed packets, etc.) conducted in a random order to determine maximum queries per second.

The table below summarizes the major results attained in the testing. It is important to note that the number of servers in the SRS test cluster were actually fewer than the number proposed for .org (3 servers instead of 6 servers), yet the performance of the 3 servers alone exceeded expected peak capacity in all cases. Across all parameters and tests, performance significantly exceeded both the expected typical and expected peak levels as determined from the publicly available data about the .org TLD.

Parameter	Expected Typical	Expected Peak	Current Capability
DNS queries/second/POP	2,000	5,000	18,202[4]
Whois queries/second/cluster	35	350	815[5]
SRS checks/second	56	560	842[6]
SRS successful adds/second	<1	50	182
SRS changes/second	2	50	559
SRS deletes/second	<1	50	562
SRS infos/second	8	25	579
SRS connections/server	200	900	1007

Translating the SRS performance capability numbers into hourly peaks, Registry Advantage’s SRS servers are capable of up over 3 million checks per hour, in addition to the successful registration of over 655,000 new domain names in the same one-hour period.

These levels of capacity and redundancy mean that, even in a scenario where one or more components fail at the same time as increased registration volume, the systems will continue to process expected peak registration volumes. In the unlikely event that additional hardware is needed, Registry Advantage can re-deploy hardware on an emergency basis to accommodate larger-than-expected demands on the system during the initial months of operations. For example, if database resources are under strain, processors and memory can be moved from the standby Sun E-6500 to the active database server. This configuration leaves the primary site with only active and backup database servers, but given the other layers of system redundancy described in C17.1 and C17.3, this is an acceptable contingency for a short period of time. In the event that the SRS front-end systems become a bottleneck, servers from Whois and name server clusters can be moved to the SRS cluster, as these other services will be under relatively lower load during the initial months of the .org registry’s operation.

To monitor and manage peak loads in real-time, Registry Advantage will utilize state-of-the-art performance management and profiling tools. These tools are capable of modeling performance and correlating application work loads with specific infrastructure components and component sets. Peaks in various application workloads can be modeled and their impact anticipated with operational responses, including adding hardware in key areas of the infrastructure and modifications to logical storage layouts to avoid disk I/O bottlenecks.

Registry Advantage will use its performance monitoring tools to carefully manage the transition during the first few weeks of registry operation. As described in section C18.1, the Registry plans to pursue a cautious approach with the transition. The proposal is to gradually ramp up the operation of the SRS over the course of a week in order to prevent the system from being overloaded by pent-up demand at the time of launch.

It is also worthwhile to note that Registry Advantage plans to use extremely common Intel x86-based servers in all of its front-end systems. This hardware is nearly ubiquitous within the market and it would be extremely straightforward to obtain additional servers if needed. The use of layer 4 load balancing techniques as described in Section C17.1 makes it simple to rapidly move additional servers into a cluster. Although the Sun Enterprise hardware used for database servers is not as common, it is widely available and additional hardware could be readily obtained if necessary. The initial database server configurations allow for significant additions of CPUs and memory within the E-6500 systems that Registry Advantage will deploy.

As mentioned above, the full .org dataset was loaded into the database to test the DNS, Whois and SRS applications using real data. Loading the full dataset also provided the opportunity to confirm that other parts of the registry system, such as the database, storage, backup and escrow functions can handle this size of dataset. These systems are more than adequate for the 2.7 million names in the .org dataset. Indeed these systems can support at least 20 million names using the proposed hardware configurations. Considering potential growth, it is worth noting that these systems are taxed by a large number of total domains registered, and not by high day-to-day or hour-to-hour volumes. Short-term surges in registration volume should not affect these systems. If long-term registration projections do not adequately anticipate the growth of the .org TLD, these components can be upgraded well in advance of their limits being reached. Whois and name servers use a clustered approach, which allows the easy addition of inexpensive new hardware as the capacities of the system are reached. Once again, storage and backup systems have been engineered to accommodate expected volumes for over twenty million active domain names, with the Whois servers capable of handling millions of queries per day and the name service infrastructure capable of handling over 100,000 queries per second.

C17.11 Technical and other support. Support for registrars and for Internet users and registrants. Describe technical help systems, personnel accessibility, web-based, telephone and other support, support services to be offered, time availability of support, and language-availability of support.

Overview RegisterOrg intends to outsource most billing, customer and technical support for the .org registry to Registry Advantage. Problems and questions from registrars will be initially handled by the Registry Advantage Customer Support Center. The Center staff will attempt to identify and diagnose the cause of any problems and rectify them. Registry Advantage anticipates that the customer support team will be able to address any billing and customer support issues, as well as many technical issues. Some technical problems may be beyond the capabilities of the Center’s staff to resolve directly. In these cases, they will promptly escalate the issue to specialized technical staff (for example, network engineers or database administrators) for resolution. Any problems reported by RegisterOrg or any .org registrars will be tracked through a ticketing system so that Registry Advantage can monitor issue resolution and timing.

Certain problems or questions from registrars related to the registry-registrar agreement or RegisterOrg initiatives outside the Registry Function for and with registrars may be more appropriately addressed directly by the RegisterOrg. If such questions come in through the Service Center, they will be referred to RegisterOrg.

Ticketing System
In order to track customer service issues, the Registry would operate a trouble ticket system. A ticket would be created at the time of each new contact between registrars and the Registry. This system would provide customer service representatives with the ability to categorize, assign priorities to, and update trouble tickets to ensure that issues are routed to the appropriate personnel.

Major capabilities of the ticketing system include: · Automatic assignment of tickets to specific personnel based on problem type;
· Notifications generated when tickets are opened, modified or closed;
· Notifications to registry personnel generated when open tickets approach their expected service resolution times;
· Tracking of outages for SLA purposes; and
· Generation of reports on the status of all open tickets as well as trouble ticket histories.

Notifications
The Registry will seek to proactively notify customers of problems as they occur. Contact would generally be made through e-mail; however, if the scope of a technical problem affects the Registry’s capacity to send e-mail messages, an attempt would be made to contact registrars by telephone or facsimile. Registrars would be notified well in advance of planned maintenance events and invited to provide feedback regarding the Registry’s technical operations on a regular basis.

The Registry will provide urgent notice of any catastrophic outage or disaster recovery involving critical operations to the registry, and will follow up with regular reports by a senior registry systems engineer as long as needed. Systems outage involving non-critical operations to the registry shall receive similar regular, but less frequent, reports as long as needed.

Hours of Support
Technical support services will be provided on a 24-7 basis, 365 days per year, in order to equally accommodate registrars around the world. Although English will be the primary support language, Registry Advantage’s staff can communicate in several common languages, and will continue to seek diverse language capabilities among the additional staff hired to manage .org operations.

Customer and billing support services for non-critical issues will be provided Monday through Friday during regular business hours (Monday through Friday, 9 a.m. to 5 p.m. EST, excluding holidays).

Access to Sensitive Data
Access by Registry Advantage or RegisterOrg to sensitive registrar critical data will be limited. Read-only status will be available to help desk entry-level employees or outsource employees for such data in order to prevent inadvertent changes to registrar records. Support staff will be prohibited from providing information about other registrar’s operations. For further details, please see the security description in C17.9.

Customer Service Issue Resolution Process Registry Advantage has developed a complete issue resolution process for registrars—from ticket opening to ticket closure. The Registry Advantage Customer Support Center has established the following process to respond to inquiries most effectively and efficiently:
1. When the registrar contacts the Center to report a problem or make a request, a service request is opened and a ticket number assigned. The Center has an internal ticketing process to track progress toward resolution.
2. The first contact with the Customer Support Center is with a member of the Registry Support Team. To help expedite the customer ticket, the Team asks for account and contact information along with a description of the problem or service request. If required, additional technical information may be requested and security procedures used to verify the identity of the caller.

Average Callback Times
When Registry Advantage receives an e-mail or fax service request, the Center contacts the customer, based on the initial incident priority (on a scale of 1 to 4). Average callback time, by priority, is:
1 20 minutes
2 1-business hours
3 6-business hours
4 24-business hours

Average Resolution Time
The goal is to provide each customer with a rapid response and resolution to the inquiry, but depending on priority (on a scale of 1 to 4) the following guidelines can be used:
1 2 business hours
2 1 business day
3 2 business days
4 3 business days Ticket Prioritization All incoming tickets will receive prioritization based on the reported problem. Registry Advantage reserves the right to adjust the severity of an issue. Priority 1: A Priority 1 ticket is the highest priority within the Support Center system. The Center makes every reasonable effort to ensure that the customer is operational as soon as possible. Registry Advantage is in continuous contact with the customer until the problem is resolved. Typical Priority 1 issues include the following: · System inoperative; · Domain Name resolution impacted; · Registration activities impaired; · Letter of credit limit has been reached; · Registrar access to registry service is limited; and · Serious installation or upgrade issues (if they seriously impact progress towards completion and/or production dates).

Priority 2: Typically a Priority 2 ticket is for a problem that prevents the registrar from completing non-registration business but does not cause the registry or a registrar’s use of the registry to become completely inoperable. The Center makes every reasonable effort to resolve the reported problem as soon as possible. Typical Priority 2 issues include the following:
· Reports will not run
· Performance problems
· Functionality issues

Priority 3: A Priority 3 ticket is for a problem that causes a feature or system failure that can be avoided by the customer applying alternative methods. Typical Priority 3 issues include the following:
· Receiving error messages in the reports
· Receiving console error messages
· Exporting/Importing data files failing
· Upgrade or installation planning

Priority 4: A Priority 4 ticket is for a minor problem having only a minimal impact on the customer’s business. Typical Priority 4 issues include the following:
· General registration questions
· Changes to contact information
· General billing inquiries

Support for Registrants and Internet Users Primary support for registrants will be provided by registrars. Inquiries made by registrants directly to the Registry, either by phone or e-mail, will be referred to the appropriate registrar or possibly ICANN if the complaint is against the registrar. Similarly, neither phone nor e-mail support will be provided for Internet users.

RegisterOrg will, however, provide a publicly available Web site with useful information for both registrants and Internet users. Information on the public Web site will include:
· Registration policies
· A listing of ICANN-Accredited Registrars who offer .org registrations
· A Web-based interface to the Whois service
· Frequently asked questions about the .org domain name

RegisterOrg’s Web site will be designed to provide maximum utility to a broad spectrum of registrars, registrants and Internet users, including those with visual handicaps. In order to make the site broadly accessible, simple design concepts will be employed and text will be emphasized over graphics, Flash, and other visual elements.

C17.12 Compliance with specifications. Describe the extent of proposed compliance with technical specifications, including compliance with at least the following RFCs: 954, 1034, 1035, 1101, 2181, 2182.

Registry Advantage services comply with all relevant RFCs. In addition, Registry Advantage is an active participant in the standards-setting process.

All Registry Advantage systems will comply with the relevant specifications from IETF RFCs. The compliance of various services is summarized in the table below:

Service	Compliance
DNS	RFCs 1034, 1035, 1101, 2181 and 2182
Whois	RFC 954
SRS (RRP)[7]	RFC 2832 or draft-hollenbeck-rfc2832bis-01
SRS (EPP)[8]	draft-ietf-provreg-epp-06, draft-ietf-provreg-epp-domain-04, draft-ietf-provreg-epp-host-04, draft-ietf-provreg-epp-contact-04, draft-ietf-provreg-epp-tcp-04 draft-ietf-provreg-epp-tcp-04

It is important to note that Internet standards are continuously evolving, and the Registry intends to be an active participant within the IETF and to track the changes of relevant working groups. For example, the IETF’s provreg working group is currently in the process of defining a generic registry/registrar protocol, which would be of considerable interest to both the Registry and the registrar community. Registry Advantage intends to participate in this process and to implement the standard protocols that may emerge from this group’s work. Similarly, RegisterOrg and Registry Advantage expect to be active participants in current and existing IETF working groups, such as provreg, dnsop, and dnsext. Registry Advantage is currently an active participant in IETF standards-setting activities, both through mailing lists such as NAMEDROPPERS, as well as through regular attendance at IETF meetings. As new standards that affect .org registry services are developed, Registry Advantage and RegisterOrg will work closely with ICANN to ensure compliance with all relevant standards.

C17.13 System reliability. Define, analyze, and quantify quality of service.

RegisterOrg defines system reliability as a function of availability, performance and accuracy. Registry Advantage will engineer its systems to meet the strict quality-of-service definitions for reliable operation of the .org registry.

Domain-name registries provide some of the most essential functions of the modern Internet. The Internet is increasingly relied upon to perform critical functions for companies, organizations, and individuals throughout the world. These realities dictate that the reliability of a registry’s functions must approach 100% despite the constant need for upgrades to both hardware and software. RegisterOrg has taken those demands into consideration and proposes strict quality of service definitions as the basis for a best-of-breed SLA that will ensure and enforce the reliable operation of the .org registry. Registry Advantage, as a subcontractor RegisterOrg, has the capability to maintain and operate a high-availability and high-performance registry that will achieve a level of reliability that is unparalleled on the Internet today.

The Registry defines system reliability as encompassing several interrelated dimensions: availability (uptime), performance (response time and throughput) and accuracy (correctness of query response and resultant database transactions). A multidimensional analysis of system reliability is important because a system could be available, but with performance so degraded as to be useless and unreliable. Or, a system could be available and responding quickly to queries, but with inaccurate responses to those queries or inaccurate data entries, which makes it unreliable for use. In analyzing and formulating metrics of system reliability, the Registry consequently recognizes that the measurement of reliability involves measuring availability, performance and accuracy. The following specific definitions of system reliability are proposed for each of the major services in the .org registry:

Service description	System reliability definition
Shared registry service (RRP)	The shared registry service shall be considered to be up during any minute long interval in which 95% of add, modify and delete transactions are successfully completed within 800 milliseconds and 95% of check transactions are completed within 400 milliseconds.
Whois	The Whois service shall be considered to be up during any minute long interval in which 95% of all requests return correct data within 800 milliseconds.
DNS (cluster)	An individual DNS cluster shall be considered to be up during any minute long interval in which 95% of all requests to the cluster return a correct response within 300 milliseconds.
DNS (service)	The DNS service shall be considered to be up during any interval in which at least half of the total clusters are considered to be up.

The quality-of-service-definitions above assume that the measurement is being made from the same network as the measured service. In order to measure the responsiveness of .org registry services from various portions of the Internet, Registry Advantage will contract with a service measurement company such as Keynote or Exodus Monitoring and Management Services to provide data regarding the performance of core .org registry functions.

RegisterOrg will commit to these system reliability definitions in conjunction with “best-of-breed” uptime hurdle rates for DNS, Whois and SRS reliability. As explained in C17.10, Registry Advantage has the capacities in its existing systems to readily meet the exacting requirements of ICANN and RegisterOrg for system reliability and performance. C28 takes the definitions of system reliability provided in this section as the basis for describing the service level commitments that RegisterOrg will enforce and that Registry Advantage will achieve.

C17.14 System outage prevention. Procedures for problem detection, redundancy of all systems, back up power supply, facility security, technical security, availability of back up software, operating system, and hardware, system monitoring, technical maintenance staff, server locations.

Every major component of the .org registry has been designed with significant redundancy to make a major system outage extremely unlikely.

For critical .org registry subsystems, such as the database and storage, Registry Advantage will use highly available hardware from Sun, EMC and Network Appliance. These components are widely deployed for mission-critical applications in a variety of sectors including the Internet, financial, medical and government. The EMC Symmetrix storage arrays that Registry Advantage uses in both its primary and secondary locations feature hot-swappable drives, redundant power and cooling, RAID 0+1 protection of all data, battery-backed NVRAM to guarantee write completion, proactive monitoring of the environment and internal systems, and feature a 100% uptime guarantee backed by a $1,000,000 commitment. The Sun E-6500 database servers feature fully redundant internal power and cooling, the ability to repair or reconfigure components while the system remains online, CPU power control and hardware failure prediction. The Network Appliance and EMC ClarIIon storage arrays feature many of the same reliability features as the EMC Symmetrix, including redundant controllers and online standby drives.

Front-end .org registry services (including SRS, Whois and DNS) use redundant clusters of inexpensive, Intel-based hardware. While individual hosts lack the complete redundancy of the Sun Enterprise hardware used for database servers, RegisterOrg will use modern layer 4 load balancing techniques to instantly remove failed servers from production should a fault arise. The load balancers that RegisterOrg will deploy allow the use of extensive health checks on both the server and the application. If a particular node fails a health check, the load balancer will not direct any additional user requests to that node. In this model, each server operates as a redundant compliment to all of the other servers in its cluster.

All elements of the network (as described in C17.1) have been designed with redundancy in mind. All servers, storage arrays, and other production hosts will attach to the network using at least two ethernet connections. Although significantly less prone to failure than server hardware, all network components, including routers, switches, firewalls, and load balancers, will be installed in redundant configurations running in high availability mode. Generally speaking, the failure of any network element will result in a seamless failover to redundant hardware within seconds. One of the least reliable components of the network is likely to be the registry’s connection to the Internet. The Registry will address this by connecting to at least two Internet Service Providers (ISPs) at its primary location, and will use the BGP to intelligently share traffic and prevent the failure of either ISP from taking the registry offline.

Hardware redundancy will be complimented by advanced monitoring, clustering and data replication software. RegisterOrg will use advanced software to monitor, fault-detect and recover Oracle database services at both the primary and secondary location. This software allows for the creation of flexible policies, will notify RegisterOrg personnel in the event of a failure, and allows for cascading recovery from consecutive failures. As described in the attachment (Section XI, Software) Registry Advantage’s DNS server application implements multiple layers of internal redundancy, and allows for real-time notification of system managers or other key employees in the event of a failure. Question C17.9 has described the .org registry’s security model in detail, but it is worth re-iterating that in addition to the passive protections provided by firewalls, hardened systems and robust authentication measures, Registry Advantage will actively monitor for attempted breaches in security through the use of intrusion detection systems, log file analysis, and other tools such as Sam Hain.

In order to detect problems before they result in outages, Registry Advantage will engage in aggressive 24-7 monitoring of all .org registry systems and applications. High server loads, unexpected volumes of traffic, suspicious error messages in logs, slow responses to application queries and commands, and other telltale signs of problems to come will be automatically detected so that Registry Advantage staff can be notified and set to work correcting any potential problem. Registry Advantage will use a suite of monitoring tools to preemptively detect failures in either hardware or software components well before they develop into downtime events. Many of these detection modules also provide automated corrective actions in addition to notifications, so that potential downtime situations are averted without human intervention.

Registry Advantage will also contract with its hardware and software vendors to obtain enterprise class third party support available for all of its mission critical components. This includes Cisco, Sun Microsystems, EMC, Network Appliance, Extreme Networks, Oracle, and IBM. Support from these entities and the monitoring systems described above will all be managed through an enterprise class systems management console and framework that can integrate monitoring and management services from many difference sources into one seamless Network Operations Center (NOC), operated by Registry Advantage's personnel.

In addition, Registry Advantage will maintain and operate a complete replication of the production infrastructure in a parallel Quality Assurance Testing (QAT) environment. Prior to deploying any new or modified hardware or software into the production environment, these changes will be staged in the QAT environment and subjected to rigorous regression and stress testing by dedicated operations and quality control staff. This environment also allows RegisterOrg personnel to accurately replicate production conditions to help reproduce even the most obscure problems to help in the rapid resolution of any issues.

Core RegisterOrg services will be housed within world-class hosting facilities meeting the most stringent specification. These facilities offer multiple layers of physical security combined with 24-7 monitoring, fire suppression, and redundant communications facilities. In order to ensure power availability and conditioning, UPS systems are maintained throughout the facilities, and generators have been installed for the contingency of a long-term power failure.

In the event of a catastrophic failure at the primary site, a second site (also housed within world class hosting facilities in Tokyo, Japan, with the same characteristics as the primary site) will be able to take over full operation of the .org registry within 2 hours, and with all updates within the 5 minutes preceding the outage. (While 0-5 minutes worth of updates may be lost in this scenario, they may be recovered once the primary site is accessible.) This is among the best RTO/RPO commitments in the industry. Registry Advantage will be able to deliver on this RTO/RPO commitment by maintaining fully replicated systems in both the primary and alternate sites, and by replicating data through transaction log replication in one minute intervals, at both the Oracle level, and the front end systems level. This is discussed more fully in C17.15, and in C17.1 and C17.3 above.

C17.15 System recovery procedures. Procedures for restoring the system to operation in the event of a system outage, both expected and unexpected. Identify redundant/diverse systems for providing service in the event of an outage and describe the process for recovery from various types of failures, the training of technical staff who will perform these tasks, the availability and backup of software and operating systems needed to restore the system to operation, the availability of the hardware needed to restore and run the system, backup electrical power systems, the projected time for restoring the system, the procedures for testing the process of restoring the system to operation in the event of an outage, the documentation kept on system outages and on potential system problems that could result in outages.

No single point of failure exists within the registry infrastructure. Common failure modes have been considered, and can be corrected rapidly and with minimal disruption to operations. Many failure modes result in automatic recovery with zero downtime.

Although the extensive redundancy and monitoring described above make a major system outage unlikely, Registry Advantage has created a number of contingencies for recovery from various types of failures. Because of the complete replication of every component within the system, the highly resilient DNS network, and the multiple geographic locations, even significant downtime events should not impact the ongoing operations of the system.

Several types of failure and the recovery mechanism are described below.

Database Failure
A total of three database servers and three storage arrays provide cascading fallbacks and allow the registry to recover from multiple consecutive failures. The complete design of this database redundancy is described in C17.3. The Registry will use software to detect and recover from failures in database hardware or the storage subsystem.

In the event that the currently active database server should fail, but the standby database server and primary storage array remain operational, a simple database failover occurs. The standby database mounts the database filesystem that had formerly been used by the failed server, shuts down its standby database instance, and starts a new database instance using the primary storage array so that it can assume the role of the active database server. This is managed by Veritas Cluster Server software and recovers the database instance in less than three minutes with no data loss. This process is illustrated above.

Storage Failure
A description of database server modes is provided above in C17.3. Each of these database server modes is tied to a separate storage subsystem. At the primary location, the active and standby storage subsystems support the active and standby database server modes. At the alternate site, the backup storage subsystem supports the backup database server mode. Under certain failure conditions, the roles of these database servers and storage subsystems changes.

For example, if the active storage subsystem fails, however unlikely, the standby storage subsystem becomes active. If the primary site fails—or if cascading component failures cause a site failover, or both the active and standby storage subsystems fail—the backup storage subsystem becomes active, and the alternate site becomes the active site. This is described in our responses to Questions C17.1, C17.3, and C17.14 and depicted below.

Server Failure
Generally speaking, the failure of an individual Whois, SRS, or name server will not result in downtime. The Registry’s layer-4 load balancers will rapidly detect these failures, stop sending requests to the failed node, and notify system administration staff, who can then repair the failed system without incurring downtime. Similarly, individual servers behaving erratically or in need of maintenance can be intentionally disabled at the load balancer and repaired.

Cluster Failure
In the unlikely event of the failure of an entire cluster of servers at the primary site, such as all of the SRS servers or the master DNS cluster, Registry Advantage could perform a controlled database failover from the active site to the standby site. Management software would facilitate the actual failover process. Upon completion of the failover, database operations and other registry functions would resume at the secondary site. In such an eventuality, this becomes a special case of the storage and data server related site failures described above. Registry Advantage specifically designed its server clusters to be highly redundant, however, so this scenario is extremely unlikely.

Failure of any of the following application cluster pools in their entirety will result in a controlled site failover to the secondary site:
· SRS: If any of the supported protocol pools failed completely;
· AMI: If the secure web based application pool failed completely and could not be restored within a preset time specific to this application; and
· WHOIS: If the entire set of WHOIS pools or any set of pools dedicated to a portion of the distributed cache all failed.

Failure of any of these application clusters would not result in a site failover:
· Cluster Masters: All registry operations can proceed without the cluster masters for extended periods of time, so a site failover is usually not required;
· Systems and Network Management: If any of these services failed completely, short term measures can be implemented until the cluster was restored;
· Development and Testing: Although these are critical to the ongoing operation of the registry, extended outages of these clusters could be tolerated without directly affecting registry operations;
· QAT: Like the development and testing clusters, these clusters could be down for extended periods of time without directly affecting registry operations; and
· DNS: The geographically distributed architecture and unique data distribution model implemented in the Registry Advantage DNS architecture allow for the entire cluster at the primary facility to fail, however unlikely, without impacting the globally distributed service.

DNS Satellite Failure
Satellite DNS clusters may also fail. In that event, attempting to move database operations from the primary to secondary site is neither useful nor appropriate. Instead, by altering BGP route announcements, the IP traffic originally being sent to the failed cluster will be routed to an alternate cluster, which will answer requests directed at both its own IP addresses and those previously allocated to the failed cluster.

Registry Advantage has also been conducting research into the use of using IP “anycast” to provide its DNS system with improved redundancy and performance. Using this technique, the same IP address would be simultaneously announced from multiple PoPs, and each DNS cluster would be capable of responding directly to queries sent to any of these addresses.

In the event that a site failed, any IP addresses in use at that location would be seamlessly re-routed to one or more other locations. In the extreme case, each DNS Satellite site announces BGP routes for all DNS PoPs and each cluster can respond directly to any query. Additionally, this architecture supports MxN redundancy by having more physical DNS Satellite sites than announced name server addresses.

Each cluster will also have a set of non-production IP addresses that can be used to administer the hosts in the event the BGP announcements for the PoP service IP ranges were stopped for any reason. Additionally, each server in a DNS cluster has a non-routable RFC 1918 private IP address for communication with the cluster master and the other cluster members. Communications between DNS Satellite sites and the primary site hosting the cluster masters is done over an IPSEC Virtual Private Network.

Network Failure
The .org registry is built with extensive network redundancy. The use of high-availability techniques, such as HSRP and the spanning tree algorithm, mean that the failure of a network component will create no more than seconds of downtime before redundant equipment resumes the operation of the failed component. As described in C17.1, both the primary and secondary sites for the .org registry will have redundant connections to the Internet through diverse Internet Service Providers. Registry Advantage will use BGP to announce its space to each provider; in the event of the failure of an individual link or even an entire ISP, the Internet’s routing architecture will automatically redirect traffic through the other ISP. If both ISPs at the primary site should fail automatically, registry operations will automatically be moved to the secondary site.

Power Failure
All sites will be protected by both UPS systems and generators in order to ensure continuous electrical power. Sites are additionally protected with on-site power generation capabilities. Generally, these generators will come on line automatically and provide power to the facility in the event of a complete failure of the local power grid. In the event of an unexpected power failure at the primary site despite these precautions, a failover to the secondary site will allow operations to be resumed within minutes of the failure.

Unexpected Failure
There may be types of failure that are not currently anticipated. These failures could be catastrophic and theoretically both the primary and secondary site could be affected. In order to protect against unknown threats, several disaster recovery contingencies exist:

Because satellite DNS clusters do not depend on either the presence of a functioning database or the continuous operation of the master DNS cluster, they should continue to operate normally even in the event of an event affecting both the primary and secondary site. Due to the importance of DNS services to the overall stability of the Internet, satellite clusters have been designed to operate nearly indefinitely even without any contact from centralized administration

All data is backed up frequently. Physical and logical database backups are performed on a daily basis, and stored both on magnetic tape using state-of-the-art AIT3 tape backup technology and on the NFS filer. Application data is stored both on off-site development systems and is backed up on a daily basis. Tape backups are routinely moved to off-site locations, and data escrow services will be used to ensure that this information is stored and available. In the event of a catastrophic failure affecting both sites, it would be possible to restore data from tape in a relatively straightforward manner. Registry Advantage will use Veritas Net Backup to manage the backup process and to facilitate the speedy recovery of data if needed.

Registry Advantage’s systems have been designed primarily using commonly available hardware and software. The Intel x86 hardware used in server clusters are nearly ubiquitous in today’s market, and Sun Enterprise hardware is readily available throughout the world. Even in a situation in which both the primary and secondary sites were completely destroyed, it would be possible to rebuild an operational portion of the .org registry infrastructure using relatively common components combined with stored application and database information.

Although Registry Advantage is prepared for the possibility of having both the primary and alternate sites simultaneously destroyed, and could recover the full operation of the registry to a point in time previously moved off site on magnetic tape, the location of the alternate site and the diversity of the Internet Service Providers used at either site significantly reduce the likelihood of this occurrence. Detailed (re)construction guides and operations manuals will be stored off site along with the magnetic tapes to assist in the rapid reconstruction of the registry operation if such a need should arise, and these are audited on a quarterly basis for accuracy.

The Fault, Configuration, Accounting, Performance, and Security (FCAPS) management procedures in operating the registry are derived from many years of IT operations and management experience, as well as procedures and guidelines taken from the British Office of Government Commerce’s Information Technology Infrastructure Library (ITIL) . All of the relevant documentation and systems governing the configuration, capacity, change and release management process is also backed up and stored at the off site location. This can be used for validation of a reconstructed infrastructure as well as providing a baseline for restoring all operational functions in the event of a major disaster where both sites were destroyed.

The Registry has planned extensively for the possibility of losing a single database, or storage subsystem, or hosting facility, or network component, or ISP, or frontend server, or cluster, and leverages consistent technical and operational procedures for these scenarios.

For example, the database replication between the active and the standby database servers works by log switching one per minute, archiving those logs to a staging area, and copying them to the standby system so they can be automatically applied to the standby database server (running in recovery mode). This mechanism is virtually identical to the one used for replication to the backup system, although the distance the copy needs to travel before it can be applied to the backup database server is significantly greater. The software framework used to distribute the log files and apply them to the recovery mode database server instance guarantees the proper sequencing and data integrity of the log file applications in both instances.

In addition to the Oracle transaction logs, the same replication process copies the frontend systems’ logs to the alternate site so they can be used to recover any data lost in the Oracle log transition. The process of recovering the database (transitioning either the standby or the backup database server to become active) includes replaying the frontend system logs in addition to the Oracle transaction logs. This decreases the risk of data loss and improves the overall RPO capability.

The operational model used for change management is also key in our recovery capability. All changes to the production applications and infrastructure undergo rigorous stress and regression testing in a Quality Assurance Testing environment, which is a complete functional replication of the production systems. Once QAT testing is completed and approved by an independent testing and quality control team, the changes will be scheduled for release into both the primary and alternate sites. The primary site is validated immediately after the change is deployed. The alternate site will be validated internally on a monthly basis; a full-site functionality test and audit, including database instance recovery from both the replicated logs and a randomly selected magnetic tape set, will be performed quarterly.

C17.16 Registry failure provisions. Please describe in detail your plans for dealing with the possibility of a registry failure due to insolvency or other factors that preclude restored operation.

Registry Advantage has practically eliminated the chance of registry failure by thoroughly planning and implementing its systems, processes, and staffing plans. However, to address even the most unlikely risks, Registry Advantage has several addressed the unlikely event of registry failure.

Registry Advantage has built redundancy, backup, and security into the registry. This is documented throughout the rest of C17, in particular in C17.14 and 17.15.

Registry Advantage will use data-escrow services to protect data from possible failures including business failures, system failures, natural disasters, and sabotage. The data in escrow would be released to ICANN per the terms of the model registry agreement. (For further detail, please see C17.7.) The registry will also encourage registrars to deposit into escrow all registrar data on a similar schedule. But as the registry will have a centralized shared database, including all Whois records listed herein, the registry’s escrow commitment will ensure the availability of accurate records in case of possible registrar failure.

As to insolvency concerns, RegisterOrg is well-capitalized and presently maintains $10,000,000 in its account (see attachment in Section X, C50.5). Moreover, Register.com, RegisterOrg’s parent company, has cash and cash equivalents in excess of $200 million. Accordingly, RegisterOrg and Registry Advantage are capable of continuing registry operations under the most unstable of times. For a full description of Register.com’s resources, please refer to C13.6 and the Register.com securities reports (attached in Section X, C50.4, Register.com SEC Filings).