|
|
|
|
|
|
TABLE OF CONTENTS 1) PRODUCT DESCRIPTION
|
PRODUCT DESCRIPTION H.A. Technical Solutions H.A. software is high availability software designed to manage and control the automatic takeover of applications running on servers that are networked in a client server environment, servers running special applications in the telecommunications market or applications running on dedicated boards in the real-time or other operating environments. This is achieved by providing local and/or long distance monitoring of the status of the servers or boards in order to notify key personnel of the status of the mission critical application. It also allows managers to do maintenance on mission critical applications either onsite or remotely and automatically will failover mission critical applications to a hot standby server or other running server to prevent downtime. The H.A. software is designed using the latest techniques in high availability technology to prevent false failovers and to have the fastest detection and response time in the industry. The design of the H.A. software is modular allowing it to be modified to fit the needs of customers with special requirements. The design also makes H.A. from H.A. Technical Solutions the easiest to install and maintain. Key features of H.A. Technical Solutions H.A. software: Quick installation – 20 minutes. Provides automatic notification of hardware or software problems. GUI management tools. Software structure conducive to customization. Fast failover detection: less than 2 seconds. Fast failover and restore to operation: 4-120 seconds, including detection time. Client machines believe they are logged into the same server being used at time of failure. Supports all database applications, Internet, Intranet, any stateless application, telecommunication applications, and can be customized to handle custom OEM applications. Supported Operating Systems: Sun OS 4.1.4 or Solaris 1.x Operating System Support Under Development: ISI pSOS Available on all models that run supported operating systems. INSTALLATION Q. Is there a simple way to test an HA system?
A. Yes, H.A. Technical Solutions has provided a simple setup drawing and instructions in the manual
which makes it easy to configure and test the software. By doing this test, one can become familiar with
the operation of the H.A. software without having to configure the application. Once one has learned
and has become comfortable that the H.A. software is working correctly, it can be configured to the application. Q. What is necessary to create a highly available application service? A. To create a truly comprehensive high availability environment (which covers system, data, and application service availability) the following components are recommended: 1) High availability storage options. 2) Multi-computer configurations or boards that run a supported operating system. 3) Customizable on-line takeover management software. 4) Logical volume manager with high availability features (optional) Q. Which of these availability-enhancing products listed above can be obtained from A. H.A. Technical Solutions, LLC can provide: 1) Customizable on-line takeover management software: 2) Robust, file system based data services with journalizing: 3) Logical volume manager with high availability features: H.A.T.S. recommends purchase of File System from Programmed Logic High availability hardware options such as disk arrays with redundant power/cooling, un-interruptible power supplies, dual-ported disk subsystems, etc. would need to be purchased from other vendors of robust data services which support fault tolerance (the ability to maintain the integrity of mission critical data across application). There are over 160 vendors of array equipment to choose from. Q. How does HATS H.A. work? How do the nodes work together to insure application up time? A. The HATS H.A. application, running on a cluster node, maintains a comprehensive view of application service status. Application-specific information is collected by software-based agents that communicate directly with five H.A. daemons running on that node. These daemons communicate with other cluster nodes, exchanging status information. If a failure in a monitored node occurs, HATS H.A. takes the necessary action to recover any affected application service. An application can either be automatically restarted on a node or failed over to another node, which impersonates the network address of the failed node. The HATS H.A. Product Overview has an excellent description of the operational mechanisms which HATS H.A. employs for monitoring and taking action to perform takeovers. Q. How do I use HATS H.A. to make any application service highly available? A. To make any application highly available, you need to be able to do three (3) things to it: start it, gracefully stop it, and test it for its status. HATS H.A. includes the HATS H.A. Agent Template. Using this template users can build an agent that can automatically start, stop, and test any application service. This agent communicates with the five daemons the status information of the monitored application service. If the application service fails, HATS H.A. can either attempt to restart it on the same cluster node (a programmed number of times) or migrate it to another cluster node. Q. Are any special patches or drivers required for HATS H.A.? A. Due to its non-intrusive design, HATS H.A. works normally with standard OS configurations without any modifications. Other high availability products run into problems executing on a wide variety of hardware as there may be platform specific requirements or changes. HATS H.A. has no such limitations. HATS H.A. also runs on all models of the platform from a supplier, which the competitors do not. Q. How much storage space does HATS H.A. require? A. The entire HATS H.A. product consumes 5.9MB of disk space. The HATS H.A. Error Log file is started over every day and is as large as is required to report the number of problems that have accrued each day. These can be deleted at the discretion of the customer. Storage space required by our competitors' high availability solutions are greater both for the software and the error logs. Q. What is the processing overhead associated with HATS H.A.? A. CPU utilization by the HATS H.A. daemons are almost undetectable by CPU resource monitoring tools. A test configuration, done at the H.A. Technical Solutions, LLC lab, using two active servers consumed less that 2% of the system's resources. HATS H.A. needs the least amount of resources when compared to other high availability software. Q. What conditions can cause a takeover to occur? A.: • Processing halt or hang • Re-boot • Monitored application failure • Any user defined condition or event that would cause the application to halt • Power supply or PDU failure • Console keyboard-entered reset • Manually induced takeover/surrender • Any hardware failure causing applications to halt. • Any network failure that prevents transmission of data or heartbeat. Q. How does the backup node know that a primary application service has failed? A. Cluster nodes communicate with each other through a private, redundant interconnect across which heartbeats are exchanged. When no heartbeat is detectable from a certain node (for a pre-defined number of heartbeats) and verification has been performed through the public network, a hardware failure is detected. An application-specific agent detects an application software failure when it is unable to obtain service from the application after a pre-defined number of re-tries. Q. How long does HATS H.A. require to do a takeover? A. Takeover time depends on a number of variables, some of which HATS H.A. cannot control. Failure detection time is configurable at installation, but is typically about two seconds. Disk ownership and network address migration take on the order of a few more seconds. Application crash recovery is typically what takes the most time and can be little affected by HATS H.A. parameters. It is not unreasonable to expect recovery times of four seconds to two minutes for many applications, but use of journalizing, either in a file system or DBMS, can shorten the recovery time. Our customers tell us that our failover and recovery times are faster than other high availability products offered - other high availability software has shown response times to take up to two hours. Q. What is the impact of a failover on network clients? A. This depends on how the application architecture has been implemented. If the back-end application is using a stateless connection (e.g. UP based applications such as NFS), the clients will see only a slight application service delay and will not lose any work in process as a result of the failover. If the client application is using a stateful connection (e.g. telnet, ftp, DBMS), the clients will see a service disruption and will have to log back into the recovery node once the takeover is complete. Since failover requires crash recovery of the backend application, any work in progress at the time of failure must be re-submitted by the client. Q. What percentage of uptime does HATS H.A. provide? A. It is not unreasonable to achieve 99.9994% uptime with a properly configured highly available application service using HATS H.A. and a disk array. Q. Will HATS guarantee up time? A. H.A. Technical Solutions, LLC does not provide a standard guarantee in this area as there are too many determinants of overall application service availability over which HATS has no control. Q. What determines downtime? A.. 2) Time to initiate the failover. 3) Time to assure the integrity of the shared disk file system and/or DBMS data. 4) Time to initialize and start up all user applications. Q. How long does it take to sense a failure? A. The time to sense a failure is directly dependent upon the sensitivity of the HATS H.A. agents. In most cases this will be under two seconds for critical processes. Q. Are there any delays when a failover is requested from the main menu? A. Since H.A. does not need to sense a failure there is no time-loss involved when the command is issued from main menu. Q. How long does it take to failover the application to the alternate server? A. It takes about 2 seconds to move the applications. Q. Which items effect the time to do data integrity verification? A. The shared disk file system consistency check time is a critical factor in the total failover time. This time is dependent upon the size, type and condition of the file system that has to be checked. Again, a file system with a journal (e.g. File System from Programmed Logic or NTFS) provides dramatic improvements in this area. Q. What is the estimated time to run a data consistency check? A. As an example, a 1GB disk with no journalizing file system has a check time of 20 seconds while a 60GB RAID disk system with a journalizing file system also has a check time of only 20 seconds. Q. In a system with a data base, does the transaction monitor in the data base effect or add time to the failover? A. The transaction log that is played back at restart in order to ensure the database's integrity usually takes no more than a few seconds on database servers. Q. How long to restart the application? A. The time to restart is time to reload the application from the shared disk, which is usually a few seconds. There are some applications that could take much longer. To solve this, HATS H.A. can have the application preloaded on the alternate server which reduces restart time back to a few seconds. Q. Can you provide a typical example of the total failover time for a system? A. The following is a typical system configuration and the times involved in a failover. Q. How many failover can be expected in a year? A. From our own experience, and the experience reported by major data base suppliers, one can expect to experience 43 hours of downtime a year. Typically downtime is an average of 4 hours which means you can expect 10 downtimes a year. See the attached article from Datamation magazine with its' report from Oracle on downtime. Q. How much downtime will H.A. from HATS save me? A. If there are 10 downtimes * 18 seconds = 180 seconds of downtime divided by 60 seconds per minute = 3 minutes of total downtime a year instead of 43 hours. This is 99.9994 % up time. NOTE: These numbers are based on tests we have run - your number may differ based on differences in your system. Please calculate your exact failover times based on actual conditions in your environment. HATS H.A. DOES NOT GUARANTEE SYSTEM UPTIME. Q. What does this relate to in terms of money? A. The article from Oracle reports the down cost at $1,400 per minute. At 3 minutes a year, that is $4,200 lost due to downtime. At 43 hours down, it is $3,612,000 lost due to downtime. This was based on a survey of 500 companies that run 24 / 7 operations. CONFIGURATION Q. Can I failover between non-identical machines as long as they are running the same operating system? A. Yes, however, this can create administrative and performance issues. If the systems are not identical, device names may differ between the two systems which may cause administrative mistakes. Also, differing system architectures do not always allow for transparent NFS file server failover. See the NFS file server section for more information about how this can be addressed. HATS H.A. offers the most flexibility of any high availability software solution. Q. Is HATS H.A. failover bi-directional? Can either system take over the other system's applications and network services while continuing its own network services? A. Yes. In HATS H.A. this is called a symmetrical configuration. Each system will assume the other's services if it fails. The take over machine will mount the file system and impersonate the failed server's service network and TCP/IP address. High availability configuration from other vendors cannot always offer this capability. Q. Are two heartbeat links recommended? A. H.A. Technical Solutions software does not require two heartbeat links because it reroutes the heartbeat through the public network if the private network fails; but HATS recommends having two heartbeat links. Other solutions require having two links. Q. What types of heartbeat links are supported? A. HATS H.A. supports TCP/IP over Ethernet links and a serial connection if only two servers are connected. HATS H.A. also supports any other type of network protocol. To ensure that the H.A. system is reliable and not subject to a problem of having the heartbeat cause the system to failover due to TCP/IP problems, broken cable or failed network connection cards, HATS H.A. will reroute the heartbeat packets through the public network. This assures that a minor failure will not cause a failover. This method has proved to be more reliable than trying to route through the disk array as some other high availability solutions do. Q. What does the heartbeat consist of? A. The HATS H.A. heartbeat packets consist of the following information: 1) The name of the server sending the packet, 2) The status of the server sending the packet, and 3) The status of all jobs running on the server sending the packet. Q. Are the intervals for HATS H.A. heartbeat and agent tests configurable? A. Yes. Q. Can I use caching for performance in HATS H.A. configurations? A. If data is cached to a volatile space which is unavailable to the backup node in the event of a primary node failure (like UNIX buffer caches), then the answer is no. Data in the cache often reflects rights that the primary application thinks are committed, but in fact have not yet been flushed to disk. If the recovery node cannot access this information, it cannot determine the most up-to-date state of the data and will resume processing after recovery with corrupt data. If data is cached to a disk, and this disk is accessible from the backup node after the recovery is complete, then this does not cause a potential data corruption problem and is safe to use in HATS H.A. configurations. Non-volatile disk array caches can be used as long as they are available to the backup node during the takeover process. Use of Sun's SPARC storage array write caches is not recommended, but disk array based caching on other storage subsystems can be used. Be sure that these caches are non-volatile (battery backed) and available from the recovery node. The HATS H.A. lets you control how often you write to the disk storage, thus lowering the chance that there is information in the cache that could be lost if the system goes down. No other high availability solution offers this capability. Q. How difficult is it to install and configure HATS H.A.? A. It is very easy with the HATS H.A. easy-to-use documentation. HATS H.A. comes with standard installation routines that makes installation straight forward. Type 'install HA' and the process is automatic from that point on. One file needs to be completed on one server, then the information can be copied to each server thereafter (the config.file needs to be identical on each server in the H.A. system). Any editor can be used to fill out the config.file. Other vendors may require you to have them install the high availability, and charge you for their time. Q. How much control does the system administrator have over the events of the failover? A. The system administrator controls all takeover conditions. HATS H.A. can be configured to failover automatically, manually, or not at all. To failover manually, the system administrator can change the "states" of the HATS H.A. servers, and can lock the servers into specific states. All failover conditions may be controlled and defined by the HATS H.A. user and are not restricted to preset behaviors of the HATS H.A. product. Q. Does HATS H.A. support shared concurrent access to disk among nodes in a single cluster? A. No. Shared concurrent access requires a cluster-aware volume manager and application as well as some sort of distributed lock management capability. With HATS H.A., disks are physically connected to multiple cluster nodes, but logically defined as being owned by only one node at any one time. Change of disk ownership requires a HATS H.A. state change like that induced by a takeover or surrender; if the solutions is set up as a hot standby only then can you do this. Q. In a symmetrical failover environment, System 1 is running applications A, B & C and System 2 is running applications D & E. How does HATS H.A. start and stop service? A. When System 1 goes down, System 2 starts A, B & C. When System 1 comes back up, System 2 stops A, B, & C and System 1 starts A, B, & C. When System 2 goes down System 1 starts applications D & E, when system 2 comes back up System 1 stops D & E and System 2 starts them. At the time of installation, you can choose for this to occur automatically or under manual control. Q. When the failed cluster node is repaired and brought back on-line, will its former applications services, now running on the backup node, automatically fail back? A. By default, HATS H.A. locks the takeover server into the failover state (DUAL-SERVICES or TAKEOVER, depending upon whether you are running a symmetric or asymmetric configuration). When the failed node is brought back on-line, it requires manual intervention to fail the backup's services back to the primary. The system administrator may choose to configure HATS H.A. for automatic failback once the primary comes back on-line but it is not recommended. The HATS H.A. documentation describes how to do this. Q. What is an N-into-1 configuration and how does it work? A. An N-into-1 configuration involves more that two machines; at least two or more systems provide services (in our example, server-a and server-b ) and an additional one acts as the failover (hot-standby) server. If one or both production server(s) fails, the failover server is capable of impersonating the failed system(s). To setup a 2-into-1 configuration, HATS H.A. software is installed once on server-a and server-b, and twice on the standby server. One copy of HATS H.A. on the hot standby communicates with server-a ; the other copy communicates with server-b. HATS H.A. also allows 3-into-1 and 4-into-1 configurations through a similar mechanism. N-into-1 configurations are currently supported for all systems supported by HATS H.A.. N-into-1 configurations are not available through all vendors. Q. How many interfaces are required for the private heartbeat links in a 2-into-1 configuration? A. In all HATS H.A. N-into-1 configurations, each server will use the same single private network interfaces for the heartbeat links with backup through the public network. Q. How does failover occur into a 2-into-1 configuration? A. As defined in the earlier example, when System A fails, the interface A' on the standby server will impersonate interface A. Likewise, interface B' will impersonate B if System B fails. The standby system can impersonate both server-a and server-b simultaneously if they both happen to fail at the same time. The 2-into-1 configuration is not widely supported by all vendors of high availability solutions. Q. Can the hot-standby system provide other services, or does it have to strictly be an idle server waiting for a failover condition? A. A hot-standby system does not have to be idle; it can provide services that may be discontinued when a failover occurs. This is an asymmetrical configuration. The standby server may also provide a full set of applications, which could be failed over to the "primary" server; this is a symmetrical configuration. Having the capability to run applications on the hot-standby is not standard in most high availability software. Q. Is it possible to configure 2-into-1 symmetrical failover where C is the failover server for both A and B, but C itself has to be highly available, so A and B are used to back up C? A. This is possible, although it increases the complexity of system administration. To maintain the ease-of-use qualities of HATS H.A., we recommend using only asymmetrical configurations in N-into-1 environments. There is also a potential problem: if C fails, A and B will back it up. If A then fails, no server will be there to take over for it. Q. On a failover, does the hostname get configured or de-configured along with the IP address and MAC address? A. Most of our customers choose not to failover the hostname of the machine, since a majority of the applications are not dependent on the machine's hostname. Under most circumstances a machine has the name as its hostname and its primary interface name. For example, our primary server HATS H.A.-a has an IP address of 10.3.198.201. When you ping HATS H.A.-a, you are effectively pinging the IP address 10.3.198.201. On our second server, the system hostname might be HATS H.A.-b . If HATS H.A.-b is a standby server (as it would be in an asymmetric configuration), it does not need to be connected to the public network at startup. When HATS H.A.-a fails, HATS H.A.-b will go into takeover mode. It will impersonate the IP address of HATS H.A.-a, 10.3.198.201. When takeover completes you will have a system with hostname HATS H.A.-b , but its interface name will correspond to HATS H.A.-a. Q. Is there any advantage to swapping the IP and MAC addresses between the primary and secondary workstation on failover? A. Yes. Using the example above, by impersonating IP and MAC addresses in a failover, users can connect to 10.3.198.201 even though the HATS H.A.-a server is down. This is especially important if HATS H.A.-a is providing NFS services, since you don't need to un-mount NFS file systems on the client machines and re-mount them every time a failover occurs. Q. Is it possible to use the standby systems for normal-computing tasks unrelated to the failover configuration? A. Yes, the secondary server does not necessarily have to be idle while it is waiting for the primary to fail, it can be doing something useful. Q. What happens if a failover is initiated from the primary to the secondary and the secondary is unreachable? A. If you want to issue failover from the primary server, which can be done by the system administrator using H.A. mainmenu or the graphical interface, you can change the server state from ONLINE_PRIMARY to IDLE. But since the secondary is unreachable (i.e. no communications from secondary via the heartbeats) the primary will not be able to transfer services to the secondary. Q. Is there a limit to the number of services that can be monitored? A. Yes. Up to 16 jobs can run simultaneously, whereas some other high availability software only allows one application. Q. How do I determine the software limitations and configuration requirements for HATS H.A.? A. All software limitations are listed in the manual, including the requirement for dual heartbeat links, multiple interfaces, etc. HATS H.A. has fewer limitations than any other high availability vendors. Q. Is there a difference between the primary and secondary heartbeat? A. Both heartbeats are crucial in your H.A. configuration since information is exchanged across both heartbeats. Dual heartbeats give us redundancy and high availability since we still can have communication between servers even if one heartbeat is down. It also means that a single NIC failure in the heartbeat link will not result in a cluster failure, although the event is logged. Q. Can I use my regular LAN Ethernet connection (the public network) for the secondary heartbeat? A. Yes, as described previously. Not all high availability solutions allow this method – the alternatives can cause slower response time and are sometimes less reliable. NETWORKS Q. During failover, how are the IP and hardware Ethernet addresses of the primary system assumed by the secondary system? A. During failover, the secondary machine will execute a HATS H.A. transition program, the TAKEOVER program, to impersonate the IP and MAC (Ethernet) addresses of the primary machine. Standby interfaces remain idle otherwise. Q. How many network interfaces do I need to support a HATS H.A. configuration? A. At a minimum, an asymmetric will require 1 network interface, however, this is a minimal configuration. 2-network interfaces are preferred. Q. Can static routing be used? A. You can use static routing by adding route entries into the route table manually, or you can set a default route to a gateway machine which contains routing information to other hosts or networks. Q. How are ARP tables on client nodes effected by the takeover of the failed server's IP and MAC addresses? A. The client node's ARP table should remain the same, since the takeover server will assume the IP and Ethernet addresses during takeover; the takeover interface's address values will be consistent with ARP tables on all clients. Q. Are MAC layer addresses required to be unique worldwide? A. No, MAC layer addresses are only required to be unique in their Ethernet segment. Q. What happens if two cards with the same MAC address appear on a network? A. The effect of this occurrence on an Ethernet is unpredictable. Packets may be routed to the wrong host and be lost. On FDDI and Token Ring networks this is more serious since it will lock up the entire ring, and may require that all nodes on the ring be rebooted. Q. Does HATS H.A. support failovers on FDDI and Token Ring networks? A. HATS H.A. supports the FDDI and Token Ring networks the same as it supports standard Ethernet networks. Q. How can MAC addresses be duplicated on a ring, if one of the servers is not in service? A. When a node is halted, there is still power coming into the Token Ring interface. In this state, the interface card retains its MAC address and still responds to all network requests for that MAC address. It does not, however, pass on any information to the driver. When this node is halted and is part of an H.A. setup, the standby node will detect that the primary node is down, due to loss of heartbeat, and issue a failover. Part of the normal failover procedure is to impersonate the IP and MAC address of the primary, therefore both the primary and standby servers will be responding to all network activity for their common MAC address. Ethernet handles this situation in a graceful manner while the Token Ring networks simply shut down. Q. How does H.A. Technical Solutions, LLC recommend resolving potential Token Ring network issues to enable failover? A. A possible solution for resolving this issue is to assign each interface card a unique MAC. The failover script should be modified to impersonate only the IP address, not the MAC address. Depending upon the implementation, the ARP table on all nodes on the ring may have to be updated. If this is required, this solutions may only be feasible for small networks. Another possible solution for the Token Ring issue is to install an intelligent Token Ring hub. When a failover is imminent, the standby server connects to the hub command interface and disables the port connected to the primary server. This allows the standby server to impersonate the MAC address in an environment that prevents duplicate MAC addresses from appearing on a Token Ring network. This is the preferable configuration. To summarize, HATS H.A. can be made to support failover on Token Ring networks provided that the configuration addresses the protocols' sensitivity to duplicate MAC addresses on the ring and other hub-specific issues. In all cases where Token Ring failover is required, the configuration is a consulting special. Q. Is there a requirement to use the on-board Sun le0 Ethernet interface as the primary heartbeat link? A. No, but this is shown because it is more cost-effective to use le0 instead of buying a separate controller, and because it saves precious network connection slots. As long as the two heartbeat links run into network interfaces that fail independently (i.e. a single board failure will not bring them both down) that meets the requirement. Please note, however, that in asymmetric configurations the recovery node (standby) must use le0 for the heartbeat network since le0 must always be functional in a Sun server. This is not the case with other platforms. Q. Is network failover supported on ATM networks? A. Not as a part of the standard HATS H.A. product, but LAN emulation across ATM networks can be delivered as a consulting special. DISK ISSUES Q. How is the location and ownership of the devices and file systems arranged between two systems in a normal and in a failover mode? A. The devices and file systems should be identical on both systems for the same job (i.e., the file system mount point, device name and major and minor numbers should be the same). Q. Does HATS H.A. require any additional software or device drivers to work with third party disk drives? A. HATS H.A. uses standard operating system utilities to configure devices, resulting in no requirements for additional software, drivers or kernel modifications. As long as the devices follow standard OS device naming conventions and are transparent to file system utilities (such as mount) they are usable to HATS H.A.. Normally, disks shared between the HATS H.A. servers and used with HATS H.A. must be connected directly to both HATS H.A. servers. To work correctly in such a dual-connected mode, the system's SCSI drivers and hardware have to support "multi-initiator" capabilities. This feature allows multiple systems to simultaneously issue instructions to a common SCSI or other peripherals bus. For SCSI-based disk subsystems, users also need to change the SCSI initiator ID on one of the servers to a value other than 7, the default. Most modern Unix disk drivers support both multi-initiator and programmable SCSI initiator ID capabilities. Q. Are CLARiiON disk arrays supported in HATS H.A. configurations? A. The CLARiiON disk subsystems have a storage processor architecture that does not require you to connect all disks directly to both servers. In CLARiiON-based clusters, both systems are not actually on the same SCSI bus. The systems access the SCSI disks through a storage processor in the CLARiiON box. The CLARiiON storage processors transfer disk control to the backup system via the CLARiiON "trespass" facility, which is called from the HATS H.A., scripts when using HATS H.A. with CLARiiON disk arrays. Dual-bus configurations (one cluster node attached to each storage processor) are supported but split-bus configurations (both nodes attached to both storage processors simultaneously) are not. CLARiiON also provides an agent, which monitors the status of the CLARiiON and makes this information available to HATS H.A.. Q. Why do I need to change the SCSI initiator ID on one of the servers? A. SCSI initiator ID's are like network addresses; each device needs one that is unique. SCSI cards normally have the default SCSI initiator-ID value of 7. If both servers have SCSI initiator-ID of 7 and are connected to the same (shared) devices, the SCSI bus will become confused. Therefore, to set up shared disks on a single SCSI chain, you need to change the SCSI initiator-ID on one of the systems. On Sun systems you can changed this default by using the eeprom command, and you only need to change it on one controller card. On other systems, all shared SCSI controllers may have to be changed independently. Q. How do I know what SCSI initiator-ID value to use? A. The SCSI initiator-ID value has to be unique on the SCSI bus. Usually it is set to 5 for the second system on regular SCSI interfaces because many systems don't have a device set to target 5. The following is a sample SCSI device map:
0-Disk 1, 1-Disk 2, 2-Disk 3, 3-Disk 4, 4-Tape 1, 5-Second controller, 6-CD-ROM, Q. How does the secondary system access the file systems owned by the primary system after a failover? A. The file systems typically reside on the shared disks on a SCSI bus that is connected to both systems. By configuring the HATS H.A. transition scripts, you can define the file systems to be mounted by the failover system. When the primary system fails, the secondary system will invoke the transition script to make the same file systems available. If you use a logical volume manager (such as the SPARC storage Array Volume Manager or the Volume Manager from Programmed Logic), which is highly recommended for better overall availability, you will need to import the disk groups before mounting and NFS-sharing the file systems. Q. Can Sun's Solstice DiskSuite (SDS) product be used on HATS H.A.-enabled Sun servers? A. Yes, but the use of Solstice DiskSuite will impose limitations in performance, number of supported disk groups, and ability to use a high performance journalizing file system. Use of a DiskSuite release prior to 4.0 limits you to asymmetric configurations only, as it supports only one disk group per physical host. Use of DiskSuite is a supported configuration, but use of the Volume Manager from Programmed Logic is recommended since it suffers from none of these limitations. Q. What kind of RAID array problems will cause failover to occur? A. Most RAID array problems are solved internally to the RAID subsystem, and are not noticed or monitored by any outside systems. HATS H.A. runs as an application and is independent of the kernel and all drivers. Therefore, as it is shipped, it does not detect kernel-specific or driver-specific failures (like a RAID disk driver problem). However, most RAID vendors provide utilities for checking the status of the array and detect array failures. The user can write a simple HATS H.A. agent to monitor the status of their RAID array. In this HATS H.A. agent, the user would define the conditions for RAID failure. Note that all conditions do not necessarily result in a failover condition. The user defines the exact failure conditions and remedies. Many users have their agents inform them of critical conditions and then manually failover the system as required. Some RAID vendors, like CLARiiON, provide their own HATS H.A. agents to monitor their storage array status. Q. In normal operation, can both the primary and secondary systems read and write to the RAID array? A. Yes, but not simultaneously to the same disk or disk group. You can partition your storage into two or more groups of disks; the primary system can access one disk group while the secondary owns the other. Shared concurrent access to a disk group is not supported – a takeover must occur to move disk group ownership form one node to another. Q. In failover, what happens on the secondary system to allow it to use the RAID array? A. This depends on the RAID vendor. If the RAID is multi-initiated, the secondary system can simply mount the devices in the RAID array. If it is dual-ported or has a split bus, you may need to run utility programs provided by the vendor to migrate the devices from one host's view to the other. If you use the SPARC storage, Volume Manager from Programmed Logic, or Solstice DiskSuite, you will need to import/reserve the disk groups/disk sets before mounting and NFS-sharing the volumes. NETWORK FILE SYSTEM (NFS) SERVER ISSUES Q. Can I failover between non-identical NFS servers? A. Yes. However, the NFS file handle presented for each file must be the same on each server to prevent "stale NFS file handle" errors. This automatically corrects for HP UX and AIX servers. However, in Sun SPARC system environments, the file handle includes the major and minor device numbers of the file system's underlying device on the server. Different Sun architecture families (sun4c, sun4e, etc.) use different major numbers for the same devices. Add-on product drivers may differ as well. The SPARC storage or Volume Manager from Programmed Logic present a simplified method for getting major and minor numbers to be consistent on both systems; this could otherwise only be achieved with manual effort and scrupulously identical system configurations. Q. What is the impact of a takeover on an NFS client? A. The NFS clients will see an "NFS server x down" message when the primary server is down and the secondary server is in transition to failover. Once failover has completed, the clients will get an "NFS server x OK" message. Solaris clients re-connect quickly to the recovery node, but HP UX and AIX clients may take longer to perform the reconnect. In all cases, because NFS uses stateless connections between the clients and the server, clients will see only a slight service delay rather than a disruption. Q. In normal operation can both systems actively provide NFS services using the devices in the shared disks? A. In a symmetric configuration, provided that Node A and Node B are accessing separately owned disk groups, both nodes can share access to the shared disks. Shared concurrent access by both nodes to a single disk group is not supported. In asymmetric configurations normally only one server is accessing the shared disks at a time. Load balancing across cluster nodes is not supported in either symmetric or asymmetric configurations. Q. How is the primary system's NFS service established on the secondary system during failover? A. The secondary system starts up the NFS daemons if necessary, then mounts and NFS shares the shared disks which have been identified by the user in the TAKEOVER.SCRIPT. These disks are then available to the NFS clients. Q. Does HATS H.A. work well with NFS for PC and Macintosh NFS clients? A. Yes, HATS H.A. enables transparent NFS failover for all NFS clients. Q. How safe are client NFS packets during failover? Would data be lost if it was received and acknowledged by the server but the system crashed before it was cleanly written to disk? If the data is received but the system crashes even before acknowledgement, would the data be re-transmitted and re-written? A. HATS H.A. has no way of detecting these two conditions. The first condition could happen if the system had caching (including Prestoserve) enabled, or was using asynchronous NFS. These should not be used in H.A. configurations. If these conditions are serious issues for you, you may want to investigate the purchase of a journalizing file system, like the File System for your NFS servers which could reduce – but not eliminate – the likelihood of these failures. Q. What happens to packets that are sent to the primary system during failover before the secondary system has assumed the primary's NFS workload? Will they time-out and be resent by the client NFS? When failover is complete, will the secondary system respond to the packets? A. In most cases the primary will go down before it can send an acknowledgement to the client, so the client will continue to resend the packet as the secondary machine takes over. Once the takeover is complete, the packet will arrive at the secondary machine and will be serviced and acknowledged. Q. How long does NFS failover take? A. The length of time for the NFS failure is dependent upon the number of disks and the amount of data and service that you have. HATS H.A. has to check the consistency of all shared file systems and shut down services before failing over. On our test systems, with 400MB of data and no journalizing file system, it takes approximately 4 seconds to failover. The longest process in the failover is the check of the disk systems. If using a journalizing file system, the check can take as little as a couple of seconds. When we measured failover using a 600GB NPI RAID subsystem with 18 disks in the array and file system journalizing, the entire failover took only 20 seconds. Q. What happens during normal operation, failover operation, and after failover operation in an NFS configuration? A. Assume a configuration with two servers and a shared disk array; server-a and server-b are in a symmetrical configuration. The array contains drives with the device identifiers clt0d0s0 through clt3d0s0 and c2t0d0s0 through c2t3d0s0. During normal operation: • server-a mounts clt0d0s0-clt3d0s0 as/export/{clt0,clt1,clt2,clt3} • server-b mounts c2t0d0s0-c2t3d0s0 as/export/{c2t0,c2t1,c2t2,c2t3} During failover of server-a: • if manually initiated, server-a disables NFS sharing and unmounts/export/{clt0,clt1,clt2,clt3} • if automatically initiated by server-a failure or reboot, it will not be possible to disable sharing and unmount the disks. After failover: • server-b mounts and NFS-shares clt0d0s0-clt3d0s0 as/export/{clt0,clt1,clt2,clt3} • server-b will then control two sets of disks – c1 and c2. Q. Do all applications failover as transparently as NFS? A. All applications which use stateless connections between client and server, and can gracefully retry or recover in the event of a back end service failure should experience only slight delays rather than service disruption as a result of a takeover. If applications use a stateful connection or will not automatically attempt to re-connect to a service that appears to be unavailable, then clients will experience a service disruption and typically will need to log back into the back end service. HATS H.A. does have a special version available that can failover software applications that are specifically registered to a single host ID. Q. Do I have to utilize the NFS failover capabilities of HATS H.A.? A. No. Most capabilities of HATS H.A. are user-configurable, including transparent NFS failovers and Ethernet impersonation. All features are controlled by the user and by the config.ha file in HATS H.A.. DATABASE ISSUES Q. How do I failover an Oracle, Sybase, SQL server or Informix database? A. When a server fails or the agent requests a failover, the database will fail over to the takeover server. To support DBMS failover, servers are generally configured so that shared disks contain the database data and software. These disks should have the same mount points (if used for file systems) and device names on both machines. When a failover occurs, the service is restarted on the secondary machine, using the database data and executables on the shared disk. H.A. Technical Solutions, LLC can provide agents for monitoring each of these databases on UNIX servers. Q. How can we configure HATS H.A. to reduce data loss for database data in memory upon CPU failure? A. HATS H.A. does not monitor data in memory. You should attempt to configure the database to leave as little data as possible in buffers, which are not yet written to disk. For instance, Sybase uses a daemon that monitors the pages in memory and occasionally writes them to disk. You can set the time to a small value so that writing to the disk is done frequently. Q. Which releases of common databases have been used with HATS H.A.? A. HATS H.A. customers have successfully failed over Oracle 7.x.x and 8.x.x, Sybase 4.9, 10 and 11, and Informix OnLine 7 DBMS servers in the past. Please contact your reseller for more information and reference sites. Q. What does a database client see when a failover occurs? A. This is dependent on whether the client/server database environment is implemented as stateful or stateless. Database systems like Oracle, Sybase and Informix tend to be stateful, since client programs must connect and authenticate to the server. If a database application were implemented as stateless, then when failover occurred it would appear to the end user much like the failover of NFS to another server (i.e., there would be a pause of a few seconds to a few minutes until the new server completely failed the services over, then processing would continue without user intervention). However, database applications are usually written as one long-lived session to a server. This session-orientation introduces 'statefulness' into database application usage. Therefore when a normal database server fails, the end user application dies. From a users' point of view, this forces a re-launching of the application. Although this helps databases perform better, it is not an optimum design for high availability. Database applications can be written to appear stateless to the user. Each autonomous set of requests to the SQL server should be constructed as a single transaction, and should be checked for successful completion. If the request does not complete successfully due to server failure, the application would try to re-connect to the server. Once the re-connection is complete, the application would re-transmit the SQL requests. From a user's perspective, the transaction would experience slow response, but would successfully complete. There is additional session management and checking overhead for the programmer in this approach, but it is more user-friendly in the event of a failure. Unfortunately, many database applications written to date do not implement this strategy, and modifying an existing application to use this strategy is not normally an easy chore. Database APIs currently have no hooks to make this a simple process, although some APIs are now providing automatic reconnection hooks. Q. What happens if the server crashes when a request is being processed? A. All modern, industrial-strength databases use journals or logs to record all database activity. The transaction log is written prior to related database and table and index updates and marks an activity as complete once the entire database table and index records have been written. When a database server process crashes and then gets re-started (either on the original node or the recovery node), the database server "rolls forward" transactions in the log starting from the last known consistent state, to ensure the completion of transactions. Entries for incomplete transactions are not applied to the database. This process leaves the database in a consistent state. Q. Are all related database operations saved in the database transaction log? A. Not necessarily. The programmer must define transactions properly, including all related database operations, for proper post-crash consistency. For example: User sends "Sold a baseball card" transaction: 1) Send transaction 'start software'. These disks should have the same mount points (if used for file systems) and device names on both machines. When a failover occurs, the service is restarted on the secondary machine, using the database data and executables on the shared disk. 2) Delete: YEAR=1960 CARD=Mays, Willie APPRAISED_VALUE=$150 From BaseballCardInventory 3) Add: YEAR=1960 CARD=Mays, Willie SELLING_PRICE=$170 to BaseballCardRevenue 4) Send transaction. If the database crashes at any time before the completion of step 4, the takeover server will "roll over" (undo) all of the steps. Depending on the database server and application program, the client could receive a message saying that the transaction was not completed and direct the client to re-submit this request. It is prudent for a high availability implement to ask database programmers the following questions (database programmers sometimes overlook crash and recovery issues): • Are all database transactions properly constructed for correctness across a recovery? (for example, a "send transaction complete" message appearing after step 2 above would be inappropriate.) • Do client applications handle broken connections? They do not have to reconnect, but they must report to the user that there was a broken connection and that their transaction did not complete. H.A. Technical Solutions, LLC can provide agents for monitoring each of these databases on UNIX servers, and has developed an Oracle and Sybase agent. DATA INTEGRITY ISSUES Q. What happens if power to the disk is turned off while a write is in progress? A. For a database, this would be handled as part of the transaction log based recovery. For a file system, the check program will run on power-up and reboot. If the sector being written contains structural (metadata) information, the check program will try to repair it. If the sector cannot be repaired the file or directory will be removed. (the likelihood can often be reduced by use of a journalizing file system. If the interrupted write was a data block, the check program will probably be able to detect the incomplete write, and data corruption may result. File systems are at a disadvantage to databases with regard to data integrity, since databases generally log all data transactions. Q. What should I do if my disk crashes? A. If you have a disk configuration with no redundancy (mirroring or parity), you must replace the drive and restore the data from backup and incremental logs. The system or applications affected will be down until the disk is fully restored. Data loss is possible if backups are not up-to-date or have errors. If you have a RAID subsystem (implemented with RAID controller hardware or with software that simulates RAID) with redundant storage and hot-swap and/or hot spare capability, you can merely swap out the bad disk and, if necessary, cause the data to be rebuilt onto the replacement drive. No loss of data or access should result. |
 |
 |
 |
 |
 |
 |
Look for $avings on surplus and/or re-furbished Sun Storage Products, Servers, and Workstations!