13
High Availability

This chapter provides information about high availability within Oracle9i Application Server.

It contains the following topics:

• Introduction to High Availability

• Oracle Process Management and Notification

• Configuring the Oracle Process Manager

• Configuring the Oracle Notification System

• Failure Scenarios

Introduction to High Availability

The availability of a system or any component in that system is defined by the percentage of time that it works normally. A system works normally when it meets its correctness and performance specifications. For example, a system that works normally for twelve hours per day is 50% available. A system that has 99% availability is down 3.65 days per year on average. System administrators can expect critical systems to have 99.99% or even 99.999% availability. This means that the systems experience as little as four to five minutes of downtime per year.

Availability may not be constant over time. For example, availability may be higher during the daytime when most transactions occur, and lower during the night and on weekends. However, because the Internet provides a global set of users, it is a common requirement that systems are always available.

Redundant components can improve availability, but only if a spare component takes over immediately for a failed component. If it takes ten minutes to detect a component failure and twenty additional minutes to start the spare component, then the system experiences a 50% reduction in availability for that hour of service.

Oracle9iAS is designed to provide maximum system availability during many types of hardware and software failures.

Benefits of High Availability

Oracle9i Application Server keeps your system available by providing the following benefits:

• Fault Tolerance: Oracle9iAS has a flexible deployment model so you can deploy an architecture that exposes no single point of failure. This means that despite the failure of any component in the system, the application server is tolerant to failures and continues to function and service client requests.

• Death Detection and Restart: To have maximum availability, when a server process fails, the system should take appropriate action by cleaning up memory and restarting the failed processes. The management processes in each Oracle9iAS instance monitor the HTTP server and OC4J processes and restart failed processes as required.

• Failover: Failover hides system failure from users. Failures do occur, but clients should not perceive any loss of service as their requests are failed-over/re-routed to another redundant service that takes over for the failed one.

  At the mod_oc4j-OC4J level, after the failure of a stateless service, the application server routes requests to alternate instances of the service in a similar fashion to connection rerouting. After a failure of a stateful service, the application server reroutes the request to an alternate instance to which the state has been replicated.

  At application server instance level, you can use a hardware load balancer, Oracle9iAS Web Cache, or a DNS round robin strategy to load balance requests across redundant application server instances. This allows your Web site to continue functioning even if one of the application server instances goes down.

  See Also: For instructions on configuring Oracle9iAS Web Cache as a load balancer to route requests, refer to Oracle9iAS Web Cache Administration and Deployment Guide.

• Connection Rerouting: Rerouting occurs when a process or node servicing a request fails while a session is still executing. To address this failure, client requests to the session are transparently rerouted to alternate application server instances in the cluster.

High Availability with Oracle9iAS Web Cache

Oracle9iAS Web Cache supports the following content-aware load balancing and failover detection features:

• Load Balancing: Most Web sites are served by multiple origin servers running on multiple computers that share the load of HTTP and HTTPS requests. This feature enables Web sites to be built with a collection of servers for better scalability and reliability. Oracle9iAS Web Cache sends requests to the origin server with the most available load.

• Failover: After a specified number of continuous request failures, Oracle9iAS Web Cache considers an origin server as failed. When an origin server fails, Oracle9iAS Web Cache automatically distributes the load over the remaining origin servers based on the available load. Oracle9iAS Web Cache polls the failed origin server for its current up/down status until it is back online. When the failed server returns to operation, Oracle9iAS Web Cache includes its available load to load balance requests.

• Surge Protection: To prevent an overload of requests on the origin servers, Oracle9iAS Web Cache provides a surge protection feature that enables you to set a limit on the number of concurrent requests to the origin servers. When the limit is reached, subsequent requests are queued to wait up to a maximum amount of time. If the maximum wait time is exceeded, Oracle9iAS Web Cache rejects the request and serves a site busy apology page to the Web browser that initiated the request.

• Cache Clusters: With cache clusters, cache content is replicated among multiple Oracle9iAS Web Cache servers. If a cluster member fails, other members of the cache cluster detect the failure and take over ownership of the cacheable content of the failed cluster member. Because requested cacheable documents are likely to be stored in the cache of surviving cluster members, fewer requests for cacheable documents need to be routed to the Web server.

  See Also: Chapter 14, "Application Server Clustering"

• Auto-Restart: To handle cache failures, Oracle9iAS Web Cache provides an auto-restart process that automatically restarts the cache.

• Dynamically Generated Content: By caching dynamically-generated content, Oracle9iAS Web Cache adds a buffer between the user load and the origin server. As a result, if you configure Oracle9iAS Web Cache to cache and assemble some of the dynamically generated content, then Oracle9iAS Web Cache takes the burden off the origin servers and databases.

  For instance, certain viruses or attacks attempt to bring down sites by flooding servers with malicious requests, some of which are intended to exploit weaknesses in the application Web server. Because attacks are typically distributed, it is often difficult to block all the IPs that are sending the requests. However, by caching the responses to the initial requests, Oracle9iAS Web Cache can shield the backend servers from having to respond to subsequent requests of this nature. In other words, caching provides some level of denial-of-service protection.

Architecture

Figure 13-1 shows an example of high availability architecture where an Oracle9iAS instance can contain one Oracle HTTP Server instance and zero or more OC4J processes. Here, Oracle9iAS Web Cache is used as a load balancer to route requests. OPMN within the Oracle9iAS instance environment monitors the Oracle HTTP Server and OC4J processes. If either Oracle HTTP Server or OC4J fail, the OPMN restarts the failed process. This enables a maximum level of availability of the components within an Oracle9iAS instance.

Figure 13-1 High Availability Architecture

Text description of the illustration opmn.gif

The mod_oc4j Module

The mod_oc4j module routes requests from the Oracle HTTP Server to Oracle Containers for J2EE (OC4J). It works in conjunction with OPMN to keep its routing table updated so that it load balances across only live OC4J processes in an OC4J instance. When a request comes in, mod_oc4j selects a process from its shared memory table. If a session is created with this request, related requests are routed to the same process. Otherwise, a new process is selected based on the load balancing algorithm. To allow fault tolerance, this module supports OC4J instances and islands. To handle failover with stateless requests, it tries to select an alternative OC4J process within the same island and routes the request to that process. If all the OC4J processes in the instance have been tried and none of them are successful, mod_oc4j returns request failure. For session enabled requests, the module tries to failover to another OC4J process within the same island as the original process that serviced the request. Again, if all the OC4J processes in the island are unable to service the request, mod_oc4j returns request failure.

See Also: Oracle HTTP Server Administration Guide

Oracle Process Management and Notification

Oracle Process Management and Notification Service (OPMN) manages the processes within an Oracle9iAS instance and also channels all notifications from different components instances to all interested in receiving them.

OPMN consists of the following two components:

• Oracle Process Manager

• Oracle Notification System

Oracle Process Manager

Oracle Process Manager (PM) is the centralized process management mechanism in Oracle9iAS and is used to manage all Oracle HTTP Server and OC4J related processes. It starts, stops, restarts, and detects death of these processes. When these processes are configured to start up, the characteristics of each set is specified in a configuration file called opmn.xml. The Oracle Enterprise Manager Web site also uses PM to manage processes.

The PM starts and then waits for a command to start specific or all processes. Similarly, at shutdown, the PM receives a request to stop one or more processes, or all processes and itself as specified by the HTTP request parameters.

Oracle Notification System

Oracle Notification System (ONS) is the transport mechanism for failure, recovery, startup, and other related notifications between components in Oracle9iAS. It operates according to a subscriber-publisher model, wherein any component that wishes to receive a notification of a certain type subscribes to the ONS. When such a notification is published, ONS sends it to all subscribers. PM and ONS exist within the same process, and comprise Oracle Process Management and Notification. OPMN is monitored by a shadow process that restarted upon request or after a catastrophic failure

Configuring the Oracle Process Manager

The Oracle Process Manager uses a configuration file, opmn.xml, that contains information about all the processes it manages. This information includes parameters to start a process as well as metadata for bookkeeping. Each process type that OPMN manages has a corresponding element defined in opmn.xml. This section discusses the following elements:

• process-manager

• ohs

• oc4j

• custom

process-manager

This is the topmost element for the process manager. The administrative process defines a single process manager. A process-manager element contains elements of types:

• ohs,oc4j,custom: Specify managed processes.

• log-file: Specifies the location of the process manager log file. It is optional. The default is ORACLE_HOME/opmn/logs/ipm.log.

  The log-file element can have one attribute: level (optional). The level attribute can be one of 1=FATAL, 2=ERROR, 3=WARN, 4=NOTIFY, 5=DEBUG, 6=VERBOSE. The default is 3 (warning, error, and fatal messages are logged).

ohs

This element defines an Oracle HTTP Server process and has the following attributes:

• gid: A unique identifier for the process. It is optional. The default is "HTTP Server".

• maxRetry: Specifies the number of times PM should check the Oracle HTTP Server process before declaring it dead, and the number of times to try to start an Oracle HTTP Server process before giving up. It is optional. The default value is 3.

• timeout: Specifies the maximum time needed to wait for Oracle HTTP Server to completely start. It is optional. The default value is dependent on system load, but is at least 300 seconds.

The ohs element can contain optional elements of the following types:

config-file: Specifies the server configuration file. It is optional. The default value is ORACLE_HOME/Apache/Apache/conf/httpd.conf.

The config-file element can have these attributes:

• path: The path to the configuration file. If you specify a relative path, then the current Oracle HTTP Server working directory is prepended to the path you specify.

• start-mode: Specifies the mode in which the server starts: ssl or non-ssl. It is optional. The default is non-ssl.

  The start-mode element can have the mode attribute. Specify ssl to start Oracle HTTP Server in SSL mode.

oc4j

This element defines an OC4J process group and has the following attributes:

• gid: Unique identifier for the OC4J process group. It is optional. The default is home.

• numProcs: Number of OC4J processes to start in this OC4J process. It is optional. This value is ignored if the island element is specified for this OC4J process. The default is 1.

• maxRetry: Number of times PM checks OC4J before declaring it dead, and the number of times to try to start it before giving up. It is optional. The default is 3.

• instanceName: Defines a group of OC4J processes, or islands, that service the same set of applications. It is optional. The default is home.

• timeout: Specifies the maximum time needed to wait for Oracle HTTP Server to completely start. It is optional. Default value is dependent on system load, but is at least 300 seconds.

The oc4j element can contain elements of the following types:

• config-file: Typically, specifies the server.xml file if it is in a non-default location. It is required. If a relative path is specified, the current OC4J working directory is prepended to the path.

• java-bin: Specifies the path to the Java executable that starts OC4J. It is optional. The default is ORACLE_HOME/jdk/bin/java.

• java-option: Specifies JVM options required by the OC4J command line. It is optional.

• oc4j-option: Specifies OC4J options required by the OC4J command line. It is optional.

• port: Specifies the individual port or range of ports on which OC4J listens. It is optional if numProcs has not been specified or is equal to 1, required if numProcs is greater than 1.

  The port element requires the following attributes:

  AJP: AJP ports on which the OC4J process listens. If the value is set to 0, then OC4J picks the AJP port.

  JMS: JMS ports on which the OC4J process listens.

  RMI: RMI ports on which the OC4J process listens.

  For example:

  <port ajp="8000,8010-8020,8003" jms="7030-7050,7780-7790" 
  rmi="9000,9001,9002,9003,9004" />
  

• island: A set of JVMs sharing session state within an OC4J process. For example, if a session is being handled by OC4J process 1 in island 1 of specifying this element and process 1 dies, mod_oc4j fails the session over to OC4J process 2, which is also a member of island 1.

  To start multiple islands within an OC4J process, you must specify an island element for each island. In this case the numProcs attribute, if specified, for OC4J is ignored. If you want to start only one island within an OC4J process, you can specify numProcs either as an OC4J attribute or as an island attribute.

  The island element requires the following attributes:

  • id: Unique identifier for the island.

  • numProcs: Number of processes to start for this island and OC4J process.

  For example:

  <island id="island1" numProcs="2"/>
  <island id="island2" numProcs="3"/>
  

The environment element can have the following element type:

prop: Represents one environment name value pair for the custom process.

The prop element can have the following attributes:

• name: Name of the environment name value pair

• value: Value of the environment name value pair

Example OC4J Configurations:

This section contains examples of OC4J configurations.

Start Four JVMs in the Same Island in One OC4J Process

<oc4j instanceName="oc4jInstance1">
  <config-file path="/oracle/j2ee/home/config/server.xml" />
  <port ajp="6666-6670" rmi="6686-6690" jms="6786-6790"/>
  <island id="island" numProcs="4"/>
</oc4j>

or

<oc4j instanceName="oc4jInstance1" numProcs="4">
  <config-file path="/oracle/j2ee/home/config/server.xml" />
    <port ajp="6666-6670" rmi="6686-6690" jms="6786-6790"/>
</oc4j>

These two configurations achieve the same results except that the second one uses default_island as its OC4J island ID.

Start Four JVMs (Two in Island 1, Two in Island 2) in One OC4J Process

<
<oc4j instanceName="oc4jInstance1">
  <config-file path=/oracle/j2ee/home/config/server.xml />
  <port ajp="6666-6670" rmi="6686-6690" jms="6786-6790"/>
  <island id="island1" numProcs="2"/>
  <island id="island2" numProcs="2"/>
</oc4j>

or

<oc4j instanceName="oc4jInstance1" numProcs="4">
  <config-file path=/oracle/j2ee/home/config/server.xml />
  <port ajp="6666-6670" rmi="6686-6690" jms="6786-6790"/>
  <island id="island1" numProcs="2"/>
  <island id="island2" numProcs="2"/>
</oc4j>

These two configurations achieve the same result. Since the island element is specified, the OC4J attribute numProcs is ignored.

custom

This element defines a generic process group and has the following attributes:

• gid: Unique identifier for this custom process group. It is optional. The default is default_gid1.

• num_of_proc: Number of processes to start in this group. It is optional. The default is 1.

• maxRetry: Specifies the number of times to try to start the process before giving up. It is optional. The default is 3.

The custom element can contain elements of type:

• start: Specifies the absolute path to the command used to start the process It is required.

• environment: Specifies the environment required by the process when it is spawned.

The environment element can have one of the following element types:

prop: Represents one environment name value pair for the custom process.

The prop element can have the following attributes:

• name: Name of the environment name value pair

• value: Value of the environment name value pair

Sample Configurations for Oracle Process Manager

This section provides some sample configurations for the Oracle Process Manager. Electronic copies of these are available at

http://otn.oracle.com

One Oracle HTTP Server Process, One OC4J Process with Defaults

This configuration starts one Oracle HTTP Server and one OC4J process with default values.

<notification-server>
  <port local="6001" 
        remote="6002"> 
  </port> 
  <log-file path="/private/my_directory/tmp/opmn_logs/ons.log" 
            level="5"> 
  </log-file>
</notification-server>
<process-manager> 
  <ohs/>
  <oc4j> 
    <config-file path=ORACLE_HOME/j2ee/home/config/server.xml /> 
  </oc4j> 
  <log-file path="/private/my_directory/tmp/opmn_logs/ipm.log" 
           level="4"> 
  </log-file> 
</process-manager> 

One Oracle HTTP Server Process, Two OC4J Processes, and One Generic Process with User-Specified Values

This configuration starts one Oracle HTTP Server process, two OC4J processes, and one generic process with several user-specified values.

<process-manager> 
  <ohs gid="a1" maxRetry="3"> 
    <config-file path="/my_directory/conf/httpd.conf"/> 
  </ohs> 
  <oc4j  gid="o1" numProcs="2"  maxRetry="4" clusterID="myClusterA"
                                                 islandID="myIslandA"> 
 <config-file path="/my_directory/conf/oc4j.xml"/> 
     <port http="8401" rmi="8600" jms="6577" ajp="3456"/>
  </oc4j> 
  <custom gid="g1" num_of_proc = "1">
    <start path="/my_directory/bin/exec1"/> 
      <environment> 
      <prop name="PATH" value="/my_directory/ias/lib"/> 
      <prop name="CLASSPATH" value="/my_directory/ias/bin" 
    </environment>
  </general> 
</process-manager> 

One Oracle HTTP Server Process, Two OC4J Processes with User-Specified Values

This configuration starts one Oracle HTTP Server and two OC4J processes with some user-specified values.

<notification-server> 
  <port local="6001"
        remote="6002"> 
  </port> 
    <log-file path="/private/my_directory/tmp/opmn_logs/ons.log" 
              level="5"> 
    </log-file> 
</notification-server> 
  <process-manager>
  
<!-- Start one ohs process with a process group ID of a1, with a config file in 
a non-default directory location -- > 

  <ohs gid="a1" maxRetry="3"> 
            <config-file path="Apache/Apache/my_conf/httpd.conf"/>
  </ohs> 

<!-- Start two oc4j processes with a process group ID of o1, with a instanceName 
of myClusterA (This should be the worker specified                          in 
the mod_oc4j config file), and an islandID of myIslandA. Since we are starting 
two processes we have to specify the port range used when starting the 
processes. So OC4J process1 starts with ajp port 8010, jms port 8020                          
and rmi port 8030. The second OC4J process uses ajp port 8011, jms port 8021, 
and rmi port 8031. --> 

  <oc4j gid="o1" numProcs="2" instanceName="myInstanceA"
                                       islandID="myIslandA"> 
   <port ajp="8010-8012", jms="8020-8022" rmi="8030-8032"/> 
  </oc4j> 

<!-- If the logs should be created in a specific directory provide that location 
here --!> 

  <log-file path="/tmp/ipm.log" level="4"/> 
</process-manager> 

Configuring the Oracle Notification System

The Oracle Notification System (ONS) uses a configuration file, opmn.xml. This section discusses the following elements in opmn.xml, under the "notification-server" section:

• port

• ssl

• log-file

port

This specifies the ports to which ONS binds and listens for notifications or the start, stop, restart, and statistical dump requests.

Attributes

• local: Bound to "localhost", this port is required and is used by all ONS clients on this host (including Oracle HTTP Server and OC4J).

• remote: Bound to the current host, this optional port is used for communication with other OPMN servers.

• request: Bound to the current host, this optional port is used for statistical dump requests only. This port is required by the DCM layer, which is the layer about OPMN.

ssl

This specifies SSL information for the ONS remote port. It is optional.

Attributes

• enabled: Set to "true" if ssl is enabled, otherwise "false". It is required.

• wallet-file: Specifies the path of the wallet file to use. It is optional.

• wallet-password: Specifies the password for the given wallet-file. It is required if wallet-file is specified.

log-file

This specifies the location of the notification manager log file. It is optional. Default is ORACLE_HOME/opmn/logs/ons.log.

The log-file element can have one attribute: level. It is optional. The level attribute can be one of 1=FATAL, 2=ERROR, 3=WARN, 4=NOTIFY, 5=DEBUG, 6=VERBOSE. The default is 3 (warning, error, and fatal messages are logged).

Sample Configuration Examples for Oracle Notification System

This section provides some sample configurations for the Oracle Notification System.

Example 1

<notification-server>
<port local ="6100" request="6300"/>
<log-file path="/ora/opmn/logs/ons.log" level="4"/>
</notification-server>

Example 2

<notification-server>
<port local ="6100" remote="6200" request="6300"/>
<ssl enabled="true" wallet-file="/ora/wal" wallet-password="foo">
<log-file path="/ora/opmn/logs/ons.log" level="4"/>
</notification-server>

Failure Scenarios

This section discusses the common failure scenarios and explains the response from the high availability infrastructure in Oracle9i Application Server. Topics include:

• Oracle HTTP Server Goes Down

• OC4J Process Goes Down

• Hi-AV Framework (OPMN) Goes Down

Oracle HTTP Server Goes Down

Figure 13-2 depicts a scenario where Oracle HTTP Server goes down.

Figure 13-2 Oracle HTTP Server Goes Down

Text description of the illustration ohsdown.gif

• Death Detection: The external systems, hardware load balancers or Oracle9iAS Web Cache, detect this and do not route requests to the dead Oracle HTTP Server, but instead route to the live one. OPMN is also notified of this event through one of its missed heartbeats, or from the OS, since it monitors that process through its process identifier.

• Restart: OPMN restarts the dead Oracle HTTP Server automatically. No external intervention is required. Oracle HTTP Server automatically retrieves the updated configuration file through DCM. In general, the restart is so quick that there should not have been any major configuration changes in the brief time frame.

• Fault Impact: The Oracle HTTP Server connection to OC4J is terminated. However, the session is not lost. All requests in process continue through the live Oracle HTTP Server.

• States Recovered: When Oracle HTTP Server comes back up, it rebuilds its routing information to all OC4J processes in the cluster and resumes steady state almost instantly. For non-J2EE applications, there is an additional impact of rebuilding all database connections and the like. Note that the J2EE session information is kept in the OC4J processes and is not impacted by Oracle HTTP Server being down.

OC4J Process Goes Down

Figure 13-3 depicts a scenario where OC4J goes down.

Figure 13-3 OC4J Goes Down

Text description of the illustration oc4jdown.gif

• Death Detection: OPMN is informed through one of the missed heartbeats or from the OS, since it monitors that process through its process identifier.

• Restart: OPMN restarts OC4J automatically. No external intervention is required. OC4J automatically retrieves the updated configuration, including new applications deployed, through DCM. However, in general, the restart is so quick that there should not have been any major changes in the brief time window.

• Fault Impact: The impact on a J2EE application is minimal. The session information is distributed to other OC4J processes in an OC4J cluster. Through OPMN, all Oracle HTTP Server routing tables are updated to reflect the downed OC4J. No further J2EE requests are sent to that OC4J process. There might be some performance impact since there are fewer OC4J processes running in the system. However, most systems should be able to handle this.

  For non-J2EE applications, such as PL/SQL or Perl, there is no impact at all.

• States Recovered: There is no impact on existing session states, since they are replicated across all OC4J processes within an island. The request is routed to another process in the same island in the same cluster. OC4J process automatically receives the updated configuration, including new application deployed, through DCM. OPMN updates the Oracle HTTP Server routing tables to reflect the re-addition of the new process, so routing to it resumes.

Hi-AV Framework (OPMN) Goes Down

Since OPMN does not play a role in actively servicing requests, this is not a critical failure. However, since it is critical in ensuring system-wide updates, its failure recovery is important to ensuring a healthy steady state.

Figure 13-4 depicts a scenario where OPMN goes down.

Figure 13-4 OPMN Goes Down

Text description of the illustration opmndown.gif

• Death Detection: A shadow process monitors OPMN. No other processes in the system monitor it.

• Restart: The shadow process that continually watches over OPMN ensures it comes back in case of a failure.

• Fault Impact: The impact is minimal. The routing information that OPMN communicates to the critical modules, such as mod_oc4j, might get stale while OPMN is down, resulting in less optimal routing.

• States Recovered: All routing table information and the past events are recovered. The regular heartbeats -- ping and reverse ping -- help the restarted OPMN re-build the cluster wide map in its memory.