3 Configuring Oracle Clusterware and Oracle Database Storage

3.1.1 Overview of Oracle Clusterware Storage Options

There are three ways of storing Oracle Clusterware files:

• A supported shared file system: Supported file systems include the following:

  • Cluster File System with TruCluster v. 5.1B with Advanced File System (AdvFS)

• Raw partitions: Raw partitions are disk partitions that are not mounted and written to using the file system, but instead are accessed directly by the application.

• Logical Storage Manager with TruCluster v. 5.1B

3.1.2 Overview of Oracle Database and Recovery File Options

There are three ways of storing Oracle Database and recovery files:

• Automatic Storage Management: Automatic Storage Management (ASM) is an integrated, high-performance database file system and disk manager for Oracle Database files.

• A supported shared file system: Supported file systems include the following:

  • HP Tru64 TruCluster

• Raw partitions (database files only): A raw partition is required for each database file.

3.1.3 General Storage Considerations

For all installations, you must choose the storage option that you want to use for Oracle Clusterware files and Oracle Database files. If you want to enable automated backups during installation, then you must also choose the storage option that you want to use for recovery files (the flash recovery area). You do not have to use the same storage option for each file type.

http://metalink.oracle.com

3.1.4 After You Have Selected Disk Storage Options

When you have determined your disk storage options, you must perform the following tasks in the following order:

1: Check for available shared storage with CVU

Refer to Checking for Available Shared Storage with CVU.

2: Configure shared storage for Oracle Clusterware files

• To use a File System for Oracle Clusterware Files, refer to "Configuring Storage for Oracle Clusterware Files on a Supported Shared File System"

• To use raw devices (partitions) for Oracle Clusterware files, refer to "Configuring Storage for Oracle Clusterware Files on Raw Devices".

3: Configure storage for Oracle Database files and recovery files

• To use a file system for database or recovery file storage, refer to Configuring Storage for Oracle Clusterware Files on a Supported Shared File System, and ensure that in addition to the volumes you create for Oracle Clusterware files, you also create additional volumes with sizes sufficient to store database files.

• To use Automatic Storage Management for database or recovery file storage, refer to "Configuring Database File Storage on ASM and Raw Devices".

• To use raw devices (partitions) for database file storage, refer to "Configuring Database File Storage on Raw Devices".

3.3.1 Requirements for Using a File System for Oracle Clusterware Files

To use a file system for Oracle Clusterware files, the file system must comply with the following requirements:

• To use a cluster file system, it must be a supported cluster file system, as listed in the section "Deciding to Use a Cluster File System for Data Files".

• If you choose to place your Oracle Cluster Registry (OCR) files on a shared file system, then one of the following must be true:

  • The disks used for the file system are on a highly available storage device, (for example, a RAID device that implements file redundancy).

  • At least two file systems are mounted, and use the features of Oracle Database 10g Release 2 (10.2) to provide redundancy for the OCR.

• If you intend to use a shared file system to store database files, then use at least two independent file systems, with the database files on one file system, and the recovery files on a different file system.

• The oracle user must have write permissions to create the files in the path that you specify.

Use Table 3-2 to determine the partition size for shared file systems.

In Table 3-2, the total required volume size is cumulative. For example, to store all files on the shared file system, you should have at least 3.4 GB of storage available over a minimum of two volumes.

3.3.2 Creating Required Directories for Oracle Clusterware Files on Shared File Systems

Use the following instructions to create directories for Oracle Clusterware files. You can also configure shared file systems for the Oracle Database and recovery files.

To create directories for the Oracle Clusterware files on separate file systems from the Oracle base directory, follow these steps:

1. If necessary, configure the shared file systems that you want to use and mount them on each node.

   Note:

   The cluster file system directory that you use for the file system must be identical on each node. Ensure that the file systems are configured to mount automatically when a node restarts.

2. Use the df -h command to determine the free disk space on each storage area.

3. From the display, identify the file systems that you want to use:

   File Type	File System Requirements
   Oracle Clusterware files	Choose a file system with at least 512 MB of free disk space (one OCR and one voting disk, with external redundancy)
   Database files	Choose either: A single file system with at least 1.2 GB of free disk space. Two or more file systems with at least 1.2 GB of free disk space in total.
   Recovery files	Choose a file system with at least 2 GB of free disk space.

   
   

   If you are using the same file system for more than one type of file, then add the disk space requirements for each type to determine the total disk space requirement.

4. Note the names of the cluster file system directories for the file systems that you identified.

5. If the user performing installation (typically, oracle) has permissions to create directories on the disks where you plan to install Oracle Clusterware and Oracle Database, then OUI creates the Oracle Clusterware file directory, and DBCA creates the Oracle Database file directory, and the Recovery file directory.

   If the user performing installation does not have write access, then you must create these directories manually using commands similar to the following to create the recommended subdirectories in each of the cluster file system (cfsdirectory) directories and set the appropriate owner, group, and permissions on them:

   • Oracle Clusterware file directory:

     # mkdir /cfsdirectory/oracrs
     # chown oracle:oinstall /cfsdirectory/oracrs
     # chmod 775 /cfsdirectory/oracrs
     
     

   • Database file directory:

     # mkdir /cfsdirectory/oradata
     # chown oracle:oinstall /cfsdirectory/oradata
     # chmod 775 /cfsdirectory/oradata
     
     

   • Recovery file directory (flash recovery area):

     # mkdir /cfsdirectory/flash_recovery_area
     # chown oracle:oinstall /cfsdirectory/flash_recovery_area
     # chmod 775 /cfsdirectory/flash_recovery_area
     
     

By making the oracle user the owner of these directories, this permits them to be read by multiple Oracle homes, including those with different OSDBA groups.

When you have completed creating subdirectories in each of the mount point directories, and set the appropriate owner, group, and permissions, you have completed TruCluster configuration.

3.4.1 Identifying Required Raw Partitions for Clusterware Files

Table 3-3 lists the number and size of the raw partitions that you must configure for Oracle Clusterware files.

3.4.2 Configuring Raw Devices

To configure raw devices and volumes on Tru64 systems:

1. If necessary, install or configure the shared disk devices that you intend to use for the raw partitions and restart the system.

   Ensure that the partitions you are adding are on a shared disk. You can store the files that Oracle Real Application Clusters requires directly on the cluster file system.

   Note:

   Because the number of partitions that you can create on a single device is limited, you might need to create the required raw partitions on more than one device.

2. Determine the names of the free disk partitions.

   A free disk partition is one that is not used for a Tru64 file system that complies with the following restrictions:

   • It is not listed when you run the command /usr/sbin/mount.

   • It is not in use as a swap device.

   • It does not overlap a swap partition.

   • It is not in use by other Tru64 applications.

   • It does not overlap the Tru64 file system.

   • It does not use a space already used by the file system.

   To determine if a partition is free, obtain a complete map of the starting locations and sizes of the partitions on the device and check for free space. Some partitions may contain file systems that are currently not mounted and are not listed in the /usr/sbin/mount output.

   Note:

   Ensure that the partition does not start at cylinder 0.

3. Set up the raw device for use by Oracle Clusterware.

   Begin by verifying that the disk is partitioned. If it is not, then use the command disklabel to partition it.

4. Enter the ls command to view the owner and permissions of the device file. For example:

   $ ls -1a
   
   

5. Ensure that the partition is owned by the oracle user. If necessary, use the command chown to change the ownership on the block and character files for the device. For example:

   # chown oracle /dev/rdisk/dsk10c
   
   

6. Ensure that the partition has the correct permissions. If necessary, use the command chmod to make the partition accessible only to the oracle user. For example:

   # chmod 600 /dev/rdisk/dsk10c
   
   

7. Create a symbolic link to the raw devices you require. For example:

   $ ln -s /dev/rdisk/dsk10c /oracle_data/datafile.dbf
   
   

   To verify that you have created the symbolic link, use the character special device (not the block special device) and enter the following command:

   $ ls -Ll datafile
   
   

   The following message should appear:

   crwxrwxrwx oracle dba datafile
   
   

3.4.3 Completing Supported Shared Storage Configuration

When you have completed creating subdirectories in each of the mount point directories, and set the appropriate owner, group, and permissions, you have completed supported shared storage configuration.

3.5.1 Deciding to Use a Cluster File System for Data Files

If you choose to place the Oracle database files on a file system, then use the following guidelines when deciding where to place them:

• You must choose a shared file system on shared disk; the file system must be a cluster file system.

• The default path suggested by the Installer for the database file directory is a subdirectory of the Oracle base directory. You can choose this path only if you are using an Oracle base directory that is on a shared file system.

  This default location is not recommended for production databases.

• You can choose either a single file system or more than one file system to store the database files:

  • If you want to use a single file system, choose a file system on a physical device that is dedicated to the database.

    For best performance and reliability, choose a RAID device or a logical volume on more than one physical device and implement the stripe-and-mirror-everything (SAME) methodology.

  • If you want to use more than one file system, choose file systems on separate physical devices that are dedicated to the database.

    This method enables you to distribute physical I/O and create separate control files on different devices for increased reliability. It also enables you to fully implement the OFA guidelines. You must choose either the Advanced database creation option or the Custom installation type during the installation to implement this method.

• If you intend to create a preconfigured database during the installation, the file system (or file systems) that you choose must have at least 1.2 GB of free disk space.

  For production databases, you must estimate the disk space requirement depending on the use that you want to make of the database.

• For optimum performance, the file systems that you choose should be on physical devices that are used only by the database.

• The oracle user must have write permissions to create the files in the path that you specify.

3.5.2 Guidelines for Placing Oracle Recovery Files on a File System

If you choose to place the Oracle recovery files on a file system, use the following guidelines when deciding where to place them:

• To prevent disk failure from making both the database files and the recovery files unavailable, place the recovery files in a file system on a different physical disk from the database files.

  Note:

  Alternatively use an ASM disk group with a normal or high redundancy level for either or both file types.

• The file system that you choose should have at least 2 GB of free disk space.

  The disk space requirement is the default disk quota configured for the flash recovery area (specified by the DB_RECOVERY_FILE_DEST_SIZE initialization parameter).

  If you choose the Custom installation type or the Advanced database configuration option, you can specify a different disk quota value. After you create the database, you can also use Oracle Enterprise Manager Grid Control or Database Control to specify a different value.

  For more information about sizing the flash recovery area, refer to the Oracle Backup and Recovery Basics manual.

• You must use TruClusters.

• The default path suggested by the Installer for the flash recovery area is a subdirectory of the Oracle base directory. You can choose this path only if you are using an Oracle base directory that is on a shared file system.

  This default location is not recommended for production databases.

• The oracle user must have write permissions to create the files in the path that you specify.

3.6.1 Identifying Storage Requirements for Automatic Storage Management

To identify the storage requirements for using Automatic Storage Management, you must determine how many devices and the amount of free disk space that you require. To complete this task, follow these steps:

1. Determine whether you want to use Automatic Storage Management for Oracle Database files, recovery files, or both.

   Note:

   You do not have to use the same storage mechanism for database files and recovery files. You can use the file system for one file type and Automatic Storage Management for the other.

   If you choose to enable automated backups and you do not have a shared file system available, then you must choose Automatic Storage Management for recovery file storage.

   If you enable automated backups during the installation, you can choose Automatic Storage Management as the storage mechanism for recovery files by specifying an Automatic Storage Management disk group for the flash recovery area. Depending on how you choose to create a database during the installation, you have the following options:

   • If you select an installation method that runs Database Configuration Assistant in interactive mode (for example, by choosing the Advanced database configuration option) then you can decide whether you want to use the same Automatic Storage Management disk group for database files and recovery files, or use different disk groups for each file type.

     The same choice is available to you if you use Database Configuration Assistant after the installation to create a database.

   • If you select an installation method that runs Database Configuration Assistant in noninteractive mode, then you must use the same Automatic Storage Management disk group for database files and recovery files.

2. Choose the Automatic Storage Management redundancy level that you want to use for the Automatic Storage Management disk group.

   The redundancy level that you choose for the Automatic Storage Management disk group determines how Automatic Storage Management mirrors files in the disk group and determines the number of disks and amount of free disk space that you require, as follows:

   • External redundancy

     An external redundancy disk group requires a minimum of one disk device. The effective disk space in an external redundancy disk group is the sum of the disk space in all of its devices.

     Because Automatic Storage Management does not mirror data in an external redundancy disk group, Oracle recommends that you use only RAID or similar devices that provide their own data protection mechanisms as disk devices in this type of disk group.

   • Normal redundancy

     In a normal redundancy disk group, Automatic Storage Management uses two-way mirroring by default, to increase performance and reliability. A normal redundancy disk group requires a minimum of two disk devices (or two failure groups). The effective disk space in a normal redundancy disk group is half the sum of the disk space in all of its devices.

     For most installations, Oracle recommends that you select normal redundancy disk groups.

   • High redundancy

     In a high redundancy disk group, Automatic Storage Management uses three-way mirroring to increase performance and provide the highest level of reliability. A high redundancy disk group requires a minimum of three disk devices (or three failure groups). The effective disk space in a high redundancy disk group is one-third the sum of the disk space in all of its devices.

     While high redundancy disk groups do provide a high level of data protection, you should consider the greater cost of additional storage devices before deciding to select high redundancy disk groups.

3. Determine the total amount of disk space that you require for the database files and recovery files.

   Use the following table to determine the minimum number of disks and the minimum disk space requirements for installing the starter database:

   Redundancy Level	Minimum Number of Disks	Database Files	Recovery Files	Both File Types
   External	1	1.15 GB	2.3 GB	3.45 GB
   Normal	2	2.3 GB	4.6 GB	6.9 GB
   High	3	3.45 GB	6.9 GB	10.35 GB

   
   

   For RAC installations, you must also add additional disk space for the Automatic Storage Management metadata. You can use the following formula to calculate the additional disk space requirements (in MB):

   15 + (2 * number_of_disks) + (126 * number_of_Automatic_Storage_Management_instances)

   For example, for a four-node RAC installation, using three disks in a high redundancy disk group, you require an additional 525 MB of disk space:

   15 + (2 * 3) + (126 * 4) = 525

   If an Automatic Storage Management instance is already running on the system, then you can use an existing disk group to meet these storage requirements. If necessary, you can add disks to an existing disk group during the installation.

   The following section describes how to identify existing disk groups and determine the free disk space that they contain.

4. Optionally, identify failure groups for the Automatic Storage Management disk group devices.

   Note:

   You need to complete this step only if you intend to use an installation method that runs Database Configuration Assistant in interactive mode, for example, if you intend to choose the Custom installation type or the Advanced database configuration option. Other installation types do not enable you to specify failure groups.

   If you intend to use a normal or high redundancy disk group, then you can further protect your database against hardware failure by associating a set of disk devices in a custom failure group. By default, each device comprises its own failure group. However, if two disk devices in a normal redundancy disk group are attached to the same SCSI controller, then the disk group becomes unavailable if the controller fails. The controller in this example is a single point of failure.

   To protect against failures of this type, you could use two SCSI controllers, each with two disks, and define a failure group for the disks attached to each controller. This configuration would enable the disk group to tolerate the failure of one SCSI controller.

   Note:

   If you define custom failure groups, then you must specify a minimum of two failure groups for normal redundancy disk groups and three failure groups for high redundancy disk groups.

5. If you are sure that a suitable disk group does not exist on the system, then install or identify appropriate disk devices to add to a new disk group. Locate appropriate disk devices with the following command:

   # hwmgr -view devices
   # df -k
   
   

   When you select Automatic Storage Management (ASM), OUI displays an ASM Management Options page showing the disk partition locations from which you must select. The default partitions are in the following path:

   /dev/rdisk/*
   
   

   Use the following guidelines when identifying appropriate disk devices:

   • All of the devices in an Automatic Storage Management disk group should be the same size and have the same performance characteristics.

   • Do not specify more than one partition on a single physical disk as a disk group device. Automatic Storage Management expects each disk group device to be on a separate physical disk.

   • Although you can specify a logical volume as a device in an Automatic Storage Management disk group, Oracle does not recommend their use. Logical volume managers can hide the physical disk architecture, preventing Automatic Storage Management from optimizing I/O across the physical devices. They are not supported with RAC.

   See Also:

   The "Configuring Disks for Automatic Storage Management" section for information about completing this task

3.6.2 Using an Existing Automatic Storage Management Disk Group

If you want to store either database or recovery files in an existing Automatic Storage Management disk group, then you have the following choices, depending on the installation method that you select:

• If you select an installation method that runs Database Configuration Assistant in interactive mode (for example, by choosing the Advanced database configuration option), then you can decide whether you want to create a disk group, or to use an existing one.

  The same choice is available to you if you use Database Configuration Assistant after the installation to create a database.

• If you select an installation method that runs Database Configuration Assistant in noninteractive mode, then you must choose an existing disk group for the new database; you cannot create a disk group. However, you can add disk devices to an existing disk group if it has insufficient free space for your requirements.

To determine if an existing Automatic Storage Management disk group exists, or to determine if there is sufficient disk space in a disk group, you can use Oracle Enterprise Manager Grid Control or Database Control. Alternatively, you can use the following procedure:

1. View the contents of the oratab file to determine if an Automatic Storage Management instance is configured on the system:

   $ more /etc/oratab
   
   

   If an Automatic Storage Management instance is configured on the system, then the oratab file should contain a line similar to the following:

   +ASM2:oracle_home_path
   
   

   In this example, +ASM2 is the system identifier (SID) of the Automatic Storage Management instance, with the node number appended, and oracle_home_path is the Oracle home directory where it is installed. By convention, the SID for an Automatic Storage Management instance begins with a plus sign.

2. Set the ORACLE_SID and ORACLE_HOME environment variables to specify the appropriate values for the Automatic Storage Management instance that you want to use.

3. Connect to the Automatic Storage Management instance as the SYS user with SYSDBA privilege and start the instance if necessary:

   $ $ORACLE_HOME/bin/sqlplus "SYS/SYS_password as SYSDBA"
   SQL> STARTUP
   
   

4. Enter the following command to view the existing disk groups, their redundancy level, and the amount of free disk space in each one:

   SQL> SELECT NAME,TYPE,TOTAL_MB,FREE_MB FROM V$ASM_DISKGROUP;
   
   

5. From the output, identify a disk group with the appropriate redundancy level and note the free space that it contains.

6. If necessary, install or identify the additional disk devices required to meet the storage requirements listed in the previous section.

   Note:

   If you are adding devices to an existing disk group, then Oracle recommends that you use devices that have the same size and performance characteristics as the existing devices in that disk group.

3.6.3 Configuring Database File Storage on ASM and Raw Devices

To configure disks for Automatic Storage Management (ASM) using raw devices, complete the following tasks:

1. To use ASM with raw partitions, you must create sufficient partitions for your data files, and then bind the partitions to raw devices. To do this, follow the instructions provided for Oracle Clusterware in the section "Configuring Storage for Oracle Clusterware Files on Raw Devices". If you want to configure partitions after installing the database, then refer to Additional Postinstallation Tasks for Tru64.

2. Make a list of the raw device names you create for the data files, and have the list available during database installation.

When you have completed creating and configuring ASM with raw partitions, proceed to Chapter 4, "Installing Oracle Clusterware".

3.7.1 Identifying Required Raw Partitions for Database Files

Table 3-4 lists the number and size of the raw partitions that you must configure for database files. If you need to create additional raw partitions, then refer to the section Configuring Raw Devices. If you want to configure partitions after installing the database, then refer to Additional Postinstallation Tasks for Tru64.

3.7.2 Creating the Database Configuration Assistant Raw Device Mapping File

To allow Database Configuration Assistant to identify the appropriate raw device for each database file, you must create a raw device mapping file, as follows:

1. Set the ORACLE_BASE environment variable to specify the Oracle base directory that you identified or created previously:

   • Bourne, Bash, or Korn shell:

     $ ORACLE_BASE=/u01/app/oracle ; export ORACLE_BASE
     
     

   • C shell:

     % setenv ORACLE_BASE /u01/app/oracle
     
     

2. Create a database file subdirectory under the Oracle base directory and set the appropriate owner, group, and permissions on it:

   # mkdir -p $ORACLE_BASE/oradata/dbname
   # chown -R oracle:oinstall $ORACLE_BASE/oradata
   # chmod -R 775 $ORACLE_BASE/oradata
   
   

   In this example, dbname is the name of the database that you chose previously.

3. Change directory to the $ORACLE_BASE/oradata/dbname directory.

4. Edit the dbname_raw.conf file in any text editor to create a file similar to the following:

   Note:

   The following example shows a sample mapping file for a two-instance RAC cluster.

   system=dev/rdisk/raw1
   sysaux=dev/rdisk/raw2
   example=dev/rdisk/raw3
   users=dev/rdisk/raw4
   temp=dev/rdisk/raw5
   undotbs1=dev/rdisk/raw6
   undotbs2=dev/rdisk/raw7
   redo1_1=dev/rdisk/raw8
   redo1_2=dev/rdisk/raw9
   redo2_1=dev/rdisk/raw10
   redo2_2=dev/rdisk/raw11
   control1=dev/rdisk/raw12
   control2=dev/rdisk/raw13
   spfile=dev/rdisk/raw14
   pwdfile=dev/rdisk/raw15
   
   

   Use the following guidelines when creating or editing this file:

   • Each line in the file must have the following format:

     database_object_identifier=raw_device_path
     
     

   • For a single-instance database, the file must specify one automatic undo tablespace data file (undotbs1), and at least two redo log files (redo1_1, redo1_2).

   • For a RAC database, the file must specify one automatic undo tablespace data file (undotbsn) and two redo log files (redon_1, redon_2) for each instance.

   • Specify at least two control files (control1, control2).

   • To use manual instead of automatic undo management, specify a single rollback segment tablespace data file (rbs) instead of the automatic undo management tablespace data files.

5. Save the file, and note the file name that you specified.

6. If you are using raw devices for database storage, then set the DBCA_RAW_CONFIG environment variable to specify the full path to the raw device mapping file:

   Bourne, Bash, or Korn shell:

   $ DBCA_RAW_CONFIG=$ORACLE_BASE/oradata/dbname/dbname_raw.conf
   $ export DBCA_RAW_CONFIG
   
   

   C shell:

   $ setenv DBCA_RAW_CONFIG=$ORACLE_BASE/oradata/dbname/dbname_raw.conf