CRS-4640 Error on Starting 11gR2 clusterware

CRS-4640 Error on Starting 11gR2 clusterware

I was working on a issue where in Clusterware was not coming up as private interface was down. Following errors were recorded in ocssd.log which informed that issue was with private interface

In   ocssd.log  following message appears
2012-01-16 18:45:04.647: [    CSSD][1110051136]clssnmvDHBValidateNCopy: node 1, rac1prod, has a disk HB, but no network HB, DHB has rcfg 21996
[root@rac2prod ~]# cd /u01/app/11.2.0/grid/bin


Checking status of crs informed that the OHASD process was up and running but CRS,CSSD and EVMD processes were not running.

[root@rac2prod bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager


I checked Private Interconnect Cables that were lose from the switch port.
After fixing the interface issue, we tried starting CRS with ‘crsctl start crs‘ command and it failed with following errors

[root@rac2prod bin]# ./crsctl start crs
 CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.

Since ohasd was already running, I tried crsctl start cluster (this command requires ohasd to be up), and this command succeeded

[root@rac2prod bin]# ./crsctl start cluster
CRS-2672: Attempting to start 'ora.cssd' on 'rac2prod'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac2prod'
CRS-2676: Start of 'ora.diskmon' on 'rac2prod' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac2prod' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac2prod'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rac2prod'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rac2prod' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac2prod'
CRS-2676: Start of 'ora.ctssd' on 'rac2prod' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac2prod'
CRS-2676: Start of 'ora.evmd' on 'rac2prod' succeeded
CRS-5702: Resource 'ora.evmd' is already running on 'rac2prod'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac2prod' succeeded
CRS-2679: Attempting to clean 'ora.asm' on 'rac2prod'
CRS-2681: Clean of 'ora.asm' on 'rac2prod' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac2prod'
CRS-2676: Start of 'ora.asm' on 'rac2prod' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac2prod'
CRS-2676: Start of 'ora.crsd' on 'rac2prod' succeeded
CRS-4000: Command Start failed, or completed with errors.

Ideally crsctl start crs should be used to start the Clusterware components. But in case they fail to come up due to some issue (e.g voting disk inaccessible,interface issue) and you are in situation when ohasd is up then you can use crsctl start cluster to start the remaining clusterware processes after fixing underlying issue.I believe crsctl stop crs -f option can also be used, though I didn’t try it for this issue.

[root@rac2prod bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

=======================================

Single Client Access Name (SCAN) Explained

11gR2 Grid Infrastructure Single Client Access Name (SCAN) Explained

SCAN Concepts

• Single client access name (SCAN) is the virtual hostname to provide for all clients connecting to the cluster (as opposed to the vip hostnames in 10g and 11gR1).  
• SCAN is a domain name registered to at least one and up to three IP addresses, either in the domain name service (DNS) or the Grid Naming Service (GNS).
• By default, the name used as the SCAN is also the name of the cluster and must be globally unique throughout your enterprise. The default value for the SCAN is based on the local node name. SCAN name must be at least one character long and no more than 15 characters in length, must be alphanumeric - cannot begin with a numeral and may contain hyphens (-). If you require a SCAN that is longer than 15 characters, then select an Advanced installation.
• For installation to succeed, the SCAN must resolve to at least one address.
• SCAN VIP addresses must be on the same subnet as virtual IP addresses and public IP addresses.
• Oracle strongly recommends that you do not configure SCAN VIP addresses in the hosts file. But if you use the hosts file to resolve SCAN name, you can have only one SCAN IP address.
• If hosts file is used to resolve SCAN hostname, you will receive Cluster Verification Utility failure at end of installation (see Note: 887471.1 for more details)
• For high availability and scalability, Oracle recommends that you configure the SCAN to use DNS Round Robin resolution to three addresses.
• Because the SCAN is associated with the cluster as a whole, rather than to a particular node, the SCAN makes it possible to add or remove nodes from the cluster without needing to reconfigure clients. It also adds location independence for the databases, so that client configuration does not have to depend on which nodes are running a particular database.
• Clients can continue to access the cluster in the same way as with previous releases, but Oracle recommends that clients accessing the cluster use the SCAN. Clients using the SCAN can also access the cluster using EZCONNECT.
• Grid Infrastructure will start local listener LISTENER on all nodes to listen on local VIP, and SCAN listener LISTENER_SCAN1 (up to three cluster wide) to listen on SCAN VIP(s); 11gR2 database by default will set local_listener to local LISTENER, and remote_listener to SCAN listener.
• SCAN listener will be running off GRID_HOME, and by default, in 11gR2 local listener will be running off GRID_HOME as well.

Common Questions Regarding SCAN

The following is a list of commonly asked questions regarding SCAN:

How can we configure the SCAN and SCAN listener?

During Typical installation, you are prompted to confirm the default Single Client Access Name (SCAN), which is used to connect to databases within the cluster irrespective of which nodes they are running on.  If you change the SCAN from the default, then the name that you use must be globally unique throughout your enterprise.

Also we can use 'srvctl' to add/modify the scan resource and the listeners.  Please refer to "Real Application Clusters Admin and Deployment Guide" or Note 1053147.1 for more information.

Do we still need to configure local listeners on each node?

Yes, you would need to configure independent local listeners for each node.  SCAN listeners are not replacements for the node listeners.

A new set of cluster processes called scan listeners will run on three nodes in a cluster (or all nodes if there are less than 3).  If you have more than three nodes, regardless of the number of nodes you have, there will be at most three scan listeners.  The database registers with the SCAN listener through the remote listener parameter in the init.ora/spfile.  If any of these clustered processes fail, they are automatically restarted on a new node.

How does SCAN work ?

 The following is the explanation given in Appendex D "Oracle Grid Infrastructure for a Cluster Installation Concepts" of the Oracle Grid Infrastructure Installation Guide:

"When a client submits a request, the SCAN listener listening on a SCAN IP address and the SCAN port is contracted on a client's behalf. Because all services on the cluster are registered with the SCAN listener, the SCAN listener replies with the address of the local listener on the least-loaded node (Each scan listener keeps updated cluster load statistics) where the service is currently being offered. Finally, the client establishes connection to the service through the listener on the node where service is offered.All of these actions take place transparently to the client without any explicit configuration required in the client."

$ SRVCTL STATUS SCAN_LISTENER
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node TEST1
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node TEST2
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node TEST3

Instead of DNS or GNS, Can we use '/etc/hosts' to resolve SCAN?

Oracle strongly recommends that you do not configure SCAN VIP addresses in the hosts file. But if you use the hosts file to resolve SCAN name, you can have only one SCAN IP address.

If hosts file is used to resolve SCAN hostname, you will receive Cluster Verification Utility failure at end of installation (See NOTE 887471.1 for more details)

Can we use the previous method (Using VIP) for client connection?

Clients can continue to access the cluster in the same way as with previous releases. Vips are still used internally, and can still be used for connections. But Oracle strongly recommends that clients accessing the cluster use the SCAN. Clients using the SCAN can also access the cluster using EZCONNECT.

Is it mandatory to use SCAN?

It's highly recommended to use SCAN unless there's strong business reason preventing it from being used.

Is it supported to remove SCAN?

SCAN is an elementary part of 11gR2 Grid Infrastructure, it's not supported to remove SCAN.

Demo

SCAN DEMO

Sample TNS entry for SCAN

TEST.ORACLE.COM =
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=SCAN-TEST.ORACLE.COM)(PORT=1521))
(CONNECT_DATA=(SERVICE_NAME=11GR2TEST.ORACLE.COM))
)

Sample TNS Entry without SCAN

TEST.ORACLE.COM =
(DESCRIPTION=
(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=tcp)(HOST=TEST1-vip.ORACLE.COM)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=TEST2-vip.ORACLE.COM)(PORT=1521))
)
(CONNECT_DATA=(SERVICE_NAME=11GR2TEST.ORACLE.COM))
)

 

RAC 11.2.0.2 Grid infrastructure, private interconnect bonding new feature HAIP

With Redundant Interconnect Usage, you can identify multiple interfaces to use for the cluster private network, without the need of using bonding or other technologies . This functionality is available starting with Oracle Database 11g Release 2 (11.2.0.2). When you define multiple interfaces, Oracle Clusterware creates from one to four highly available IP (HAIP) addresses. Oracle RAC and Oracle ASM instances use these interface addresses to ensure highly available, load-balanced interface communication between nodes. The installer enables Redundant Interconnect Usage to provide a high availability private network. Oracle Database, CSS, OCR, CRS, CTSS, and EVM components in 11.2.0.2 employ Redundant Interconnect Usage. By default, Oracle Grid Infrastructure software uses all of the HAIP addresses for private network communication, providing load-balancing across the set of interfaces you identify for the private network. If a private interconnect interface fails or become non-communicative, then Oracle Clusterware transparently moves the corresponding HAIP (Highly Available IP Address) to one of the remaining functional interfaces. If you have two different interface names to be used for the private interconnect you can provide both the interface names to the oracle grid infrastructure installer and oracle clusterware will create a Highly Available IP Address (HAIP). Oracle recommends: that you use the Redundant Interconnect Usage feature to make use of multiple interfaces for the private network. However, you can also use third-party technologies to provide redundancy for the private network. Note: If you define more than four interfaces as private network interfaces, be aware that Oracle Clusterware activates only four of the interfaces at a time. However, if one of the four active interfaces fails, then Oracle Clusterware transitions the HAIP addresses configured to the failed interface to one of the reserve interfaces in the defined set of private interfaces. Thanks & Regards, Zaman Ahmed Tatari Sr. Oracle Support Analyst OCP 9i/10g(DBA/Developer Track) Ora-Tech Systems (PVT) Ltd. UAN #: 111-ORACLE ext(216) Cell #: 92-300-2542241

NTP Failed with Oracle 11gR2 RAC on Linux

NTP configuration issues with Oracle 11gR2 RAC setup During the Grid infrastructure setup for Oracle 11gR2 RAC, I ran into a weird error with ntp not working with the Grid infrastructure installer. edit ntp configuration on both cluster nodes and restart ntp services. To do so, logon to each node as root and edit  the /etc/sysconfig/ntpd configuration file to make change for options settings as shown below: [root@vxnode2 ntp]# service ntpd stop [root@vxnode2 ntp]# cat /etc/sysconfig/ntpd # Drop root to id 'ntp:ntp' by default. OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid" # Set to 'yes' to sync hw clock after successful ntpdate SYNC_HWCLOCK=no # Additional options for ntpdate NTPDATE_OPTIONS="" now add -x before -u  see below [root@vxnode2 ntp]# vi /etc/sysconfig/ntpd OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid" [root@vxnode2 ntp]# cat /etc/sysconfig/ntpd # Drop root to id 'ntp:ntp' by default. OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid" # Set to 'yes' to sync hw clock after successful ntpdate SYNC_HWCLOCK=no # Additional options for ntpdate NTPDATE_OPTIONS="" You will need to restart ntp services for this to take affect: root@vxnode2 ntp]# service ntpd stop Shutting down ntpd: [ OK ] [root@vxnode2 ntp]# service ntpd start ntpd: Synchronizing with time server: [ OK ] Starting ntpd: [ OK ] [root@vxnode2 ntp]# ps -ef|grep ntp root 7363 1 0 03:51 ? 00:00:00 ntpd -x -u ntp:ntp -p /var/run/ntpd.pid Now we can move forward in the 11gR2 RAC Grid Infrastructure setup.

GSD Is Used Only if 9i RAC Database is Present

GSD Is Used Only if 9i RAC Database is Present [ID 429966.1] Applies to: Oracle Server - Enterprise Edition - Version: 10.1.0.2 to 11.2.0.0 Information in this document applies to any platform. Purpose The purpose of this document is to clarify the functionality of GSD in 10g and above. Scope This article is informational and intended for any user. GSD Is Used Only if 9i RAC Database is Present The function of GSD (10g and above) is to service requests for 9i RAC management clients and therefore when there are no 9i databases present, there is nothing for GSD to do. Consequently, there will be no impact on a RAC cluster if GSD is offline and 9i is not used. If gsd fails to start due to whetever reasons then best thing is to work with Oracle support to analyze and fix the issue. Until that time, gsd can be temporarily disabled. In 11.2 GSD is disabled by default and the service will show as target:offline, status:offline. Disable GSD (pre 11.2) After confirming that there are no 9i databases being used you can disable GSD by adding 'exit 0' after the initial comments in the script $ORACLE_HOME/bin/gsdctl where $ORACLE_HOME is the home from which nodeapps are running (i.e. crs home). #case $ORACLE_HOME in  # "") echo "****ORACLE_HOME environment variable not set!"  # echo " ORACLE_HOME should be set to the main"  # echo " directory that contains Oracle products."  # echo " Set and export ORACLE_HOME, then re-run."  # exit 1;;  #esac  exit 0 ## Manually added as a temporary workaround  A backup of the original script should be made before making the above change. Disable GSD (11.2) You may want to disable GSD after you upgraded all your Oracle9i RAC databases. srvctl stop nodeapps srvctl disable nodeapps -g srvctl start nodeapps Enable GSD in 11.2 srvctl enable nodeapps -g srvctl start nodeapps

Oracle 11gR2 New Features for RAC

The Basics: The clusterware is now known as Grid Infrastructure. CRS and ASM binaries reside under a single Oracle Home known as GRID Home. GRID Home and the Oracle Home must be installed in different locations. Oracle Grid Infrastructure cannot be placed under Oracle Base since permissions will be changed to root. OCR and Voting File can no longer be placed on RAW or block devices. They need to be placed either in ASM or on Shared NFS Storage. Voting file does not need to be backed up using dd command and is not supported anymore.The contents of Voting file are backed into the OCR. For more details on the architecture and understanding of background processes, refer Metalink Note: 1053147.1 SCAN: Single Client Access Name(SCAN) as the name suggests is a single name for clients to access RAC database. The obvious benefit is as nodes get added to the cluster, client TNS remains unchanged. It's like a virtual load balancer. You can find more details on Metalink Note:887522.1 Always remember the following about SCAN during your RAC installation: SCAN name can be maximum up to 15 characters excluding the domain name. Keep this in mind while requesting for scan name and IPs otherwise you won't be able to proceed with your installation since SCAN is a mandatory requirement. Ask your clients to use 11gR2 version of client coz with older version(11gR1/10gR2), if one of the SCAN listener goes down, the connection will fail. Only 11gR2 client will give full functionality of SCAN. You can have workarounds though by explicitly specifying all the 3 SCAN IPs in the client TNS. Set REMOTE_LISTENER to SCAN. DO NOT set to TNSNAMES alias. Example: remote_listener='scan-clu:1521'. Default installation will not update remote_listener and it's imperative to set it correctly if you wish your clients to use single name in the TNS. LOCAL_LISTENER should be set to node VIP as FQDN(Fully Qualified Domain Name). Default install would put the VIP but not as fully qualified. FQDN is important otherwise you may notice intermittent connection failures. Best way to troubleshoot SCAN is to set ORACLE_HOME to GRID_HOME since SCAN listener runs from Grid Home and check for 'lsnrctl servics SCAN_LISTENER' to verify if all instances are registered correctly.