CRS-4640 Error on Starting 11gR2 clusterware

CRS-4640 Error on Starting 11gR2 clusterware

I was working on a issue where in Clusterware was not coming up as private interface was down. Following errors were recorded in ocssd.log which informed that issue was with private interface

In   ocssd.log  following message appears
2012-01-16 18:45:04.647: [    CSSD][1110051136]clssnmvDHBValidateNCopy: node 1, rac1prod, has a disk HB, but no network HB, DHB has rcfg 21996
[root@rac2prod ~]# cd /u01/app/11.2.0/grid/bin


Checking status of crs informed that the OHASD process was up and running but CRS,CSSD and EVMD processes were not running.

[root@rac2prod bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager


I checked Private Interconnect Cables that were lose from the switch port.
After fixing the interface issue, we tried starting CRS with ‘crsctl start crs‘ command and it failed with following errors

[root@rac2prod bin]# ./crsctl start crs
 CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.

Since ohasd was already running, I tried crsctl start cluster (this command requires ohasd to be up), and this command succeeded

[root@rac2prod bin]# ./crsctl start cluster
CRS-2672: Attempting to start 'ora.cssd' on 'rac2prod'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac2prod'
CRS-2676: Start of 'ora.diskmon' on 'rac2prod' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac2prod' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac2prod'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rac2prod'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rac2prod' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac2prod'
CRS-2676: Start of 'ora.ctssd' on 'rac2prod' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac2prod'
CRS-2676: Start of 'ora.evmd' on 'rac2prod' succeeded
CRS-5702: Resource 'ora.evmd' is already running on 'rac2prod'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac2prod' succeeded
CRS-2679: Attempting to clean 'ora.asm' on 'rac2prod'
CRS-2681: Clean of 'ora.asm' on 'rac2prod' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac2prod'
CRS-2676: Start of 'ora.asm' on 'rac2prod' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac2prod'
CRS-2676: Start of 'ora.crsd' on 'rac2prod' succeeded
CRS-4000: Command Start failed, or completed with errors.

Ideally crsctl start crs should be used to start the Clusterware components. But in case they fail to come up due to some issue (e.g voting disk inaccessible,interface issue) and you are in situation when ohasd is up then you can use crsctl start cluster to start the remaining clusterware processes after fixing underlying issue.I believe crsctl stop crs -f option can also be used, though I didn’t try it for this issue.

[root@rac2prod bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

=======================================