CLSRSC-115: Start of resource 'ora.cssd' failed - 11g to 12c RAC Grid Interface Upgrade.

Cause:
=====

GI rootupgrade.sh to 12c fails on node 1 with the following error.

Symptoms

1) ./rootupgrade.sh

 
OLR initialization - successful
2016/12/31 11:08:21 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2016/12/31 11:47:49 CLSRSC-115: Start of resource 'ora.cssd' failed

2016/12/31 11:47:49 CLSRSC-117: Failed to start Oracle Clusterware stack

2016/12/31 11:47:49 CLSRSC-244: Failed to start Cluster Synchronization Service in clustered mode

Died at /oracle/app/12.1.0.2/grid_1/crs/install/crsupgrade.pm line 794.
The command '/oracle/app/12.1.0.2/grid_1/perl/bin/perl -I/oracle/app/12.1.0.2/grid_1/perl/lib -I/oracle/app/12.1.0.2/grid_1/crs/install /oracle/app/12.1.0.2/grid_1/crs/install/rootcrs.pl  -upgrade' execution failed
[root@hrmdb01 ~]#

2) The CSSD log or trace shows No voting files found when it starts

- For 12.1.0.1 in $NEW_GRID_HOME/log/<hostname>/log/cssd/ocssd.log

- For 12.1.0.2 or higher  $ORACLE_BASE/crs/<hostname>/crs/trace/ocssd.trc
----------------------------------------------------------
2016-31-12 22:44:33.088210 :   SKGFD:1113868608: Execute glob on the string /dev/sd*                                   <<<<------ note the wrong discovery string
2016-31-12 22:44:33.088281 :   SKGFD:1113868608: OSS discovery with ::

2016-31-12 22:44:33.088985 :    CSSD:1113868608: clssnmvDiskVerify: Successful discovery of 0 disks
2016-31-12 22:44:33.088990 :    CSSD:1113868608: clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2016-31-12 22:44:33.088992 :    CSSD:1113868608: clssnmvFindInitialConfigs: No voting files found
2016-31-12 22:44:33.089052 :    CSSD:1113868608: (:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds
-----------------------------------------------------------------

Solution  :
========


An empty discovery string in 11.2 may work but it will not work in 12.1 and higher.

Solution

1) Downgrade GRID as root user

# cd  <NEW_GRID_HOME>/crs/install/

# ./rootcrs.sh -downgrade -force

2) As GRID owner update asm_diskstring

SQL> alter system stop rolling migration;

SQL> alter system set asm_diskstring='<correct ASM discovery string>' scope=spfile;

3) Re-run rootupgrade.sh


Note : ASM instance should be use spfile, if not please create spfile as below.

Login in the 11g asm instance

$ sqlplus / as sysasm

Edit the asm pfile below parameters

asm_diskstring='/dev/raw/raw*'


SQL> shutdown abort;
ASM instance shutdown

SQL> startup;

SQL> Show parameter string

SQL> create spfile='+DBFS_DG/spfileASM.ora' from pfile;

File created.

SQL> exit

Looking for the ASM spfile again shows two entries:

$ asmcmd find --type ASMPARAMETERFILE +DBFS_DG "*"
+DATA/ASM/ASMPARAMETERFILE/REGISTRY.253.843597139
+DATA/spfileASM.ora