CRS-2883: Resource 'ora.asm' failed during Clusterware stack start

Symptoms
========

root@oraprod02:~# /u01/app/12.1.0.2/grid/crs/install/rootcrs.pl -postpatch
Using configuration parameter file: /u01/app/12.1.0.2/grid/crs/install/crsconfig_params
2017/11/05 15:57:38 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2017/11/05 15:58:08 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2017/11/05 15:58:13 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'

CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.evmd' on 'oraprod02'
CRS-2672: Attempting to start 'ora.mdnsd' on 'oraprod02'
CRS-2676: Start of 'ora.mdnsd' on 'oraprod02' succeeded
CRS-2676: Start of 'ora.evmd' on 'oraprod02' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oraprod02'
CRS-2676: Start of 'ora.gpnpd' on 'oraprod02' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'oraprod02'
CRS-2676: Start of 'ora.gipcd' on 'oraprod02' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oraprod02'
CRS-2676: Start of 'ora.cssdmonitor' on 'oraprod02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oraprod02'
CRS-2672: Attempting to start 'ora.diskmon' on 'oraprod02'
CRS-2676: Start of 'ora.diskmon' on 'oraprod02' succeeded
CRS-2676: Start of 'ora.cssd' on 'oraprod02' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'oraprod02'
CRS-2676: Start of 'ora.ctssd' on 'oraprod02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oraprod02'
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/oraprod02/crs/trace/ohasd_oraagent_oracle.trc".
CRS-2883: Resource 'ora.asm' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
2017/11/05 15:59:55 CLSRSC-117: Failed to start Oracle Clusterware stack

Died at /u01/app/12.1.0.2/grid/crs/install/crspatch.pm line 941.



Cause
=====

This issue could be due to file permission issue in some of the binary files in the Grid home.

You can check and compare the same from both nodes under $GRID_HOME/bin

Please check for any files with the wrong permission.

Also check the following files found under from both nodes.

/u01/app/12.1.0.2/grid/crs/utl/<hostname>/

* crsconfig_fileperms
* crsconfig_dirs


Solution
======

As root user:

# <GRID_HOME>/bin/crsctl stop crs -f

# cd <GRID_HOME>/crs/install

# ./rootcrs.pl -unlock

# ./rootcrs.pl -patch

Comments

Post a Comment

Popular posts from this blog

Bare Metal Restore Procedure for Compute Nodes on an Exadata Environment

Remove the Failed Database Server from the Cluster 1. Disable the listener that runs on the failed database server: [oragrid@surviving ~]$ srvctl disable listener -n dm01db01 [oragrid@surviving ~]$ srvctl stop listener -n dm01db01 PRCC-1017 : LISTENER was already stopped on dm01db01 PRCR-1005 : Resource ora.LISTENER.lsnr is already stopped [oragrid@surviving ~]$ 2. Delete the Inactive Instance using DBCA via ( Instance management ) and Oracle Home from the Oracle inventory: [oracle@surviving ~]$ dbca ( vncserver ) [oracle@surviving ~]$ cd $ORACLE_HOME/oui/bin [oracle@surviving bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome_1 "CLUSTER_NODES=dm01db02" Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB.   Actual 23984 MB    Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oraInventory 'UpdateNodeList' was successful. [or...
Read more

Supercluster App Domain RAC setup not able to start after ZFS storage rebooted.

11gR2 Grid Infrastructure on Solaris local containers, cssd process fails to start with following symptoms: ** root.sh output CRS-2674: Start of 'ora.cssd' on 'racnode1' failed .. CRS-5804: Communication error with agent process CRS-4000: Command Start failed, or completed with errors. The exlusive mode cluster start failed, see Clusterware alert log for more information Initial cluster configuration failed.  See /oracle_crs/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_racnode1.log for details ** $GRID_HOME/cfgtoollogs/crsconfig/rootcrs_<nodename>.log 2017-03-09 22:33:38: CRS-2674: Start of 'ora.cssd' on 'racnode1' failed .. 2017-03-09 22:33:38: CRS-5804: Communication error with agent process 2017-03-09 22:33:38: CRS-4000: Command Start failed, or completed with errors. 2017-03-09 22:33:38: CSS startup failed with return code 1 .. 2017-03-09 22:33:51: The exlusive mode cluster start failed, see Grid Infrastructure alert log for more ** $GRID_HOM...
Read more