Supercluster App Domain RAC setup not able to start after ZFS storage rebooted.

11gR2 Grid Infrastructure on Solaris local containers, cssd process fails to start with following symptoms:

** root.sh output

CRS-2674: Start of 'ora.cssd' on 'racnode1' failed
..
CRS-5804: Communication error with agent process
CRS-4000: Command Start failed, or completed with errors.
The exlusive mode cluster start failed, see Clusterware alert log for more information
Initial cluster configuration failed.  See /oracle_crs/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_racnode1.log for details

** $GRID_HOME/cfgtoollogs/crsconfig/rootcrs_<nodename>.log

2017-03-09 22:33:38: CRS-2674: Start of 'ora.cssd' on 'racnode1' failed
..
2017-03-09 22:33:38: CRS-5804: Communication error with agent process
2017-03-09 22:33:38: CRS-4000: Command Start failed, or completed with errors.
2017-03-09 22:33:38: CSS startup failed with return code 1
..
2017-03-09 22:33:51: The exlusive mode cluster start failed, see Grid Infrastructure alert log for more

** $GRID_HOME/log/<nodename>/cssd/ocssd.log

2017-03-09 22:33:30.467: [    CSSD][1]clssscSetPrivEnv: Setting priority to 4
2017-03-09 22:33:30.491: [    CSSD][1]clssscSetPrivEnv: unable to set priority to 4
2017-03-09 22:33:30.491: [    CSSD][1]SLOS: cat=-2, opn=scls_set_priority_realtime, dep=1, loc=setsched
unable to escalate to real time
..
2017-03-09 22:33:30.491: [    CSSD][1](:CSSSC00011:)clssscExit: A fatal error occurred during initialization




Cause

Oracle Clusterware requires additional privileges that are not provided by default.

see below Oracle White Paper for :

RUNNING ORACLE® REAL APPLICATION CLUSTERS ON ORACLE SOLARIS ZONE CLUSTERS

http://www.oracle.com/technetwork/articles/servers-storage-admin/o11-062-rac-solariszonescluster-429206.pdf


Solution

Halt local container, execute the following on global zone and restart the local container:

<local-container-name> =============> Zone Name , you can see "zoneadm list -civ

zonecfg -z <local-container-name>
set limitpriv="default,proc_priocntl,proc_clock_highres,sys_time"
commit
exit
zoneadm -z <local-container-name> reboot

Once local container is restarted, refer to note 942166.1 or note 969254.1 to proceed.



To list Privilege Set, execute "ppriv -l <zone>" in either global or local zone.


Ref:

Grid Infrastructure cssd fails to start on Solaris local containers (zones) (Doc ID 1340694.1)

Comments

Post a Comment

Popular posts from this blog

Bare Metal Restore Procedure for Compute Nodes on an Exadata Environment

Remove the Failed Database Server from the Cluster 1. Disable the listener that runs on the failed database server: [oragrid@surviving ~]$ srvctl disable listener -n dm01db01 [oragrid@surviving ~]$ srvctl stop listener -n dm01db01 PRCC-1017 : LISTENER was already stopped on dm01db01 PRCR-1005 : Resource ora.LISTENER.lsnr is already stopped [oragrid@surviving ~]$ 2. Delete the Inactive Instance using DBCA via ( Instance management ) and Oracle Home from the Oracle inventory: [oracle@surviving ~]$ dbca ( vncserver ) [oracle@surviving ~]$ cd $ORACLE_HOME/oui/bin [oracle@surviving bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome_1 "CLUSTER_NODES=dm01db02" Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB.   Actual 23984 MB    Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oraInventory 'UpdateNodeList' was successful. [or...
Read more

[INS-40718] Single Client Access Name (SCAN): could not be resolved. ( LDOMS & Zones)

Cause The name you provided as the SCAN could not be resolved using TCP/IP host name lookup. Action-Provide name to use for the SCAN for which the domain can be resolved Global Zone : ============= # svccfg svc:> select name-service/switch svc:/system/name-service/switch> setprop config/host = astring: "files dns" svc:/system/name-service/switch> select system/name-service/switch:default svc:/system/name-service/switch:default> refresh svc:/system/name-service/switch:default> validate svc:/system/name-service/switch:default> exit # svcadm refresh name-service/switch # svcadm restart /system/name-service-cache; then, I can resolve host using telnet/ping.
Read more