Supercluster App Domain RAC setup not able to start after ZFS storage rebooted.

11gR2 Grid Infrastructure on Solaris local containers, cssd process fails to start with following symptoms:

** root.sh output

CRS-2674: Start of 'ora.cssd' on 'racnode1' failed
..
CRS-5804: Communication error with agent process
CRS-4000: Command Start failed, or completed with errors.
The exlusive mode cluster start failed, see Clusterware alert log for more information
Initial cluster configuration failed.  See /oracle_crs/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_racnode1.log for details

** $GRID_HOME/cfgtoollogs/crsconfig/rootcrs_<nodename>.log

2017-03-09 22:33:38: CRS-2674: Start of 'ora.cssd' on 'racnode1' failed
..
2017-03-09 22:33:38: CRS-5804: Communication error with agent process
2017-03-09 22:33:38: CRS-4000: Command Start failed, or completed with errors.
2017-03-09 22:33:38: CSS startup failed with return code 1
..
2017-03-09 22:33:51: The exlusive mode cluster start failed, see Grid Infrastructure alert log for more

** $GRID_HOME/log/<nodename>/cssd/ocssd.log

2017-03-09 22:33:30.467: [    CSSD][1]clssscSetPrivEnv: Setting priority to 4
2017-03-09 22:33:30.491: [    CSSD][1]clssscSetPrivEnv: unable to set priority to 4
2017-03-09 22:33:30.491: [    CSSD][1]SLOS: cat=-2, opn=scls_set_priority_realtime, dep=1, loc=setsched
unable to escalate to real time
..
2017-03-09 22:33:30.491: [    CSSD][1](:CSSSC00011:)clssscExit: A fatal error occurred during initialization




Cause

Oracle Clusterware requires additional privileges that are not provided by default.

see below Oracle White Paper for :

RUNNING ORACLE® REAL APPLICATION CLUSTERS ON ORACLE SOLARIS ZONE CLUSTERS

http://www.oracle.com/technetwork/articles/servers-storage-admin/o11-062-rac-solariszonescluster-429206.pdf


Solution

Halt local container, execute the following on global zone and restart the local container:

<local-container-name> =============> Zone Name , you can see "zoneadm list -civ

zonecfg -z <local-container-name>
set limitpriv="default,proc_priocntl,proc_clock_highres,sys_time"
commit
exit
zoneadm -z <local-container-name> reboot

Once local container is restarted, refer to note 942166.1 or note 969254.1 to proceed.



To list Privilege Set, execute "ppriv -l <zone>" in either global or local zone.


Ref:

Grid Infrastructure cssd fails to start on Solaris local containers (zones) (Doc ID 1340694.1)

Comments

Post a Comment

Popular posts from this blog

Fatal agent error: Target Interaction Manager failed at Startup

Symptoms ========  $ /AGENT_INST/bin/emctl start agent Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved. Starting agent ......................................................................... failed. Fatal agent error: Target Interaction Manager failed at Startup Fatal agent error: Target Interaction Manager failed at Startup Fatal agent error: Target Interaction Manager failed at Startup EMAgent is Thrashing. Exiting watchdog $/AGENT_INST/sysman/log/emagent.nohup reports following error 2013-11-01 17:29:38,596 [1:main] WARN - Missing filename for log handler 'opsscfg' Agent is going down due to an OutOfMemoryError $/AGENT_INST/sysman/log/gcagent.log reports following error 2013-11-01 17:29:19,204 [1:3305B9] INFO - Invoking STARTUP_P1 (1) on Target Interaction Manager 2013-11-01 17:29:23,255 [1:main] FATAL - Fatal error: Target Interaction Manager failed at Startup java.lang.OutOfMemoryError: Java heap sp
Read more

[INS-40718] Single Client Access Name (SCAN): could not be resolved. ( LDOMS & Zones)

Cause The name you provided as the SCAN could not be resolved using TCP/IP host name lookup. Action-Provide name to use for the SCAN for which the domain can be resolved Global Zone : ============= # svccfg svc:> select name-service/switch svc:/system/name-service/switch> setprop config/host = astring: "files dns" svc:/system/name-service/switch> select system/name-service/switch:default svc:/system/name-service/switch:default> refresh svc:/system/name-service/switch:default> validate svc:/system/name-service/switch:default> exit # svcadm refresh name-service/switch # svcadm restart /system/name-service-cache; then, I can resolve host using telnet/ping.
Read more

CRS-2883: Resource 'ora.asm' failed during Clusterware stack start

Symptoms ======== root@oraprod02:~# /u01/app/12.1.0.2/grid/crs/install/rootcrs.pl -postpatch Using configuration parameter file: /u01/app/12.1.0.2/grid/crs/install/crsconfig_params 2017/11/05 15:57:38 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector. 2017/11/05 15:58:08 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector. 2017/11/05 15:58:13 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab' CRS-4123: Oracle High Availability Services has been started. CRS-4133: Oracle High Availability Services has been stopped. CRS-4123: Starting Oracle High Availability Services-managed resources CRS-2672: Attempting to start 'ora.evmd' on 'oraprod02' CRS-2672: Attempting to start 'ora.mdnsd' on 'oraprod02' CRS-2676: Start of 'ora.mdnsd' on 'oraprod02' succeeded CRS-2676: Start of 'ora.evmd' on 'oraprod02' succeeded CRS-2672: Attempting to st
Read more