How to Downgrade 19C Grid Infrastructure to 12.1.0.2

ENVIRONMENT DETAILS:

2 NODES – ambeth01 and ambeth02
Grid owner – oragrid
19C ORACLE_HOME(current) – > /u01/app/19.0.0.0/grid
12C ORACLE_HOME (old) -> /u01/app/12.1.0.2/grid/bin

Check the current grid version:( check on both the nodes)

[oragrid@ambeth01 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [19.0.0.0.0]
[oragrid@ambeth01 ~]$ crsctl query crs softwareversion -all
Oracle Clusterware version on node [ambeth01] is [19.0.0.0.0]
Oracle Clusterware version on node [ambeth02] is [19.0.0.0.0]
[oragrid@ambeth01 ~]$


1. As the first step for downgrade, remove the MGMTDB. Move the MGMTDB database on node1, if its running on node2.

[oragrid@ambeth01 ~]$ ps -ef|grep pmon
oragrid    952 21344  0 19:38 pts/0    00:00:00 grep --color=auto pmon
oragrid   8102     1  0 14:32 ?        00:00:02 asm_pmon_+ASM1
oragrid   8609     1  0 14:33 ?        00:00:02 apx_pmon_+APX1
oracle    9025     1  0 14:33 ?        00:00:03 ora_pmon_depuat1
[oragrid@ambeth01 ~]$ srvctl relocate mgmtdb -node ambeth01
[oragrid@ambeth01 ~]$ ps -ef|grep pmon
oragrid   1560     1  0 19:40 ?        00:00:00 mdb_pmon_-MGMTDB
oragrid   2449 21344  0 19:41 pts/0    00:00:00 grep --color=auto pmon
oragrid   8102     1  0 14:32 ?        00:00:02 asm_pmon_+ASM1
oragrid   8609     1  0 14:33 ?        00:00:02 apx_pmon_+APX1
oracle    9025     1  0 14:33 ?        00:00:03 ora_pmon_depuat1
[oragrid@ambeth01 ~]$


[oragrid@ambeth01 ~]$ dbca -silent -deleteDatabase -sourceDB -MGMTDB
[WARNING] [DBT-19202] The Database Configuration Assistant will delete the Oracle instances and datafiles for your database. All information in the database will be destroyed.
Prepare for db operation
32% complete
Connecting to database
35% complete
39% complete
42% complete
45% complete
48% complete
52% complete
65% complete
Updating network configuration files
68% complete
Deleting instance and datafiles
84% complete
100% complete
Database deletion completed.
Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb1.log" for further details.
[oragrid@ambeth01 ~]$


2. Run rootcrs.sh with downgrade option from any of the nodes. I sugest to run this command on node2 first then node1.

[root@ambeth02 ~]# /u01/app/19.0.0.0/grid/crs/install/rootcrs.sh -downgrade
Using configuration parameter file: /u01/app/19.0.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/ambeth02/crsconfig/crsdowngrade_ambeth02_2020-05-03_08-01-44PM.log
2020/05/03 20:11:11 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.
2020/05/03 20:12:07 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.
2020/05/03 20:12:08 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node
[root@ambeth02 ~]# 2020/05/03 20:13:28 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.


3. Run the downgrade from the last node.

[root@ambeth01 ~]# /u01/app/19.0.0.0/grid/crs/install/rootcrs.sh -downgrade
Using configuration parameter file: /u01/app/19.0.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/ambeth01/crsconfig/crsdowngrade_ambeth01_2020-05-03_08-43-33PM.log
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'ambeth01'
CRS-2672: Attempting to start 'ora.mdnsd' on 'ambeth01'
CRS-2676: Start of 'ora.mdnsd' on 'ambeth01' succeeded
CRS-2676: Start of 'ora.evmd' on 'ambeth01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'ambeth01'
CRS-2676: Start of 'ora.gpnpd' on 'ambeth01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ambeth01'
CRS-2672: Attempting to start 'ora.gipcd' on 'ambeth01'
CRS-2676: Start of 'ora.cssdmonitor' on 'ambeth01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'ambeth01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'ambeth01'
CRS-2672: Attempting to start 'ora.diskmon' on 'ambeth01'
CRS-2676: Start of 'ora.diskmon' on 'ambeth01' succeeded
CRS-2676: Start of 'ora.cssd' on 'ambeth01' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'ambeth01'
CRS-2672: Attempting to start 'ora.ctssd' on 'ambeth01'
CRS-2676: Start of 'ora.crf' on 'ambeth01' succeeded
CRS-2676: Start of 'ora.ctssd' on 'ambeth01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'ambeth01'
CRS-2676: Start of 'ora.asm' on 'ambeth01' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'ambeth01'
CRS-2676: Start of 'ora.storage' on 'ambeth01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'ambeth01'
CRS-2676: Start of 'ora.crsd' on 'ambeth01' succeeded
2020/05/03 21:00:17 CLSRSC-338: Successfully downgraded OCR to version 12.1.0.2.0
CRS-2672: Attempting to start 'ora.crf' on 'ambeth01'
CRS-2676: Start of 'ora.crf' on 'ambeth01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'ambeth01'
CRS-2676: Start of 'ora.crsd' on 'ambeth01' succeeded
2020/05/03 21:04:38 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.
2020/05/03 21:05:36 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.
2020/05/03 21:05:39 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node
2020/05/03 21:05:40 CLSRSC-640: To complete the downgrade operation, ensure that the node inventory on all nodes points to the configured Grid Infrastructure home '/u01/app/12.1.0.2/grid'.
2020/05/03 21:05:42 CLSRSC-592: Run 'crsctl start crs' from home /u01/app/12.1.0.2/grid on each node to complete downgrade.
[root@ambeth01 ~]# 2020/05/03 21:07:03 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.


4. Update the oracle inventory by removing the CRS association from 19c GI Home and adding it to 12.1.0.2 GI Home( only from one node).

[root@ambeth01 ~]# su - oragrid
Last login: Sun May  3 21:14:19 +03 2020
[oragrid@ambeth01 ~]$ cd /u01/app/19.0.0.0/grid/oui/bin
[oragrid@ambeth01 bin]$ pwd
/u01/app/19.0.0.0/grid/oui/bin
[oragrid@ambeth01 bin]$ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=false ORACLE_HOME=/u01/app/19.0.0.0/grid "CLUSTER_NODES=ambeth01,ambeth02" -doNotUpdateNodeList
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 16383 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
You can find the log of this install session at:
 /u01/app/oraInventory/logs/UpdateNodeList2020-05-03_09-26-06PM.log
'UpdateNodeList' was successful.
[oragrid@ambeth01 bin]$

[oragrid@ambeth01 bin]$ cd /u01/app/12.1.0.2/grid/oui/bin
[oragrid@ambeth01 bin]$ pwd
/u01/app/12.1.0.2/grid/oui/bin
[oragrid@ambeth01 bin]$ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true ORACLE_HOME=/u01/app/12.1.0.2/grid "CLUSTER_NODES=ambeth01,ambeth02"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 16383 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.
[oragrid@ambeth01 bin]$


Verify the update on inventory.xml

<HOME NAME="OraGI12Home1" LOC="/u01/app/12.1.0.2/grid" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="ambeth01"/>
      <NODE NAME="ambeth02"/>
   </NODE_LIST>
</HOME>


<HOME NAME="OraGI19Home1" LOC="/u01/app/19.0.0.0/grid" TYPE="O" IDX="3">
   <NODE_LIST>
      <NODE NAME="ambeth01"/>
      <NODE NAME="ambeth02"/>
   </NODE_LIST>
</HOME>


5. If the /etc/oratab is missing any ASM references with downgraded 12.1 GI home, add the entries manually.


### Start crs for Node1

[oragrid@ambeth01 bin]$ cat /etc/oratab
~~~~
~~~~
depuat:/u02/app/oracle/product/12.1.0.2/dbhome_1:N              # line added by Agent
+ASM1:/u01/app/12.1.0.2/grid:N          # line added by Agent
[oragrid@ambeth01 bin]$ exit
logout
[root@ambeth01 ~]# cd
[root@ambeth01 ~]#
[root@ambeth01 ~]#
[root@ambeth01 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/grid
[root@ambeth01 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@ambeth01 ~]#


### Start crs for Node2:

[root@ambeth02 ~]# cat /etc/oratab
~~~
~~~
depuat:/u02/app/oracle/product/12.1.0.2/dbhome_1:N              # line added by Agent
[root@ambeth02 ~]# vi /etc/oratab
[root@ambeth02 ~]# cat /etc/oratab
~~~~
~~~
depuat:/u02/app/oracle/product/12.1.0.2/dbhome_1:N              # line added by Agent
+ASM2:/u01/app/12.1.0.2/grid:N          # line added by Agent
[root@ambeth02 ~]# . oraenv
ORACLE_SID = [root] ? +ASM2
The Oracle base has been set to /u01/app/grid
[root@ambeth02 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@ambeth02 ~]#


### Validate Cluster:

[oragrid@ambeth01 ~]$ olsnodes
ambeth01
ambeth02
[oragrid@ambeth01 ~]$ crsctl check cluster -all
**************************************************************
ambeth01:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
ambeth02:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[oragrid@ambeth01 ~]$ crsctl query crs softwareversion
Oracle Clusterware version on node [ambeth01] is [12.1.0.2.0]
[oragrid@ambeth01 ~]$ crsctl query crs softwarepatch
Oracle Clusterware patch level on node ambeth01 is [182379635].
[oragrid@ambeth01 ~]$ crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [12.1.0.2.0]
[oragrid@ambeth01 ~]$ crsctl query crs releasepatch
Oracle Clusterware release patch level is [182379635] and the complete list of patches [20243804 20415006 20594149 20788771 20950328 21125181 21359749 21527488 21694919 21949015 22806133 23144544 24340679 24732088 25397136 25869760 26609798 26717470 26925263 26983807 27338029 27547374 28259867 28731800 29141038 29496791 29972716 30304402 30304434 30364137 ] have been applied on the local node.
[oragrid@ambeth01 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [182379635].
[oragrid@ambeth01 ~]$
   
[oragrid@ambeth02 ~]$ crsctl query crs softwareversion
Oracle Clusterware version on node [ambeth02] is [12.1.0.2.0]
[oragrid@ambeth02 ~]$ crsctl query crs softwarepatch
Oracle Clusterware patch level on node ambeth02 is [182379635].
[oragrid@ambeth02 ~]$ crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [12.1.0.2.0]
[oragrid@ambeth02 ~]$ crsctl query crs releasepatch
Oracle Clusterware release patch level is [182379635] and the complete list of patches [20243804 20415006 20594149 20788771 20950328 21125181 21359749 21527488 21694919 21949015 22806133 23144544 24340679 24732088 25397136 25869760 26609798 26717470 26925263 26983807 27338029 27547374 28259867 28731800 29141038 29496791 29972716 30304402 30304434 30364137 ] have been applied on the local node.
[oragrid@ambeth02 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [182379635].
[oragrid@ambeth02 ~]$


6. Deinstall & Detach the 19c Grid Home.

### Detach Grid 19c Home from inventory: ( onle on node1)

    [oracle@ol7-121-rac1 ~]$ /u01/app/19.0.0.0/grid/oui/bin/runInstaller -detachHome -silent ORACLE_HOME=/u01/app/19.0.0.0/grid
    Starting Oracle Universal Installer...
    Checking swap space: must be greater than 500 MB.   Actual 18431 MB    Passed
    The inventory pointer is located at /etc/oraInst.loc
    You can find the log of this install session at:
     /u01/app/oraInventory/logs/DetachHome2019-12-10_12-24-09AM.log
    'DetachHome' was successful.
    [oracle@ol7-121-rac1 ~]$


### Remove Grid 19c directory for Node1:

    [root@ambeth01 ~]# cd /u01/app/19.0.0.0/
    [root@ambeth01 19.0.0.0]# pwd
    /u01/app/19.0.0.0
    [root@ambeth01 19.0.0.0]# rm -rf grid
    [root@ambeth01 19.0.0.0]# mkdir grid
    [root@ambeth01 19.0.0.0]# chown -R oragrid:oinstall grid
    [root@ambeth01 19.0.0.0]# ls -lrt
    total 4
    drwxr-xr-x 2 oragrid oinstall 4096 May  3 21:50 grid
    [root@ambeth01 19.0.0.0]#


### Remove Grid 19c directory for Node2:

    [root@ambeth02 ~]# cd /u01/app/19.0.0.0/
    [root@ambeth02 19.0.0.0]# pwd
    /u01/app/19.0.0.0
    [root@ambeth02 19.0.0.0]# rm -rf grid
    [root@ambeth02 19.0.0.0]# mkdir grid
    [root@ambeth02 19.0.0.0]# chown -R oragrid:oinstall grid
    [root@ambeth02 19.0.0.0]# ls -lrt
    total 4
    drwxr-xr-x 2 oragrid oinstall 4096 May  3 21:51 grid
    [root@ambeth02 19.0.0.0]#


7. On any node, remove the MGMTDB resource as follows:

    [oragrid@ambeth01 ~]$ srvctl remove mgmtdb
    Remove the database _mgmtdb? (y/[n]) y
    [oragrid@ambeth01 ~]$


8. Create the MGMTDB in silent mode using the templates

    If you are downgrading to Oracle Grid Infrastructure 12c Release 1 (12.1.0.2), run the following commands to configure the Grid Infrastructure Management Database:

    8.1. Run DBCA in the silent mode from the 12.1.0.2 Oracle home and create the Management Database container database (CDB) as follows:

    cd /u01/app/12.1.0.2/grid/bin
    ./dbca -silent -createDatabase -sid -MGMTDB -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -gdbName _mgmtdb -storageType ASM -diskGroupName +DATA -datafileJarLocation /u01/app/12.1.0.2/grid/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck

    8.2. Run DBCA in the silent mode from the 12.1.0.2 Oracle home and create the Management Database pluggable database (PDB) as follows:

    cd /u01/app/12.1.0.2/grid/bin
    ./dbca -silent -createPluggableDatabase -sourceDB -MGMTDB -pdbName ambeth_scan -createPDBFrom RMANBACKUP -PDBBackUpfile  /u01/app/12.1.0.2/grid/assistants/dbca/templates/mgmtseed_pdb.dfb -PDBMetadataFile  /u01/app/12.1.0.2/grid/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true -internalSkipGIHomeCheck

[oragrid@ambeth01 ~]$ cd /u01/app/12.1.0.2/grid/bin
[oragrid@ambeth01 bin]$ ./dbca -silent -createDatabase -sid -MGMTDB -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -gdbName _mgmtdb -storageType ASM -diskGroupName +DATA -datafileJarLocation /u01/app/12.1.0.2/grid/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck
Registering database with Oracle Grid Infrastructure
5% complete
Copying database files
7% complete
9% complete
16% complete
23% complete
30% complete
37% complete
41% complete
Creating and starting Oracle instance
43% complete
48% complete
49% complete
50% complete
55% complete
60% complete
61% complete
64% complete
Completing Database Creation
68% complete
79% complete
89% complete
100% complete
Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb2.log" for further details.
[oragrid@ambeth01 bin]$

[oragrid@ambeth01 ~]$ cd /u01/app/12.1.0.2/grid/bin
[oragrid@ambeth01 bin]$ ./dbca -silent -createPluggableDatabase -sourceDB -MGMTDB -pdbName ambeth-scan -createPDBFrom RMANBACKUP -PDBBackUpfile  /u01/app/12.1.0.2/grid/assistants/dbca/templates/mgmtseed_pdb.dfb -PDBMetadataFile  /u01/app/12.1.0.2/grid/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true -internalSkipGIHomeCheck
Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb.log" for further details.
[oragrid@ambeth01 bin]$ vi /u01/app/grid/cfgtoollogs/dbca/_mgmtdb.log
[oragrid@ambeth01 bin]$ ./dbca -silent -createPluggableDatabase -sourceDB -MGMTDB -pdbName ambeth_scan -createPDBFrom RMANBACKUP -PDBBackUpfile  /u01/app/12.1.0.2/grid/assistants/dbca/templates/mgmtseed_pdb.dfb -PDBMetadataFile  /u01/app/12.1.0.2/grid/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true -internalSkipGIHomeCheck
Creating Pluggable Database
4% complete
12% complete
21% complete
38% complete
55% complete
85% complete
Completing Pluggable Database Creation
100% complete
Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/ambeth_scan/_mgmtdb0.log" for further details.
[oragrid@ambeth01 bin]$


Useful metalink notes
How to Upgrade to/Downgrade from Grid Infrastructure 12.2 and Known Issues [ID 2240959.1]