how to perform a oracle database switchover to physical standby.

In my this switchover AQBLP is the primary database and AQBLPLS is the local standby.

1. Disable the crontab jobs on your primary server. I have a job for relocation of RAC services.  

crontab -e #00 8 * * * /scripts/sh/job.sh :wq!

2. Connect to the standby server and set environment and connect to dgmgrl command as sys.
   

DGMGRL> show configuration Configuration - aqblp_dg  Protection Mode: MaxPerformance  Databases: aqblp   - Primary database aqblpls - Physical standby database Fast-Start Failover: DISABLED Configuration Status: SUCCESS ======================>> make sure status is success DGMGRL> show database aqblpls Database - aqblpls  Role:         PHYSICAL STANDBY==========make sure its physical standby  Intended State:  APPLY-ON  Transport Lag:   0 seconds (computed 0 seconds ago) ======>> make sure no lag  Apply Lag:    0 seconds (computed 0 seconds ago)  Apply Rate:   3.45 MByte/s  Real Time Query: OFF  Instance(s): AQBLPLS1 AQBLPLS2 (apply instance)========>> remember what was apply instance Database Status: SUCCESS

3. Statuses have been verified it's a good time to switchover.
   
   

Command to execute to switchover is as below.  Run it on dgmgrl command line. switchover to aqblpls; ⇒ as soon as you execute stay patient. Below is an example what will you see.   DGMGRL> switchover to aqblpls; Performing switchover NOW, please wait... Operation requires a connection to instance "AQBLPLS2" on database "AQBLPLS" Connecting to instance "AQBLPLS2"... Connected. New primary database "aqblpls" is opening... Operation requires startup of instance "aqblp1" on database "aqblp" Starting instance "AQBLP2"... Unable to connect to database ORA-12514: TNS:listener does not currently know of service requested in connect descriptor Failed. Warning: You are no longer connected to ORACLE. Please complete the following steps to finish switchover:     start up and mount instance "AQBLP1" of database "AQBLP"​

4. Do not freak out with the above warning. It is saying that your old primary instance needs to be manually mounted since its standby now.
   ===================================================================At this point, you need to connect to the old primary node indicated during step 3 in switchover. Source the environment, and connect to the instance number on the "old primary" cluster, now which is standby , and start mount the instance AQBLP1 specified with sqlplus.
   ===================================================================
   

[oracle@node1 ~]$ . oraenv ORACLE_SID = [AQBLP1] ? The Oracle base remains unchanged with value /u01/app/oracle [oracle@node1 ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.4.0 Production on Fri Mar 18 22:38:10 2016 Copyright (c) 1982, 2013, Oracle.  All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options sys@aqblp1> sys@aqblp1>startup mount ORACLE instance started. Total System Global Area 2.1379E+10 bytes Fixed Size               2262656 bytes Variable Size         6912215424 bytes Database Buffers      1.4294E+10 bytes Redo Buffers           170291200 bytes Database mounted. sys@aqblp1> set pagesize 1000 sys@aqblp1> set linesize 200 sys@aqblp1> select name, database_role from v$database; NAME           DATABASE_ROLE ----------------  ----------------------------- aqblp          PHYSICAL STANDBY

5. If you have other databases go switchover other standbys, and get to this point. AT THIS POINT AQBLPLS IS PRIMARY and AQBLP is standby.
   
6. It's a good time to balance RAC services to its preferred nodes.
   

srvctl stop service -d AQBLPLS -s service1 srvctl stop service -d AQBLPLS -s service2 srvctl start service -d AQBLPLS -s service1 srvctl start service -d AQBLPLS -s service2

7. This is a good time you can release the database or databases to whoever is waiting for them. Most likely it is the app team or middleware team waiting to perform their steps as part of the switchover activity. The switchover has successfully finished here but there are still some verification steps for new standby/old primary.
8. On the new standby which will be AQBLP now. After 5-10 minutes, shutdown the instance in sqlplus.
   

SQL> shutdown immediate ORA-01109: database not open Database dismounted. ORACLE instance shut down. SQL> exit​

9. Now, dataguard broker is supposed to modify the configuration of the standby database in CRS to be a standby, but sometimes it does not seem to do it.  Check the database and verify.
   

[oracle@standbynode ~]$ srvctl config database -d AQBLP Database unique name: AQBLP Database name: AQBLP Oracle home: /u01/app/oracle/product/11.2.0.4/dbhome_1 Oracle user: oracle Spfile: +DATA/AQBLP/spfileAQBLP.ora Domain: Start options: open Stop options: immediate Database role: PRIMARY===========>>>>>Should be Physical standby. Management policy: AUTOMATIC Server pools: AQBLP Database instances: AQBLP1,AQBLP2 Disk Groups: DATA,RECO Mount point paths: Services: AQSVC1P Type: RAC Database is administrator managed

10. See, how the new standby is still registered as a primary database?  We don't want that, because if the cluster nodes are restarted, then the database will try to open as a primary and it will fail, because the database will report that the datafiles are in recovery. So, we need to update this ourselves.  Really easy.
    
11. Run the command below.
    

​srvctl modify database -d AQBLP -s mount -r PHYSICAL_STANDBY

12. Now check the config again.
    

srvctl config database -d AQBLP [oracle@dgpldb03a ~]$ srvctl config database -d AQBLP Database unique name: AQBLP Database name: AQBLP Oracle home: /u01/app/oracle/product/11.2.0.4/dbhome_1 Oracle user: oracle Spfile: +DATA/AQBLP/spfileAQBLP.ora Domain: Start options: open Stop options: immediate Database role: PHYSICAL_STANDBY======>> Looks right this time Management policy: AUTOMATIC Server pools: AQBLP Database instances: AQBLP1,AQBLP2 Disk Groups: DATA,RECO Mount point paths: Services: AQSVC1P Type: RAC Database is administrator managed

13. Now let's start the standby database and it will start as physical standby in mount mode. Since the configuration is right.
    

srvctl start database -d AQBLP​

14. Check the status of the database which should show as standby and in mount mode. It should start in mount mode now, but the key is that CRS knows it is a standby and will not try to open it.
15. Next you should verify dataguard broker is okay, for all the databases.  You will want to wait a little while for this. If you see an error like below, when executing a "show configuration verbose" wait and try in a little bit. Data guard broker takes a little time to get updates.
    

​"show configuration verbose" Configuration Status: ORA-16610: command "SWITCHOVER TO mpaepl" in progress DGM-17017: unable to determine configuration status DO NOT PANIC.  This is okay. Broker is just still trying to get everything updated.

16. After 10 minutes or so, you should see this.
    

DGMGRL> show configuration verbose Configuration - dg  Protection Mode: MaxPerformance  Databases: AQBLPLS  - Primary database AQBLP - Physical standby database  Properties: FastStartFailoverThreshold   = '30' OperationTimeout             = '30' FastStartFailoverLagLimit    = '30' CommunicationTimeout         = '180' ObserverReconnect            = '0' FastStartFailoverAutoReinstate  = 'TRUE' FastStartFailoverPmyShutdown = 'TRUE' BystandersFollowRoleChange   = 'ALL' ObserverOverride             = 'FALSE' ExternalDestination1         = '' ExternalDestination2         = '' PrimaryLostWriteAction       = 'CONTINUE' Fast-Start Failover: DISABLED Configuration Status: SUCCESS​

17. Some extra verification can be done but not mandatory:
    

1.Check primary; log into primary and execute following sql:         select name, database_role, open_mode from v$database;           -- open_mode should be read, write2 2.Checking standby; login into standby and execute following sql:         select name, database_role, open_nmode from v$database;           -- open_mode should be mounted 3.Check the MRP process that applying is on.            select process,client_process,status,thread#,sequence#,block#,blocks  from gv$managed_standby; MRP0      N/A APPLYING_LOG          8 153008 5822899 10485760

18. Enable the crontab jobs on new primary, comment out.

crontab -e 00 8 * * * /scripts/sh/job.sh :wq!

19. YOU ARE ALL DONE HERE FOR THE SWITCHOVER STEPS.