Create Baseline to change execution plan hash value

Issue:

When a query is having a new plan which is worst and you found by SQL tuning Advisor there is another plan which is having better execution time.














       

There is a better plan in AWR as in above Fig.

Steps to change plan hash value.

Step1:  Create SQL tuning set with SQL_ID attached.

Drop if same name exists before
 exec dbms_sqltune.DROP_SQLSET('sqlid_1vktnmmbp6cz3');

Create new one as follows.


 exec DBMS_SQLTUNE.CREATE_SQLSET('sqlid_1vktnmmbp6cz3');


Step2: Load plan  you want for the SQL ID , replace sql_id and plan hash value you want in following blocks.

Following will extract the plan from awr.

declare
 baseline_ref_cursor DBMS_SQLTUNE.SQLSET_CURSOR;
 begin
 open baseline_ref_cursor for
 select VALUE(p) from table(DBMS_SQLTUNE.SELECT_WORKLOAD_REPOSITORY(1, 128,'sql_id='||CHR(39)||'1vktnmmbp6cz3'||CHR(39)||' and plan_hash_value=487911821',NULL,NULL,NULL,NULL,NULL,NULL,'ALL')) p;
 DBMS_SQLTUNE.LOAD_SQLSET('sqlid_1vktnmmbp6cz3', baseline_ref_cursor);
 end;
 /
 above when executed will throw out correct snapshot ids as follows

 08:35:15 SQL> declare
 08:35:34   2   baseline_ref_cursor DBMS_SQLTUNE.SQLSET_CURSOR;
 08:35:34   3   begin
 08:35:34   4   open baseline_ref_cursor for
 08:35:34   5   select VALUE(p) from table(DBMS_SQLTUNE.SELECT_WORKLOAD_REPOSITORY(1, 128,'sql_id='||CHR(39)||'1vktnmmbp6cz3'||CHR(39)||' and plan_hash_value=487911821',NULL,NULL,NULL,NULL,NULL,NULL,'ALL')) p;
 08:35:34   6   DBMS_SQLTUNE.LOAD_SQLSET('sqlid_1vktnmmbp6cz3', baseline_ref_cursor);
 08:35:34   7   end;
 08:35:34   8   /
 declare
 *
 ERROR at line 1:
 ORA-13768: Snapshot ID must be between 83431 and 93052.
 ORA-06512: at "SYS.DBMS_SQLTUNE", line 4715
 ORA-06512: at line 6

Now replace start and end snap ids as follows

declare
 baseline_ref_cursor DBMS_SQLTUNE.SQLSET_CURSOR;
 begin
 open baseline_ref_cursor for
 select VALUE(p) from table(DBMS_SQLTUNE.SELECT_WORKLOAD_REPOSITORY(83431, 93052,'sql_id='||CHR(39)||'1vktnmmbp6cz3'||CHR(39)||' and plan_hash_value=487911821',NULL,NULL,NULL,NULL,NULL,NULL,'ALL')) p;
 DBMS_SQLTUNE.LOAD_SQLSET('sqlid_1vktnmmbp6cz3', baseline_ref_cursor);
 end;
 /

 


 ***************** OR ***********************

Following to get plan from cursor cache.


 declare
  baseline_ref_cursor DBMS_SQLTUNE.SQLSET_CURSOR;
  begin
  open baseline_ref_cursor for
  select value(p) from table(dbms_sqltune.select_cursor_cache('sql_id=''1vktnmmbp6cz3'' and plan_hash_value=487911821',NULL,NULL,NULL,NULL,1,NULL,'ALL')) p;
  DBMS_SQLTUNE.LOAD_SQLSET('sqlid_1vktnmmbp6cz3', baseline_ref_cursor);
  end;
 /

Step3: Queries to check plan.

SELECT NAME,OWNER,CREATED,STATEMENT_COUNT FROM DBA_SQLSET where name='sqlid_1vktnmmbp6cz3';

 select sql_id, substr(sql_text,1, 15) text
 from dba_sqlset_statements
 where sqlset_name = 'sqlid_1vktnmmbp6cz3'
 order by sql_id;

 SELECT * FROM table (DBMS_XPLAN.DISPLAY_SQLSET('sqlid_1vktnmmbp6cz3','1vktnmmbp6cz3'));

select *  from dba_sql_plan_baselines;

 Step4: Create baseline and fix it.


 set serveroutput on
 declare
 my_integer pls_integer;
 begin
 my_integer := dbms_spm.load_plans_from_sqlset(sqlset_name => 'sqlid_1vktnmmbp6cz3',
                                               sqlset_owner => 'SYSTEM',
                                               fixed => 'YES',
                                               enabled => 'YES');
                                               DBMS_OUTPUT.PUT_line(my_integer);
 end;
 /

Copy files from ASM diskgroup to remote ASM diskgroup.

ASMCMD> cp  +DATA_XDB4/SRCDB/PARAMETERFILE/spfile.3151.869812217  sys@10.112.39.14.+ASM2:+RECO_XDB5/TGTDB/test/spfile_sk.ora  --port 2483
Enter password: *******

10.112.39.14=remote host IP make sure this is the IP where +ASM2 listener listening to.
+ASM2= remote ASM instance

--port 2483 = port number




To debug
export DBI_TRACE=1

then

ASMCMD> cp  +DATA_XDB4/SRCDB/PARAMETERFILE/spfile.3151.869812217  sys@10.112.39.14.+ASM2:+RECO_XDB5/TGTDB/test/spfile_sk.ora  --port 2483
Enter password: *******

Shell script to read list of databases from a file.

while read line;
do
DBNAME=`echo $line |awk {'print $1'}`
TBS_THRESHOLD=`echo $line |awk {'print $2'}`
DBA=`echo $line |awk {'print $3'}`
RMANCAT=`echo $line |awk {'print $4'}`
INSTANCES=`echo $line |awk {'print $5'}`
. /orahome/work/sk/dbmon.sh $DBNAME
. /orahome/work/sk/tbsmon.sh $DBNAME $TBS_THRESHOLD
. /orahome/work/sk/dbimon.sh $DBNAME
done < /orahome/work/sk/Listofdatabases.txt

Above script will execute following for every db listed in /orahome/work/sk/Listofdatabases.txt

. /orahome/work/sk/dbmon.sh $DBNAME
. /orahome/work/sk/tbsmon.sh $DBNAME $TBS_THRESHOLD
. /orahome/work/sk/dbimon.sh $DBNAME

 <@dbax1:/orahome/work/sk>cat Listofdatabases.txt
db1 15 SK rmandbq 2
db2 15 SK rmandbq 2
db3 15 SK rmandbq 2
db4 15 SK rmandbq 2

db1,db2..are database names in tnsnames.ora file.

RMAN restore of few tablespaces in DEV from backup of Prod.

Create a dummy temporary instance DEVDB and restore the required tablespaces skipping all other tablespaces not required(DO NOT skip system,SYSAUX,TEMP,UNDO) during restore it will check dependencies of tablespace.

copy init parameter from existing DEVDB1 and create DEVDB dummy instance and change parameters accordingly to the required locations.

*.DB_FILE_NAME_CONVERT='/oradata/PRODDB/','/orabackup/DEVDB'
*.LOG_FILE_NAME_CONVERT='/oralogs/PRODDB/','/oralogs/DEVDB/'
*.db_create_file_dest='/orabackup/DEVDB/'
*.DB_CREATE_ONLINE_LOG_DEST_1='/orabackup/DEVDB/'
*.DB_CREATE_ONLINE_LOG_DEST_2='/orabackup/DEVDB/'
 *.control_files='/orabackup/DEVDB/control01.ctl','/orabackup/DEVDB/control02.ctl'
*.db_name='DEVDB'

Add tns entries in dev server for  rmancatalogdb,DEVDB,productionDB
Add listener.ora entries for dummy DB and reload.

SID_LIST_LISTENER =
  (SID_LIST =
      (SID_DESC =
      (GLOBAL_DBNAME = DEVDB)
      (ORACLE_HOME = /orahome/oracle/product/11.2.0.3/dbhome_1)
      (SID_NAME = DEVDB)
    )
  )


This restore is from BACKUP_HOST=DDBOOSTSERVER



DEVDB is a dummy temp instance

$sqlplus "/ as sysdba"
SQL> startup nomount
  
$rman log=rmanout.log

connect catalog rman@rmancatalogdb
connect auxiliary sys@DEVDB
connect target sys@productionDB
run
    {
    set UNTIL TIME "to_date('02/06/2015 06:18:00','MM/DD/YYYY HH24:MI:SS')";
    allocate AUXILIARY CHANNEL c1 DEVICE TYPE 'SBT_TAPE' connect sys@DEVDB  PARMS 'BLKSIZE=1048576,SBT_LIBRARY=/orahome/oracle/product/11.2.0.3/dbhome_1/lib/libddobk.so,ENV=(STORAGE_UNIT=Oracle_Production,BACKUP_HOST=DDBOOSTSERVER,ORACLE_HOME=/orahome/oracle/product/11.2.0.3/dbhome_1)';
    allocate AUXILIARY CHANNEL c2 DEVICE TYPE 'SBT_TAPE' connect sys@DEVDB  PARMS 'BLKSIZE=1048576,SBT_LIBRARY=/orahome/oracle/product/11.2.0.3/dbhome_1/lib/libddobk.so,ENV=(STORAGE_UNIT=Oracle_Production,BACKUP_HOST=DDBOOSTSERVER,ORACLE_HOME=/orahome/oracle/product/11.2.0.3/dbhome_1)';
    allocate AUXILIARY CHANNEL c3 DEVICE TYPE 'SBT_TAPE' connect sys@DEVDB  PARMS 'BLKSIZE=1048576,SBT_LIBRARY=/orahome/oracle/product/11.2.0.3/dbhome_1/lib/libddobk.so,ENV=(STORAGE_UNIT=Oracle_Production,BACKUP_HOST=DDBOOSTSERVER,ORACLE_HOME=/orahome/oracle/product/11.2.0.3/dbhome_1)';
    allocate AUXILIARY CHANNEL c4 DEVICE TYPE 'SBT_TAPE' connect sys@DEVDB  PARMS 'BLKSIZE=1048576,SBT_LIBRARY=/orahome/oracle/product/11.2.0.3/dbhome_1/lib/libddobk.so,ENV=(STORAGE_UNIT=Oracle_Production,BACKUP_HOST=DDBOOSTSERVER,ORACLE_HOME=/orahome/oracle/product/11.2.0.3/dbhome_1)';
DUPLICATE TARGET DATABASE TO DEVDB skip tablespace     GSGCAPPBKP_DATA,GSGCAPPBKP_INDX,GSGCPUB_DATA,GSGCPUB_INDX,GSIRAPP_DATA,GSIRAPP_INDX,USERS;
}


If this is from DISK then use following.


connect catalog rman@rmancatalogdb
connect auxiliary sys@DEVDB
connect target sys@productionDB
run
    {
    set UNTIL TIME "to_date('02/06/2015 06:18:00','MM/DD/YYYY HH24:MI:SS')";
    allocate AUXILIARY CHANNEL c1 DEVICE TYPE DISK;
    allocate AUXILIARY CHANNEL c2 DEVICE TYPE DISK;
    allocate AUXILIARY CHANNEL c3 DEVICE TYPE DISK;
    allocate AUXILIARY CHANNEL c4 DEVICE TYPE DISK;
DUPLICATE TARGET DATABASE TO DEVDB skip tablespace     GSGCAPPBKP_DATA,GSGCAPPBKP_INDX,GSGCPUB_DATA,GSGCPUB_INDX,GSIRAPP_DATA,GSIRAPP_INDX,USERS;
}


then export the schema from dummy temp DEVDB and import into DEVDB1

Key command:

DUPLICATE TARGET DATABASE TO DEVDB skip tablespace     GSGCAPPBKP_DATA,GSGCAPPBKP_INDX,GSGCPUB_DATA,GSGCPUB_INDX,GSIRAPP_DATA,GSIRAPP_INDX,USERS;

Oracle Database restore from backup using Duplicate RMAN.

Assumptions

There is a full backup level0 with control file and spfile in +DATA/ocdb/rmandata, it can be any location not necessarily ASM

Restoring backup onto database ocdb, db_name=ocdb
Backup belongs to OCDB database.
Incarnation is set to correct one.

STEP1

Startup new instance with following



db_name=ocdb
CONTROL_FILES='+DATA','+RECO'
db_create_file_dest='+DATA'
DB_CREATE_ONLINE_LOG_DEST_1='+RECO'
DB_CREATE_ONLINE_LOG_DEST_2='+RECO'
DB_RECOVERY_FILE_DEST='+RECO'


startup nomount pfile='/tmp/init.ora'




STEP2

Run following to restore database untill time from old backup.

connect auxiliary  /
run
 {
 ALLOCATE AUXILIARY CHANNEL ch1 DEVICE TYPE disk;
 ALLOCATE AUXILIARY CHANNEL ch2 DEVICE TYPE disk;
 ALLOCATE AUXILIARY CHANNEL ch3 DEVICE TYPE disk;
 ALLOCATE AUXILIARY CHANNEL ch4 DEVICE TYPE disk;
 ALLOCATE AUXILIARY CHANNEL ch5 DEVICE TYPE disk;
 ALLOCATE AUXILIARY CHANNEL ch6 DEVICE TYPE disk;
 ALLOCATE AUXILIARY CHANNEL ch7 DEVICE TYPE disk;
 ALLOCATE AUXILIARY CHANNEL ch8 DEVICE TYPE disk;
 ALLOCATE AUXILIARY CHANNEL ch9 DEVICE TYPE disk;
 ALLOCATE AUXILIARY CHANNEL ch10 DEVICE TYPE disk;
 ALLOCATE AUXILIARY CHANNEL ch11 DEVICE TYPE disk;
 ALLOCATE AUXILIARY CHANNEL ch12 DEVICE TYPE disk;
 ALLOCATE AUXILIARY CHANNEL ch13 DEVICE TYPE disk;
 ALLOCATE AUXILIARY CHANNEL ch14 DEVICE TYPE disk;
 ALLOCATE AUXILIARY CHANNEL ch15 DEVICE TYPE disk;
 ALLOCATE AUXILIARY CHANNEL ch16 DEVICE TYPE disk;
 DUPLICATE DATABASE TO ocdb
 BACKUP LOCATION '+DATA/ocdb/rmandata'
  until time "to_date ('15-10-2014 21:00:00', 'DD-MM-YYYY HH24:MI:SS')"
  NOFILENAMECHECK;
}

Active Duplicate using RMAN.

Two big disadvantages of the ACTIVE database duplication method are:

    Negative performance impact on the source database.
    High network traffic on the connection between the source and target databases.



STEP1

On TARGET database server.

change following parameters in init parameter file specific to target database

alter system set db_name=targetdb scope=spfile;
alter system set cluster_database=false scope=spfile;
alter system set db_create_file_dest='+DATA';
alter system set db_create_online_log_dest_1='+REDO';
alter system set db_create_online_log_dest_2='+REDO';


startup nomount pfile='/tmp/init.ora'

STEP2

On TARGET database server,change listener.ora make sure ORACLE_HOME is there in the file.

SID_LIST_LISTENER =
  (SID_LIST =
    )
    (SID_DESC =
      (SID_NAME = targetdb1)
      (ORACLE_HOME = /apps/oracle/product/11.2.0/db_1)
      (GLOBAL_DBNAME = targetdb)
    )
  )

and do lsnrctl status, lsnrctl reload


STEP3

Add following entries to both tnsnames.ora files on both SOURCE and TARGET

SOURCE =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = sourceserv)(PORT = 1521))
      (CONNECT_DATA =
        (SERVER = DEDICATED)
        (SERVICE_NAME = source)
      )
    )
  
  target =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = targetserv)(PORT = 1521))
      (CONNECT_DATA =
        (SERVER = DEDICATED)
        (SERVICE_NAME = target)
      )
  )
 
   
 
STEP4

Create password file on both SOURCE and TARGET
If the password file already exists on source just copy it to TARGET $ORACLE_HOME/dbs


STEP5

Test connectivity to both SOURCE and TARGET from both servers using TNS entries created in STEP3 and
make sure it works


sqlplus sys/sys@target as sysdba
sqlplus sys/sys@source as sysdba



STEP6

On Auxillary host(TARGET) start RMAN and run DUPLICATE

Here very important auxillary connection is TARGET and connection target is SOURCE



$export ORACLE_SID=HRPRD1

$rman target sys/sys@SOURCE auxiliary sys/sys@TARGET
RMAN>run{
         DUPLICATE target DATABASE TO TARGET------> (the first target is the connection target and the second is db name of target database.)
         FROM ACTIVE DATABASE;
        }

View data in a datafile.

SQL> create tablespace testts datafile '/tmp/testts_01.dbf' size 1M;

SQL> create table mythbuster1 (col1 varchar2(200)) tablespace testts;

Insert a row:

SQL> insert into mythbuster1 values (‘ORIGINAL_VALUE’);
SQL> Commit; 

$ strings /tmp/testts_01.dbf
}|{z
-N?pD112D2
TESTTS
 1j)
 w>!
ORIGINAL_VALUE
 

Wait event.

The server process then identifies the block the row exists in. After 
the database instance just came up the buffer cache is empty and the 
block will not be found. Therefore the server process issues a read call
 from the datafile for that specific block. The block is read from the 
disk to the buffer cache.

Until the loading of the block from the disk 
to the buffer cache is complete, 

the session waits with the event – db file scattered read. 

Two Task Architecture 

 There are two tasks – the 
user task that a regular user has written and the server task that 
performs the database operations. This is an important concept 
established during the early foundations of the Oracle database to 
protect the database from errant code in the user task introduced either
 maliciously or inadvertently.
 

usertask=user processes=Java program, a Pro*C code, SQL*Plus process

server task=server processes=oracleinstance 

Oracle online redologs resize with Oracle Dataguard in place.

Step1

ON PRIMARY

check primary

select status,instance_name,database_role from v$database,v$instance;  

check current size of online redo

select THREAD# ,group#,bytes/1024/1024 from v$log ;

check current size of standby redo log

select THREAD# ,group#,bytes/1024/1024 from v$standby_log ;

Step2

ON STANDBY


check standby


select status,instance_name,database_role from v$database,v$Instance;


check current size of online redo logs


select THREAD# ,group#,bytes/1024/1024 from v$log ;

check standby redlo log size.

select THREAD# ,group#,bytes/1024/1024 from v$standby_log  ;


Step3
on standby

SET standby_file_management  to manual

show parameter standby_file_management


alter system set standby_file_management=manual sid='*' scope=both;


show parameter standby_file_management



Step4

ON PRIMARY


On the primary database, check the status of the Online Redo Logs and resize them by dropping the INACTIVE redo logs and re-creating them with the new size.

ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 50 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 51 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 52 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 53 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;


ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 54 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 55 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 56 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 57 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;


ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 58 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 59 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 60 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 61 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

alter database drop logfile group 1;

alter database drop logfile group 2;

alter database drop logfile group 3;

alter database drop logfile group 4;

alter database drop logfile group 5;

alter database drop logfile group 6;

alter database drop logfile group 7;


Step5

ON STANDBY

cancel recovery


set dg broker =flase before cancel recovery

alter database recover managed standby database cancel;



select THREAD# ,group#,bytes/1024/1024 from v$log ;



ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 50 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 51 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 52 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 53 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;




ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 54 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 55 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 56 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 57 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;


ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 58 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 59 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 60 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;

ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 61 ('+DATA1(ONLINELOG)', '+FRA(ONLINELOG)') SIZE 500M;


alter database drop logfile group 1;


Step6

Verify online redolog size on primary and dr then do following on DR

alter system set standby_file_management=auto sid='*' scope=both;




ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;


set dg broker =true

Dataguard test using flashback/restorepoint.

Pre chesk

 

 1.Verify FRA Space Availability at Primary and Standby end. – Start at 06:00PM IST – Offshore.

    SQL> SELECT file_type, sum(percent_space_used) used_space,

   100 - sum(percent_space_used) free_space,

   sum(percent_space_reclaimable) reclaimable_space,

  (100 - sum(percent_space_used)) + sum(percent_space_reclaimable) available_space

  FROM v$flash_recovery_area_usage

     Group by file_type;

 2. Run the FRA_CLEANUP to clear FRA Space.

 3. Check the FRA allocation

     SQL> Show Parameter DB_RECOVERY_FILE_DEST

4.Change the FRA allocation(If necessary)

SQL> ALTER SYSTEM SET DB_RECOVERY_FILE_DEST_SIZE=<nn>G;

SQL> ALTER SYSTEM SET DB_RECOVERY_FILE_DEST='<absolute Path>';

 

Activating the Standby as Primary

 

Issue the following command on the primary to make the standby current

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

 

Check the apply Instance on the Standby

$ dgmgrl /

DGMGRL> show database <standby database>; (Check for apply Instance)

DGMGRL> exit

 

Stop Data Guard Broker on primary.

SQL>alter system set dg_broker_start=false scope=both sid=’*’;

 

Stop Data Guard Broker on Standby

SQL>alter system set dg_broker_start=false scope=both sid=’*’;

 

Stop the Redo apply on Standby Instance (as observed in step 3)

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

 

Create the Guaranteed Restore point on the Standby Database.

SQL> CREATE RESTORE POINT BEFORE_DR_TEST GUARANTEE FLASHBACK DATABASE;

 

Issue the following command on the primary to make the standby current

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

 

Disable the redo transport to standby

                          i.      Please find the LOG_ARCHIVE_DEST_n where the service is configured for redo transport

SQL> show parameters dest;

 

                        ii.      Disable the Redo Transport

ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=DEFER scope=both sid=’*’;

 

Stop the Standby database

$ srvctl stop database –d <db_unique_name>

 

Disable all Services on Standby except the one identified for the DR

$srvctl disable service –d <db_unique_name> -s <service_name>

 

Start only first instance of Standby in mount state

$ srvctl start instance –d <db_unique_name> -i <Instance_name> -o mount

 

Activate the first instance of standby.

SQL> ALTER DATABASE ACTIVATE STANDBY DATABASE;

Remount the first Instance of standby.

SQL> STARTUP MOUNT FORCE;

 

Set the Protection mode for the first Instance of standby to ‘Maximum Performance’

SQL> ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE PERFORMANCE;

 

Open the first instance of Standby

SQL> ALTER DATABASE OPEN;

Start the service identified for the DR on the Standby

$ srvctl start service –d <unique_db_name> -s <service_name>

 

 

Converting Activated Standby as Physical Standby

Shutdown all instances of Standby

$ srvctl stop database –d <db_unique_name>

 

Startup the first Instance of Standby in mount State

$ srvctl start instance –d <db_unique_name> -i <Instance_name> -o mount

 

Restore the activated standby back to the state it was of Physical Standby

SQL> FLASHBACK DATABASE TO RESTORE POINT BEFORE_DR_TEST;

 

Convert the Activated Standby to Physical Standby

SQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY;

 

Mount the Instance and Verify the Database Role.

SQL> STARTUP MOUNT FORCE;

SQL> SELECT database_role FROM V$DATABASE;

 

Enable all the disabled Services on Standby.

$srvctl enable service –d <db_unique_name> -s <service_name>

 

Shutdown and Restart all Instances of the Database.

$ srvctl stop database –d <db_unique_name>

$ srvctl start database –d <db_unique_name>

 

Start the Redo apply on preferred instance the Standby Database

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;

 

Start the Data guard Broker on the standby.

SQL>alter system set dg_broker_start=true scope=both sid=’*’;

 

Enable the redo transport on the Primary.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_n=ENABLE;

 

Start the Data guard Broker on the Primary

SQL>alter system set dg_broker_start=true scope=both sid=’*’;

 

Verify the  primary and standby are current

On the Primary

select thread#, max(sequence#) "Last Seq Generated"

from v$archived_log val, v$database vdb

where val.resetlogs_change# = vdb.resetlogs_change#

group by thread# order by 1;

 

On the Standby

select thread#, max(sequence#) "Last Standby Seq Received"

from v$archived_log val, v$database vdb

where val.resetlogs_change# = vdb.resetlogs_change#

group by thread# order by 1;

 

select thread#, max(sequence#) "Last Standby Seq Applied"

from v$archived_log val, v$database vdb

where val.resetlogs_change# = vdb.resetlogs_change#

and val.applied='YES' group by thread# order by 1;

 

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;  (Repeat 12a and 12b to verify sync up)

 

 

Drop the Guaranteed Restore Point on the standby.

SQL> drop restore point BEFORE_DR_TEST;

 

Verify no services are up and running on the standby.

$ srvctl status service –d <unique_db_name>

 

Verify all Services up and running on the Primary.

$ srvctl status service –d <unique_db_name>

 

Restore FRA Changes on both Primary and Standby

 

Remove the standby target blackout from Grid. 

Extend Oracle 11g database to 4th node.

  Note: node1, node2 and node3 are existing nodes and node4 will be the new node

Add instance to the cluster database  on primary

1. On node4 do the following

Cd /oracle

Mkdir –p admin oper backup log(already exists)

Mkdir oppr

Mkdir –p adump dpdump hdump pfile wallet

Copy the contents of /oracle/admin/oppr/wallet from node1 to node4

Copy setdb_crs, setdb_+ASM, setdb_oppr from /oracle on node1 to /oracle on node2

2.      Make a copy of the tnsnames.ora on node1. Edit the tnsnames.ora to add the local listener for node4

For eg:

 LISTENER_ OPPR4

  (ADDRESS = (PROTOCOL = TCP)(HOST = node4_vip)(PORT = 1521))

LISTENERS_OPPR =

  (ADDRESS_LIST =

    (ADDRESS = (PROTOCOL = TCP)(HOST = node1_vip(PORT = 1521))

    (ADDRESS = (PROTOCOL = TCP)(HOST = node2_vip)(PORT = 1521))

    (ADDRESS = (PROTOCOL = TCP)(HOST = node3_vip)(PORT = 1521))

    (ADDRESS = (PROTOCOL = TCP)(HOST = node4_vip)(PORT = 1521))

  )

Make sure you are using vips and try doing tnsping

$tnsping LISTENERS_OPPR

$tnsping LISTENER_ OPPR4

3. Create undo tablespace for the 4^th instance. Do this from node1.

Create undo tablespace undotbs4 datafile ‘+DATA1’ size 2g autoextend on maxsize 32767m;

4. From node1. sqlplus into oppr (Instance1). Add redo groups and set instance level parameters

      Add redo logfiles for thread 4 on primary.
Alter database add logfile thread 4 group 7 size 100m;
Alter database add logfile thread 4 group 8 size 100m;


Note: While creating the redo logfiles on primary they will get created on standby as broker is set to true
5. From node1 sqlplus into odm01ppr(Instance1) Add standby logfiles on primary.
Alter database add standby logfile thread 4 group 31 size 100m;
Alter database add standby logfile thread 4 group 32 size 100m;
Alter database add standby logfile thread 4 group 33 size 100m;
6. Make sure that primary and standby are in  sync by switching logfiles and check if they are applied  and then make sure recovery is running on standby.
Verify online / standby  redologs for thread 4 are created on DR, if they are not created do following to create them.
To Verify:
select GROUP#,THREAD# ,BYTES/1024/1024 from v$log order by THREAD# ;
select GROUP#,THREAD# ,BYTES/1024/1024 from V$STANDBY_LOG order by THREAD# ;

To add on the standby do following:
Disable recovery:
alter database recover managed standby database cancel;
alter system set dg_broker_start=false scope=both sid='*';
alter system set standby_file_management=manual scope=both sid='*';
Add logfiles:
alter database add  logfile thread 4 group 401 size 50m;
Alter database add  logfile thread 4 group 402 size 50m;
Alter database add  logfile thread 4 group 403 size 50m;
alter database add standby logfile thread 4 group 411 size 500m;
Alter database add standby logfile thread 4 group 412 size 500m;
Alter database add standby logfile thread 4 group 413 size 500m;
Enable recovery:
alter system set standby_file_management=auto scope=both sid='*';
alter system set dg_broker_start=true scope=both sid='*';
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;

7. Make sure that primary and standby are in  sync by switching logfiles and check if they are applied  and then make sure recovery is running on standby and then only issue following command on Primary.
On primary:
Alter database enable thread 4;
Then switch logfiles.
Alter system archive log current;
Alter system checkpoint global;

8. Set instance specific parameters in the spfile
alter system set cluster_database_instances=4 scope=spfile;
alter system set undo_tablespace=UNDOTBS4 scope=spfile sid=’oppr4′;
alter system set instance_name=oppr4 scope=spfile sid=’oppr4′;
alter system set instance_number=4 scope=spfile sid=’oppr4′;
alter system set thread=4 scope=spfile sid=’oppr4′;
alter system set local_listener=LISTENER_OPPR4 scope=spfile sid=’oppr4’;

Alter system set remote_listener=LISTENERS_OPPR scope=spfile sid=’*’;

5.On node4 cd /oracle/product/11.1.0/network/admin.

Create a link for tnsnames.ora to point to tnsnames.ora on ASM home

6.Now Shutdown the database and restart the database.

Before this make sure setdb_oppr points to oppr4 and oracle password file and init file exists for oppr4

Srvctl stop database –d oppr

Srvctl add instance –d oppr –I oppr4 –n node4

Now start the database

Srvctl start database –d oppr

Modify the services to be preferred on 3 nodes and available on 4^th node. Change the reporting service if any to run only on 4^th node

Now as root run the following command

srvctl setenv nodeapps -n <new nodename>  -t ORACLE_BASE=/oracle

Recycle the instance and the listener just on the new node

Add instance to the cluster database on DR

 1. On node4 do the following

Cd /oracle

Mkdir –p admin oper backup log

Mkdir opdr

Mkdir –p adump dpdump hdump pfile wallet

Copy the contents of /oracle/admin/opdr/wallet from node1 to node4

Copy setdb_crs, setdb_+ASM, setdb_opdr from /oracle on node1 to /oracle on node4

2.      Make a copy of the tnsnames.ora on node1. Edit the tnsnames.ora to add the local listener for node4

For eg:

LISTENER_ OPDR4

  (ADDRESS = (PROTOCOL = TCP)(HOST = node4_vip)(PORT = 1521))

LISTENERS_OPDR =

  (ADDRESS_LIST =

    (ADDRESS = (PROTOCOL = TCP)(HOST = node1_vip)(PORT = 1521))

    (ADDRESS = (PROTOCOL = TCP)(HOST = node2_vip)(PORT = 1521))

    (ADDRESS = (PROTOCOL = TCP)(HOST = node3_vip)(PORT = 1521))

    (ADDRESS = (PROTOCOL = TCP)(HOST = node4_vip)(PORT = 1521))

  )

Make sure you are using vips and try doing tnsping

$tnsping LISTENERS_OPDR

$tnsping LISTENER_ OPDR4

Steps 3,4,5,6,7 need not be performed on standby as they will be created on standby when you did it on primary because broker was set to true

6. Set instance specific parameters in the spfile
alter system set cluster_database_instances=4 scope=spfile;
alter system set undo_tablespace=UNDOTBS4 scope=spfile sid=’opdr4′;
alter system set instance_name=opdr4 scope=spfile sid=’opdr4′;
alter system set instance_number=4 scope=spfile sid=’opdr4′;
alter system set thread=4 scope=spfile sid=’opdr4′;
alter system set local_listener=LISTENER_OPDR4 scope=spfile sid=’opdr4’;

Alter system set remote_listener=LISTENERS_OPDR scope=spfile sid=’*’;

7.On node4 cd /oracle/product/11.1.0/network/admin.

Create a link for tnsnames.ora to point to tnsnames.ora on ASM home

8.Now Shutdown the database and restart the database.

Before this make sure setdb_opdr points to opdr4 and oracle password file and init file exists for opdr4

Srvctl stop database –d opdr

Srvctl add instance –d opdr –I opdr4 –n node4

Now start the database

Srvctl start database –d opdr

Modify the services to be preferred on 3 nodes and available on 4^th node. Change the reporting service if any to run only on 4^th node

Now as root run the following command

srvctl setenv nodeapps -n <new nodename>  -t ORACLE_BASE=/oracle

Recycle the instance and the listener just on the new node