Tuesday, August 21, 2018

Removing a Standby Database From a Data Guard Configuration

The post shows the steps for removing a standby database from a data guard configuration and make it available for read/write access. The data guard configuration consists of two standby databases.
DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  prodcdb  - Primary database
    stbycdb  - Physical standby database
    stby2cdb - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 17 seconds ago)
The standby database named "stby2cdb" is to be removed from the configuration. This is achieved by the use of "remove database" command in the data guard broker. If "preserve destination" clause is not used then this command remove all reference to the database being removed from other databases that will remain the data guard configuration. The default behaviour is to remove all references to the removed database so simply run the remove command with the database name
DGMGRL> remove database stby2cdb;
Removed database "stby2cdb" from the configuration
The data guard configuration reflect the change.
DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  prodcdb - Primary database
    stbycdb - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 26 seconds ago)
On the alert log of the primary (prodcdb) it could be seen that log_archive_config and log_archive_destination parameter being modified. The commands to update these parameters are issue by the data guard broker.
2018-08-06T12:05:04.410188+01:00
Destination LOG_ARCHIVE_DEST_3 no longer supports SYNCHRONIZATION
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
2018-08-06T12:05:04.433444+01:00
ALTER SYSTEM SET log_archive_dest_3='' SCOPE=BOTH;
Data Guard Broker executes SQL [alter system set log_archive_config='dg_config=(prodcdb,stbycdb)']
2018-08-06T12:05:04.476398+01:00
ALTER SYSTEM SET log_archive_config='dg_config=(prodcdb,stbycdb)' SCOPE=BOTH;
2018-08-06T12:05:06.540027+01:00
ALTER SYSTEM SWITCH ALL LOGFILE start (prodcdb)
On the alert log of the remaining standby (stbycdb) it could be seen that log_archive_config parameter is updated to reflect only the databases that remain on the data guard configuration.
2018-08-06T13:38:40.813124+01:00
Data Guard Broker executes SQL [alter system set log_archive_config='dg_config=(stbycdb,prodcdb)']
2018-08-06T13:38:40.886959+01:00
ALTER SYSTEM SET log_archive_config='dg_config=(stbycdb,prodcdb)' SCOPE=BOTH;
On the alert log of the database that was removed (stby2cdb), the log_archive_config and fal_server parameter values were updated by removing any references to the database in the DG configuration.
2018-08-06T14:31:27.917074+01:00
ALTER SYSTEM SET log_archive_config='nodg_config' SCOPE=BOTH;
2018-08-06T14:31:27.927364+01:00
ALTER SYSTEM SET fal_server='' SCOPE=BOTH;
Though these DG related parameters were updated and reset all databases, the db/log file name convert related value remain the same Following output is from production DB.
DGMGRL> show database prodcdb  LogFileNameConvert
  LogFileNameConvert = '/stbycdb/,/prodcdb/,/stby2cdb/,/prodcdb/'

DGMGRL>  show database prodcdb DbFileNameConvert
  DbFileNameConvert = '/stbycdb/,/prodcdb/,/stby2cdb/,/prodcdb/'
Update DbFileNameConvert and LogFileNameConvert values on all databases that remain in DG configuration by removing any references to the removed database. The changes require a restart of the database to take effect.
DGMGRL> edit database prodcdb  set property 'LogFileNameConvert'='/stbycdb/,/prodcdb/';
Warning: ORA-16675: database instance restart required for property value modification to take effect

Property "LogFileNameConvert" updated

DGMGRL>  edit database prodcdb  set property 'DbFileNameConvert'='/stbycdb/,/prodcdb/';
Warning: ORA-16675: database instance restart required for property value modification to take effect

Property "DbFileNameConvert" updated

DGMGRL>  edit database stbycdb set property 'DbFileNameConvert'='/prodcdb/,/stbycdb/';
Warning: ORA-16675: database instance restart required for property value modification to take effect

Property "DbFileNameConvert" updated

DGMGRL>  edit database stbycdb set property 'LogFileNameConvert'='/prodcdb/,/stbycdb/';
Warning: ORA-16675: database instance restart required for property value modification to take effect

Property "LogFileNameConvert" updated
If possible carry out a switchover to test the DG configuration.
DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  prodcdb - Primary database
    stbycdb - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 54 seconds ago)

DGMGRL> switchover to stbycdb
Performing switchover NOW, please wait...
Operation requires a connection to database "stbycdb"
Connecting ...
Connected to "stbycdb"
Connected as SYSDBA.
New primary database "stbycdb" is opening...
Oracle Clusterware is restarting database "prodcdb" ...
Switchover succeeded, new primary is "stbycdb"

DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  stbycdb - Primary database
    prodcdb - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 31 seconds ago)
This conclude the removing of a standby database from a DG configuration.



The physical standby that was removed from the DG configuration could be converted to an active database open for read/write. If the DG configuration was in maximum availability or maximum protection mode then lower the protection mode of the database to maximum performance. If not opening the database will fail with LGWR complaining the following.
LGWR: Primary database is in MAXIMUM AVAILABILITY mode
LGWR: Destination LOG_ARCHIVE_DEST_1 is not serviced by LGWR
LGWR: Minimum of 1 LGWR standby database required
2018-08-06T15:01:23.198018+01:00
Errors in file /opt/app/oracle/diag/rdbms/stby2cdb/stby2cdb/trace/stby2cdb_lgwr_28545.trc:
ORA-16072: a minimum of one standby database destination is required
Run the following to lower the protection mode
SQL>  ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE PERFORMANCE;

SQL> select protection_mode,protection_level from v$database;

PROTECTION_MODE      PROTECTION_LEVEL
-------------------- --------------------
MAXIMUM PERFORMANCE  UNPROTECTED
Activate the physical standby for read/write.
ALTER DATABASE ACTIVATE PHYSICAL STANDBY DATABASE;
Reset db/log file name convert parameters. This requires restart of the database to take affect.
alter system reset db_file_name_convert scope=spfile;
alter system reset log_file_name_convert scope=spfile;
Set DG broker auto start to false
alter system set dg_broker_start=false scope=both;
If archive log deletion policy was set to DG specific policy such as "CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON ALL STANDBY" then reset it to none
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE;
The standby was managed by clusterware as part of Oracle restart configuration. The database configuration information would still be listed as physical standby and mount or read only open mode (whichever was set for standby).
srvctl config database -d stby2cdb
Database unique name: stby2cdb
...
Start options: read only
Stop options: immediate
Database role: PHYSICAL_STANDBY
Modify the database to reflect its new role and start up mode
srvctl modify database -d stby2cdb -role primary -startoption open

srvctl config database -d stby2cdb
Database unique name: stby2cdb
...
Start options: open
Stop options: immediate
Database role: PRIMARY
Modify any database services that were created to start when the database was in physical standby role.
srvctl config service -d stby2cdb -s abc
Service name: abc
Cardinality: SINGLETON
Service role: PHYSICAL_STANDBY
Modify to the service configuration so it starts when the database is in primary role (if this is applicable to business needs).
srvctl modify  service -d stby2cdb -s abc -l primary

srvctl config service -d stby2cdb -s abc
Service name: abc
Cardinality: SINGLETON
Service role: PRIMARY
Shutdown the database and start it using srvctl to validate the configuration.
srvctl start database -d stby2cdb

Tuesday, August 14, 2018

Upgrading Oracle Restart from 12.2.0.1 to 18c

This post lists the steps for upgrading an Oracle restart environment (Single instance CDB on ASM) from 12.2.0.1. to 18.0.0.0 (production version 18.3.0.0). The 12.2 setup is on RHEL 7. The OS and kernel versions are as follows
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)

uname -r
3.10.0-693.el7.x86_64
The current versions of the GI are
crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [12.2.0.1.0]
crsctl query has softwareversion
Oracle High Availability Services version on the local node is [12.2.0.1.0]
The Oracle restart environment used is a role separated setup where GI is installed as grid user while Oracle software is installed Oracle user. The GI and Oracle homes had the release update (RU) 12.2.0.1.180417 applied.
Download and run orachk -u -o pre (1268927.2) on the 12.2 setup to identify any required patches before the upgrade. With the above RU applied no other patches were required be applied prior to the upgrade.
18c grid software instillation is also similar to 12.2 and is based on a image file. Before unzipping the GI image file create the 18c GI home directory and then unzip the GI image file into it.
mkdir -p /opt/app/oracle/product/18.0.0/grid
chmod 775 /opt/app/oracle/product/18.0.0
unzip LINUX.X64_180000_grid_home.zip -d /opt/app/oracle/product/18.0.0/grid
The location GI is unzipped must have at least 6.9GB free space. If not install will not proceed forward and following message is shown.
Use cluvfy with hacfg to verify pre-reqs.
cd /opt/app/oracle/product/18.0.0/grid
./runcluvfy.sh stage -pre hacfg

Verifying Physical Memory ...FAILED (PRVF-7530)
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...FAILED (PRVF-7573)
Verifying Free Space: city7s2:/usr,city7s2:/var,city7s2:/etc,city7s2:/sbin,city7s2:/tmp ...PASSED
Verifying User Existence: grid ...
  Verifying Users With Same UID: 1001 ...PASSED
Verifying User Existence: grid ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED

Pre-check for Oracle Restart configuration was unsuccessful.

Failures were encountered during execution of CVU verification request "stage -pre hacfg".

Verifying Physical Memory ...FAILED
city7s2: PRVF-7530 : Sufficient physical memory is not available on node
         "city7s2" [Required physical memory = 8GB (8388608.0KB)]

Verifying Swap Size ...FAILED
city7s2: PRVF-7573 : Sufficient swap size is not available on node "city7s2"
         [Required = 5.7752GB (6055760.0KB) ; Found = 3.7246GB (3905532.0KB)]


CVU operation performed:      stage -pre hacfg
Date:                         06-Aug-2018 18:43:55
CVU home:                     /opt/app/oracle/product/18.0.0/grid/
User:                         grid
The failure is due to lack of space for SWAP and memory (18c expect minimum 8G) and could be ignored on a test system such as this.
ASM will be upgraded as part of the upgrade process. Therefore stop the database before starting the GI upgrade.
srvctl stop database -d stby2cdb
Start the GI upgrade by running the gridSetup.sh from the grid home.
cd /opt/app/oracle/product/18.0.0/grid
./gridSetup.sh
Select upgrade GI option.
As mentioned earlier, the installer prompts to shutdown the DB since ASM will be as part of GI upgrade.
Location of the GI home cannot be changed.
As observed during cluvfy run, only the memory related pre-reqs raise warnings.
Upgrade summary page.
Run root when prompted.

Following shows the output from running the rootugprade script.
# /opt/app/oracle/product/18.0.0/grid/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/oracle/product/18.0.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/oracle/product/18.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/city7s2/crsconfig/roothas_2018-08-06_07-20-42PM.log
2018/08/06 19:20:46 CLSRSC-595: Executing upgrade step 1 of 12: 'UpgPrechecks'.
2018/08/06 19:20:51 CLSRSC-363: User ignored prerequisites during installation
2018/08/06 19:20:52 CLSRSC-595: Executing upgrade step 2 of 12: 'GetOldConfig'.
2018/08/06 19:21:01 CLSRSC-595: Executing upgrade step 3 of 12: 'GenSiteGUIDs'.
2018/08/06 19:21:01 CLSRSC-595: Executing upgrade step 4 of 12: 'SetupOSD'.
2018/08/06 19:21:01 CLSRSC-595: Executing upgrade step 5 of 12: 'PreUpgrade'.

ASM has been upgraded and started successfully.

2018/08/06 19:22:54 CLSRSC-595: Executing upgrade step 6 of 12: 'UpgradeAFD'.
2018/08/06 19:22:54 CLSRSC-595: Executing upgrade step 7 of 12: 'UpgradeOLR'.
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
2018/08/06 19:23:06 CLSRSC-595: Executing upgrade step 8 of 12: 'UpgradeOCR'.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node city7s2 successfully pinned.
2018/08/06 19:23:12 CLSRSC-595: Executing upgrade step 9 of 12: 'CreateOHASD'.
2018/08/06 19:23:16 CLSRSC-595: Executing upgrade step 10 of 12: 'ConfigOHASD'.
2018/08/06 19:23:16 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2018/08/06 19:23:50 CLSRSC-595: Executing upgrade step 11 of 12: 'UpgradeSIHA'.
CRS-4123: Oracle High Availability Services has been started.


city7s2     2018/08/06 19:25:46     /opt/app/oracle/product/18.0.0/grid/cdata/city7s2/backup_20180806_192546.olr     70732493

city7s2     2018/07/16 15:05:07     /opt/app/oracle/product/12.2.0/grid/cdata/city7s2/backup_20180716_150507.olr     2527213670
2018/08/06 19:25:46 CLSRSC-595: Executing upgrade step 12 of 12: 'InstallACFS'.
CRS-4123: Oracle High Availability Services has been started.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'city7s2'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'city7s2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2018/08/06 19:26:54 CLSRSC-327: Successfully configured Oracle Restart for a standalone server
Click OK on the execution configuration to proceed with the rest of the upgrade steps. Following shows the end of the upgrade page.
The HAS software is now upgraded to 18c.
crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [18.0.0.0.0]
crsctl query has softwareversion
Oracle High Availability Services version on the local node is [18.0.0.0.0]
If the ASM spfile was moved to $GI_HOME/dbs (described in step 12 in an earlier post) then this need to be moved to 18c GI_HOME. Spfile is not automatically moved to new 18c GI Home and remains in the 12c home.
srvctl config asm
ASM home: 
Password file: +DATA/orapwasm
Backup of Password file:
ASM listener: LISTENER
Spfile: /opt/app/oracle/product/12.2.0/grid/dbs/spfile+ASM.ora
ASM diskgroup discovery string: /dev/oracleasm/*
This could be verified by checking the ASM alert log.
Using parameter settings in server-side spfile /opt/app/oracle/product/12.2.0/grid/dbs/spfile+ASM.ora
Once the 12c GI home is removed then the ASM spfile will be lost resulting in ASM start up issues. Use asmcmd to copy the spfile to new 18c home.
ASMCMD> spget
/opt/app/oracle/product/12.2.0/grid/dbs/spfile+ASM.ora
ASMCMD> spcopy /opt/app/oracle/product/12.2.0/grid/dbs/spfile+ASM.ora /opt/app/oracle/product/18.0.0/grid/dbs/spfile+ASM.ora
ASMCMD> spset /opt/app/oracle/product/18.0.0/grid/dbs/spfile+ASM.ora
ASMCMD> spget
/opt/app/oracle/product/18.0.0/grid/dbs/spfile+ASM.ora
Check Oracle restart configuration is also updated with the new spfile location. When spfile is copied using spcopy, Oracle restart configuration gets automatically updated.
srvctl config asm
ASM home: 
Password file: +DATA/orapwasm
Backup of Password file:
ASM listener: LISTENER
Spfile: /opt/app/oracle/product/18.0.0/grid/dbs/spfile+ASM.ora
ASM diskgroup discovery string: /dev/oracleasm/*
If the ASM Spfile was located in ASM (e.g. Spfile: +DATA/ASM/ASMPARAMETERFILE/registry.253.958818889) then there are no issues to address.
Use cluvfy post check validate as follows
cluvfy stage -post hacfg

Verifying Oracle Restart Integrity ...PASSED
Verifying OLR Integrity ...PASSED

Post-check for Oracle Restart configuration was successful.


Next step is to upgrade the database software. Similar to GI home instillation, this too is based on image (different to 12.2). Create the 18c home location and unzip the DB software image file into it.
mkdir -p /opt/app/oracle/product/18.0.0/dbhome_1
unzip LINUX.X64_180000_db_home.zip -d /opt/app/oracle/product/18.0.0/dbhome_1
To begin the installation execute runInstaller from Oracle home location.
cd /opt/app/oracle/product/18.0.0/dbhome_1
./runInstaller
Select software only setup for the instillation option.
Select single database installation as this is an Oracle restart setup.
Select appropriate edition based on the licensing.
Oracle home location is fixed and the location is where the DB software was unzipped.
There are no new OS groups introduced in 18c. So all the OS groups used 12.2 are used here as well.
DB software installation summary.
When prompted run the root.sh from Oracle. Root.sh script prompts to install TFA even for single instance databases. It is also possible to install TFA later on as well.
/opt/app/oracle/product/18.0.0/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/app/oracle/product/18.0.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Do you want to setup Oracle Trace File Analyzer (TFA) now ? yes|[no] :
no
Oracle Trace File Analyzer (TFA - Non Daemon Mode) is available at :
    /opt/app/oracle/product/18.0.0/dbhome_1/suptools/tfa/release/tfa_home/bin/tfactl

Note :
1. tfactl will use TFA Daemon Mode if TFA already running in Daemon Mode and user has access to TFA
2. tfactl will configure TFA Non Daemon Mode only if user has no access to TFA Daemon mode or TFA Daemon mode is not installed

OR

Oracle Trace File Analyzer (TFA - Daemon Mode) can be installed by running this script :
    /opt/app/oracle/product/18.0.0/dbhome_1/suptools/tfa/release/tfa_home/install/roottfa.sh


After the database software is installed next step is upgrade of the database. Run preupgrade.jar available in the 18c home (18c_home/rdbms/admin) to check the upgrade readiness of the database (for more refer 2421552.1). Since this is a CDB with one PDB the preupgrade tool will generate 3 sections (CDB$ROOT, PDB$SEE and the PDB). Below is the output from the preupgrade run.
$ORACLE_HOME/jdk/bin/java -jar preupgrade.jar TERMINAL
Report generated by Oracle Database Pre-Upgrade Information Tool Version
18.0.0.0.0 on 2018-08-08T13:22:45

Upgrade-To version: 18.0.0.0.0

=======================================
Status of the database prior to upgrade
=======================================
      Database Name:  PRODCDB
     Container Name:  CDB$ROOT
       Container ID:  1
            Version:  12.2.0.1.0
         Compatible:  12.2.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  26
  Database log mode:  ARCHIVELOG
           Readonly:  FALSE
            Edition:  EE

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  Real Application Clusters              [to be upgraded]  OPTION OFF
  Oracle Workspace Manager               [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID

==============
BEFORE UPGRADE
==============

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
  1.  Review and remove any unnecessary HIDDEN/UNDERSCORE parameters.

      The database contains the following initialization parameters whose name
      begins with an underscore:

      _rac_dbtype_reset

      Remove hidden parameters before database upgrade unless your application
      vendors and/or Oracle Support state differently.  Changes will need to be
      made in the pfile/spfile.

  2.  (AUTOFIXUP) Gather stale data dictionary statistics prior to database
      upgrade in off-peak time using:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Dictionary statistics do not exist or are stale (not up-to-date).

      Dictionary statistics help the Oracle optimizer find efficient SQL
      execution plans and are essential for proper upgrade timing. Oracle
      recommends gathering dictionary statistics in the last 24 hours before
      database upgrade.

      For information on managing optimizer statistics, refer to the 12.2.0.1
      Oracle Database SQL Tuning Guide.

  INFORMATION ONLY
  ================
  3.  To help you keep track of your tablespace allocations, the following
      AUTOEXTEND tablespaces are expected to successfully EXTEND during the
      upgrade process.

                                                 Min Size
      Tablespace                        Size     For Upgrade
      ----------                     ----------  -----------
      SYSTEM                             700 MB       866 MB
      TEMP                                54 MB       150 MB
      UNDOTBS1                           380 MB       412 MB

      Minimum tablespace sizes for upgrade are estimates.

  4.  No action needed.

      Using default parallel upgrade options, this CDB with 2 PDBs will first
      upgrade the CDB$ROOT, and then upgrade at most 2 PDBs at a time using 2
      parallel processes per PDB.

      The number of PDBs upgraded in parallel and the number of parallel
      processes per PDB can be adjusted as described in Database Upgrade Guide.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database PRODCDB container CDB$ROOT
  which are identified above as BEFORE UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following from within the container

    SQL>@/opt/app/oracle/cfgtoollogs/stby2cdb/preupgrade/preupgrade_fixups.sql

=============
AFTER UPGRADE
=============

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
  5.  Upgrade the database time zone file using the DBMS_DST package.

      The database is using time zone file version 26 and the target 18.0.0.0.0
      release ships with time zone file version 31.

      Oracle recommends upgrading to the desired (latest) version of the time
      zone file.  For more information, refer to "Upgrading the Time Zone File
      and Timestamp with Time Zone Data" in the 18.0.0.0.0 Oracle Database
      Globalization Support Guide.

  6.  (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
      command:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Oracle recommends gathering dictionary statistics after upgrade.

      Dictionary statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans. After a database
      upgrade, statistics need to be re-gathered as there can now be tables
      that have significantly changed during the upgrade or new tables that do
      not have statistics gathered yet.

  7.  Gather statistics on fixed objects after the upgrade and when there is a
      representative workload on the system using the command:

        EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

      This recommendation is given for all preupgrade runs.

      Fixed object statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans.  Those
      statistics are specific to the Oracle Database release that generates
      them, and can be stale upon database upgrade.

      For information on managing optimizer statistics, refer to the 12.2.0.1
      Oracle Database SQL Tuning Guide.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database PRODCDB container CDB$ROOT
  which are identified above as AFTER UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following from within the container

    SQL>@/opt/app/oracle/cfgtoollogs/stby2cdb/preupgrade/postupgrade_fixups.sql


Report generated by Oracle Database Pre-Upgrade Information Tool Version
18.0.0.0.0 on 2018-08-08T13:23:03

Upgrade-To version: 18.0.0.0.0

=======================================
Status of the database prior to upgrade
=======================================
      Database Name:  PRODCDB
     Container Name:  PDB$SEED
       Container ID:  2
            Version:  12.2.0.1.0
         Compatible:  12.2.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  26
  Database log mode:  ARCHIVELOG
           Readonly:  TRUE
            Edition:  EE

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  Real Application Clusters              [to be upgraded]  OPTION OFF
  Oracle Workspace Manager               [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID

==============
BEFORE UPGRADE
==============

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
  1.  Run 12.2.0.1.0 $ORACLE_HOME/rdbms/admin/utlrp.sql to recompile invalid
      objects.  You can view the individual invalid objects with

        SET SERVEROUTPUT ON;
        EXECUTE DBMS_PREUP.INVALID_OBJECTS;

      4 objects are INVALID.

      There should be no INVALID objects in SYS/SYSTEM or user schemas before
      database upgrade.

  2.  Review and remove any unnecessary HIDDEN/UNDERSCORE parameters.

      The database contains the following initialization parameters whose name
      begins with an underscore:

      _rac_dbtype_reset

      Remove hidden parameters before database upgrade unless your application
      vendors and/or Oracle Support state differently.  Changes will need to be
      made in the pfile/spfile.

  3.  (AUTOFIXUP) Gather stale data dictionary statistics prior to database
      upgrade in off-peak time using:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Dictionary statistics do not exist or are stale (not up-to-date).

      Dictionary statistics help the Oracle optimizer find efficient SQL
      execution plans and are essential for proper upgrade timing. Oracle
      recommends gathering dictionary statistics in the last 24 hours before
      database upgrade.

      For information on managing optimizer statistics, refer to the 12.2.0.1
      Oracle Database SQL Tuning Guide.

  4.  (AUTOFIXUP) Gather statistics on fixed objects prior the upgrade.

      None of the fixed object tables have had stats collected.

      Gathering statistics on fixed objects, if none have been gathered yet, is
      recommended prior to upgrading.

      For information on managing optimizer statistics, refer to the 12.2.0.1
      Oracle Database SQL Tuning Guide.

  INFORMATION ONLY
  ================
  5.  To help you keep track of your tablespace allocations, the following
      AUTOEXTEND tablespaces are expected to successfully EXTEND during the
      upgrade process.

                                                 Min Size
      Tablespace                        Size     For Upgrade
      ----------                     ----------  -----------
      SYSAUX                             285 MB       500 MB
      SYSTEM                             210 MB       589 MB
      TEMP                                20 MB       150 MB
      UNDOTBS1                           210 MB       412 MB

      Minimum tablespace sizes for upgrade are estimates.

  6.  No action needed.

      Using default parallel upgrade options, this CDB with 1 PDBs will first
      upgrade the CDB$ROOT, and then upgrade at most 1 PDBs at a time using 2
      parallel processes per PDB.

      The number of PDBs upgraded in parallel and the number of parallel
      processes per PDB can be adjusted as described in Database Upgrade Guide.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database PRODCDB container PDB$SEED
  which are identified above as BEFORE UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following from within the container

    SQL>@/opt/app/oracle/cfgtoollogs/stby2cdb/preupgrade/preupgrade_fixups.sql

=============
AFTER UPGRADE
=============

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
  7.  Upgrade the database time zone file using the DBMS_DST package.

      The database is using time zone file version 26 and the target 18.0.0.0.0
      release ships with time zone file version 31.

      Oracle recommends upgrading to the desired (latest) version of the time
      zone file.  For more information, refer to "Upgrading the Time Zone File
      and Timestamp with Time Zone Data" in the 18.0.0.0.0 Oracle Database
      Globalization Support Guide.

  8.  (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
      command:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Oracle recommends gathering dictionary statistics after upgrade.

      Dictionary statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans. After a database
      upgrade, statistics need to be re-gathered as there can now be tables
      that have significantly changed during the upgrade or new tables that do
      not have statistics gathered yet.

  9.  Gather statistics on fixed objects after the upgrade and when there is a
      representative workload on the system using the command:

        EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

      This recommendation is given for all preupgrade runs.

      Fixed object statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans.  Those
      statistics are specific to the Oracle Database release that generates
      them, and can be stale upon database upgrade.

      For information on managing optimizer statistics, refer to the 12.2.0.1
      Oracle Database SQL Tuning Guide.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database PRODCDB container PDB$SEED
  which are identified above as AFTER UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following from within the container

    SQL>@/opt/app/oracle/cfgtoollogs/stby2cdb/preupgrade/postupgrade_fixups.sql


Report generated by Oracle Database Pre-Upgrade Information Tool Version
18.0.0.0.0 on 2018-08-08T13:23:01

Upgrade-To version: 18.0.0.0.0

=======================================
Status of the database prior to upgrade
=======================================
      Database Name:  PRODCDB
     Container Name:  PDBAPP1
       Container ID:  3
            Version:  12.2.0.1.0
         Compatible:  12.2.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  26
  Database log mode:  ARCHIVELOG
           Readonly:  FALSE
            Edition:  EE

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  Real Application Clusters              [to be upgraded]  OPTION OFF
  Oracle Workspace Manager               [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID

==============
BEFORE UPGRADE
==============

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
  1.  Run 12.2.0.1.0 $ORACLE_HOME/rdbms/admin/utlrp.sql to recompile invalid
      objects.  You can view the individual invalid objects with

        SET SERVEROUTPUT ON;
        EXECUTE DBMS_PREUP.INVALID_OBJECTS;

      4 objects are INVALID.

      There should be no INVALID objects in SYS/SYSTEM or user schemas before
      database upgrade.

  2.  Review and remove any unnecessary HIDDEN/UNDERSCORE parameters.

      The database contains the following initialization parameters whose name
      begins with an underscore:

      _rac_dbtype_reset

      Remove hidden parameters before database upgrade unless your application
      vendors and/or Oracle Support state differently.  Changes will need to be
      made in the pfile/spfile.

  3.  (AUTOFIXUP) Gather stale data dictionary statistics prior to database
      upgrade in off-peak time using:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Dictionary statistics do not exist or are stale (not up-to-date).

      Dictionary statistics help the Oracle optimizer find efficient SQL
      execution plans and are essential for proper upgrade timing. Oracle
      recommends gathering dictionary statistics in the last 24 hours before
      database upgrade.

      For information on managing optimizer statistics, refer to the 12.2.0.1
      Oracle Database SQL Tuning Guide.

  INFORMATION ONLY
  ================
  4.  To help you keep track of your tablespace allocations, the following
      AUTOEXTEND tablespaces are expected to successfully EXTEND during the
      upgrade process.

                                                 Min Size
      Tablespace                        Size     For Upgrade
      ----------                     ----------  -----------
      SYSAUX                             395 MB       582 MB
      SYSTEM                             210 MB       600 MB
      TEMP                                20 MB       150 MB
      UNDOTBS1                           210 MB       412 MB

      Minimum tablespace sizes for upgrade are estimates.

  5.  No action needed.

      Using default parallel upgrade options, this CDB with 1 PDBs will first
      upgrade the CDB$ROOT, and then upgrade at most 1 PDBs at a time using 2
      parallel processes per PDB.

      The number of PDBs upgraded in parallel and the number of parallel
      processes per PDB can be adjusted as described in Database Upgrade Guide.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database PRODCDB container PDBAPP1
  which are identified above as BEFORE UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following from within the container

    SQL>@/opt/app/oracle/cfgtoollogs/stby2cdb/preupgrade/preupgrade_fixups.sql

=============
AFTER UPGRADE
=============

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
  6.  Upgrade the database time zone file using the DBMS_DST package.

      The database is using time zone file version 26 and the target 18.0.0.0.0
      release ships with time zone file version 31.

      Oracle recommends upgrading to the desired (latest) version of the time
      zone file.  For more information, refer to "Upgrading the Time Zone File
      and Timestamp with Time Zone Data" in the 18.0.0.0.0 Oracle Database
      Globalization Support Guide.

  7.  (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
      command:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Oracle recommends gathering dictionary statistics after upgrade.

      Dictionary statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans. After a database
      upgrade, statistics need to be re-gathered as there can now be tables
      that have significantly changed during the upgrade or new tables that do
      not have statistics gathered yet.

  8.  Gather statistics on fixed objects after the upgrade and when there is a
      representative workload on the system using the command:

        EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

      This recommendation is given for all preupgrade runs.

      Fixed object statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans.  Those
      statistics are specific to the Oracle Database release that generates
      them, and can be stale upon database upgrade.

      For information on managing optimizer statistics, refer to the 12.2.0.1
      Oracle Database SQL Tuning Guide.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database PRODCDB container PDBAPP1
  which are identified above as AFTER UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following from within the container

    SQL>@/opt/app/oracle/cfgtoollogs/stby2cdb/preupgrade/postupgrade_fixups.sql


==================
PREUPGRADE SUMMARY
==================
  /opt/app/oracle/cfgtoollogs/stby2cdb/preupgrade/preupgrade.log
  /opt/app/oracle/cfgtoollogs/stby2cdb/preupgrade/preupgrade_fixups.sql
  /opt/app/oracle/cfgtoollogs/stby2cdb/preupgrade/postupgrade_fixups.sql

Execute fixup scripts across the entire CDB:

Before upgrade:

1. Execute preupgrade fixups with the below command
$ORACLE_HOME/perl/bin/perl -I$ORACLE_HOME/perl/lib -I$ORACLE_HOME/rdbms/admin $ORACLE_HOME/rdbms/admin/catcon.pl -l /opt/app/oracle/cfgtoollogs/stby2cdb/preupgrade/ -b preup_stby2cdb /opt/app/oracle/cfgtoollogs/stby2cdb/preupgrade/preupgrade_fixups.sql

2. Review logs under /opt/app/oracle/cfgtoollogs/stby2cdb/preupgrade/

After the upgrade:

1. Execute postupgrade fixups with the below command
$ORACLE_HOME/perl/bin/perl -I$ORACLE_HOME/perl/lib -I$ORACLE_HOME/rdbms/admin $ORACLE_HOME/rdbms/admin/catcon.pl -l /opt/app/oracle/cfgtoollogs/stby2cdb/preupgrade/ -b postup_stby2cdb /opt/app/oracle/cfgtoollogs/stby2cdb/preupgrade/postupgrade_fixups.sql

2. Review logs under /opt/app/oracle/cfgtoollogs/stby2cdb/preupgrade/
Run the pre-upgrade fixup script and then DBUA from the 18c home. Select the database to upgrade.
The PDB is upgraded at the same the CDB. If there were multiple PDBs this step allows to set the order of priority for upgrading the PDBs.
Check and resolve any validation concerns.
Parallel upgrade will speed up the upgrade process. This step also allows to specify whether timezone is also upgraded at the same time as the database.
Specify recovery option in case upgrade fails and has to revert back to 12.2 DB.
Select the listener to be used.
The upgrade summary page.
Upgrade progress.
Upgrade result page.
Upgrade result page showing total upgrade time.
Once the upgrade has finished execute the post-upgrade fixup script (mentioned in the output when preupgrade.jar was run).
The DB components versions and status after the upgrade for each container is as follows. The full version is 18.3 due the fact that RU 18.3.0.0.180717 is bundled with the 18c installer image file (check .patch_storge after unzipping) and applied to the database during the upgrade.
SQL> select con_id,comp_name,status,version,version_full from cdb_registry order by 1,2;

    CON_ID COMP_NAME                           STATUS       VERSION         VERSION_FULL
---------- ----------------------------------- ------------ --------------- ---------------
         1 Oracle Database Catalog Views       VALID        18.0.0.0.0      18.3.0.0.0
         1 Oracle Database Packages and Types  VALID        18.0.0.0.0      18.3.0.0.0
         1 Oracle Real Application Clusters    OPTION OFF   18.0.0.0.0      18.3.0.0.0
         1 Oracle Text                         VALID        18.0.0.0.0      18.3.0.0.0
         1 Oracle Workspace Manager            VALID        18.0.0.0.0      18.3.0.0.0
         1 Oracle XML Database                 VALID        18.0.0.0.0      18.3.0.0.0
         3 Oracle Database Catalog Views       VALID        18.0.0.0.0      18.3.0.0.0
         3 Oracle Database Packages and Types  VALID        18.0.0.0.0      18.3.0.0.0
         3 Oracle Real Application Clusters    OPTION OFF   18.0.0.0.0      18.3.0.0.0
         3 Oracle Text                         VALID        18.0.0.0.0      18.3.0.0.0
         3 Oracle Workspace Manager            VALID        18.0.0.0.0      18.3.0.0.0
         3 Oracle XML Database                 VALID        18.0.0.0.0      18.3.0.0.0
The timezone file has been upgraded to 31.
SQL> select * from v$timezone_file;

FILENAME                VERSION     CON_ID
-------------------- ---------- ----------
timezlrg_31.dat              31          0
If satisfied with the upgrade and application testing then change the compatibility parameter on the DB and ASM disk groups.
SQL> alter system set compatible='18.0.0.0.0' scope=spfile;
shutdown immediate;
Login as grid user and then login to ASM instance as sysasm to change asm compatibility parameters.
SQL> alter diskgroup FRA SET attribute 'compatible.asm'='18.0.0.0.0';
SQL> alter diskgroup DATA  SET attribute 'compatible.asm'='18.0.0.0.0';
SQL> alter diskgroup fra set attribute 'compatible.rdbms'='18.0.0.0.0';
SQL> alter diskgroup data set attribute 'compatible.rdbms'='18.0.0.0.0';

SQL> select g.name,a.name,a.value from v$asm_diskgroup g, v$asm_attribute a where g.group_number=a.group_number and a.name like '%compat%';

NAME                 NAME                 VALUE
-------------------- -------------------- --------------------
DATA                 compatible.asm       18.0.0.0.0
DATA                 compatible.rdbms     18.0.0.0.0
FRA                  compatible.asm       18.0.0.0.0
FRA                  compatible.rdbms     18.0.0.0.0
After ASM compatibility parameters are updated start the database.
18c also has the patch listing error mentioned in an earlier post. The fix mentioned in that post works for 18c as well.
Finally run orachk -u -o post check the post upgrade state of the oracle restart setup.

Related Posts
Upgrading Oracle Single Instance with ASM (Oracle Restart) from 11.2.0.4 to 12.2.0.1
Upgrading Oracle Single Instance with ASM (Oracle Restart) from 12.1.0.2 to 12.2.0.1
Upgrading Single Instance on ASM from 11.2.0.3 to 11.2.0.4
Upgrading Grid Infrastructure Used for Single Instance from 11.2.0.4 to 12.1.0.2

Wednesday, August 1, 2018

Adding a New Physical Standby to Existing Data Guard Setup

This post shows the steps for adding a physical standby to an existing data guard setup. The existing data guard setup consists of a single primary and single physical standby with Oracle restart. To this DG setup a new physical standby would be added followed by DG broker update with the new standby. The summary of hostnames, DB names is shown in the table below.
ItemExisting Data Guard SetupNew Standby
PrimaryStandby
Host namecity7city7scity7s2
Database Nameprodcdbstbycdbstby2cdb
Diskgroup NamesDATA
FRA
DATA
FRA
DATA
FRA
TNS Entry NamePRODCDBTNSSTBYTNSSTBY2TNS

The steps here are of the configuration of the data guard portion only. It's assumed relevant software is installed and ASM disk configuration is done on the new standby server.

1. On the new standby create the adump directory.
cd $ORACLE_BASE/admin
mkdir stby2cdb
cd stby2cdb
mkdir adump  dpdump  hdump  pfile
2. Add the static listner entries to the listener.ora file. In this case entries for both standard listener entry and DG broker specific entry (with DGMGRL) is added the same time.
SID_LIST_LISTENER =
(SID_LIST =
        (SID_DESC =
                (GLOBAL_DBNAME = stby2cdb)
                (SID_NAME = stby2cdb)
                (ORACLE_HOME = /opt/app/oracle/product/12.2.0/dbhome_1)
        )
        (SID_DESC =
                (GLOBAL_DBNAME = stby2cdb_DGMGRL)
                (SID_NAME = stby2cdb)
                (ORACLE_HOME = /opt/app/oracle/product/12.2.0/dbhome_1)
        )

)
Start the listener on the new standby and verify static services are up
Service "stby2cdb" has 1 instance(s).
  Instance "stby2cdb", status UNKNOWN, has 1 handler(s) for this service...
Service "stby2cdb_DGMGRL" has 1 instance(s).
  Instance "stby2cdb", status UNKNOWN, has 1 handler(s) for this service...
3. Create TNS entry on all servers (both on existing and new standby) pointing to the standby instance that would be running on the new standby server.
STBY2TNS =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = city7s2)(PORT = 1581))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = stby2cdb)
    )
  )
4. Copy the Oracle password file to new standby server and rename the file to reflect the new standby instance name.
scp orapwprodcdb city7s2:$ORACLE_HOME/dbs/orapwstby2cdb
5. Create init file on new standby with just the db_name entry and start the standby instance in nomount mode
more initstby2cdb.ora
db_name=stby2cdb
6. On the primary modify the log_archive_config parameter by including the new standby DB. Also add a new log_archive_dest value pointing to the new standby instance. Update the fal_server list to include the TNS entry for the new standby. Update both DB and Log file name convert parameter considering how the file name conversion must happen if and when each of the standby databases become primary.
alter system set log_archive_config='dg_config=(prodcdb,stbycdb,stby2cdb)' scope=both ;
alter system set log_archive_dest_3='service=STBY2TNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=stby2cdb' scope=both;
alter system set fal_server='STBYTNS','STBY2TNS' scope=both;
alter system set db_file_name_convert='/stbycdb/','/prodcdb/','/stby2cdb/','/prodcdb/' scope=spfile;
alter system set log_file_name_convert='/stbycdb/','/prodcdb/','/stby2cdb/','/prodcdb/' scope=spfile;
7. Same set of parameters should be updated on the existing standby as well.
alter system set log_archive_config='dg_config=(prodcdb,stbycdb,stby2cdb)' scope=both ;
alter system set log_archive_dest_3='service=STBY2TNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=stby2cdb' scope=both;
alter system set fal_server='PRODCDBTNS','STBY2TNS' scope=both;
alter system set db_file_name_convert='/prodcdb/','/stbycdb/','/stby2cdb/','/stbycdb/' scope=spfile;
alter system set log_file_name_convert='/prodcdb/','/stbycdb/','/stby2cdb/','/stbycdb/' scope=spfile;
8. Finally run the rman command to create the new standby. Earlier post showed multiple ways of creating the standby DB. In this instance the standby is created using active database option. Following rman command is run from the current primary.
rman target sys/prodcdbdb auxiliary sys/prodcdbdb@stby2tns
duplicate target database for standby from active database
spfile
parameter_value_convert 'prodcdb','stby2cdb','PRODCDB','STBY2CDB','stbycdb','stby2cdb','STBYCDB','STBY2CDB'
set db_name='prodcdb'
set db_unique_name='stby2cdb'
set db_file_name_convert='/prodcdb/','/stby2cdb/','/stbycdb/','/stby2cdb/'
set log_file_name_convert='/prodcdb/','/stby2cdb/','/stbycdb/','/stby2cdb/'
set log_archive_max_processes='10'
set fal_server='PRODCDBTNS','STBYTNS'
reset local_listener
set log_archive_dest_2='service=PRODCDBTNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=prodcdb'
set log_archive_dest_3='service=STBYTNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=stbycdb'
set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=stby2cdb';
Full output of the duplication is shown below
rman target sys/prodcdbdb auxiliary sys/prodcdbdb@stby2tns

Recovery Manager: Release 12.2.0.1.0 - Production on Mon Jul 16 14:41:23 2018

Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.

connected to target database: PRODCDB (DBID=2963914998)
connected to auxiliary database: STBY2CDB (not mounted)

RMAN> duplicate target database for standby from active database
2> spfile
3> parameter_value_convert 'prodcdb','stby2cdb','PRODCDB','STBY2CDB','stbycdb','stby2cdb','STBYCDB','STBY2CDB'
4> set db_name='prodcdb'
5> set db_unique_name='stby2cdb'
set db_file_name_convert='/prodcdb/','/stby2cdb/','/stbycdb/','/stby2cdb/'
6> 7> set log_file_name_convert='/prodcdb/','/stby2cdb/','/stbycdb/','/stby2cdb/'
8> set log_archive_max_processes='10'
9> set fal_server='PRODCDBTNS','STBYTNS'
10> reset local_listener
set log_archive_dest_2='service=PRODCDBTNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=prodcdb'
11> 12> set log_archive_dest_3='service=STBYTNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=stbycdb'
13> set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=stby2cdb';

Starting Duplicate Db at 16-JUL-18
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=10 device type=DISK

contents of Memory Script:
{
   backup as copy reuse
   passwordfile auxiliary format  '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/orapwstby2cdb'   targetfile
 '+DATA/prodcdb/spfileprodcdb.ora' auxiliary format
 '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/spfilestby2cdb.ora'   ;
   sql clone "alter system set spfile= ''/opt/app/oracle/product/12.2.0/dbhome_1/dbs/spfilestby2cdb.ora''";
}
executing Memory Script

Starting backup at 16-JUL-18
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=26 device type=DISK
Finished backup at 16-JUL-18

sql statement: alter system set spfile= ''/opt/app/oracle/product/12.2.0/dbhome_1/dbs/spfilestby2cdb.ora''

contents of Memory Script:
{
   sql clone "alter system set  audit_file_dest =
 ''/opt/app/oracle/admin/stby2cdb/adump'' comment=
 '''' scope=spfile";
   sql clone "alter system set  control_files =
 ''+DATA/STBY2CDB/CONTROLFILE/current.266.965841019'', ''+FRA/STBY2CDB/CONTROLFILE/current.256.965841019'' comment=
 '''' scope=spfile";
   sql clone "alter system set  dispatchers =
 ''(PROTOCOL=TCP) (SERVICE=stby2cdbXDB)'' comment=
 '''' scope=spfile";
   sql clone "alter system set  db_name =
 ''prodcdb'' comment=
 '''' scope=spfile";
   sql clone "alter system set  db_unique_name =
 ''stby2cdb'' comment=
 '''' scope=spfile";
   sql clone "alter system set  db_file_name_convert =
 ''/prodcdb/'', ''/stby2cdb/'', ''/stbycdb/'', ''/stby2cdb/'' comment=
 '''' scope=spfile";
   sql clone "alter system set  log_file_name_convert =
 ''/prodcdb/'', ''/stby2cdb/'', ''/stbycdb/'', ''/stby2cdb/'' comment=
 '''' scope=spfile";
   sql clone "alter system set  log_archive_max_processes =
 10 comment=
 '''' scope=spfile";
   sql clone "alter system set  fal_server =
 ''PRODCDBTNS'', ''STBYTNS'' comment=
 '''' scope=spfile";
   sql clone "alter system reset  local_listener scope=spfile";
   sql clone "alter system set  log_archive_dest_2 =
 ''service=PRODCDBTNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=prodcdb'' comment=
 '''' scope=spfile";
   sql clone "alter system set  log_archive_dest_3 =
 ''service=STBYTNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=stbycdb'' comment=
 '''' scope=spfile";
   sql clone "alter system set  log_archive_dest_1 =
 ''location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=stby2cdb'' comment=
 '''' scope=spfile";
   shutdown clone immediate;
   startup clone nomount;
}
executing Memory Script

sql statement: alter system set  audit_file_dest =  ''/opt/app/oracle/admin/stby2cdb/adump'' comment= '''' scope=spfile

sql statement: alter system set  control_files =  ''+DATA/STBY2CDB/CONTROLFILE/current.266.965841019'', ''+FRA/STBY2CDB/CONTROLFILE/current.256.965841019'' comment= '''' scope=spfile

sql statement: alter system set  dispatchers =  ''(PROTOCOL=TCP) (SERVICE=stby2cdbXDB)'' comment= '''' scope=spfile

sql statement: alter system set  db_name =  ''prodcdb'' comment= '''' scope=spfile

sql statement: alter system set  db_unique_name =  ''stby2cdb'' comment= '''' scope=spfile

sql statement: alter system set  db_file_name_convert =  ''/prodcdb/'', ''/stby2cdb/'', ''/stbycdb/'', ''/stby2cdb/'' comment= '''' scope=spfile

sql statement: alter system set  log_file_name_convert =  ''/prodcdb/'', ''/stby2cdb/'', ''/stbycdb/'', ''/stby2cdb/'' comment= '''' scope=spfile

sql statement: alter system set  log_archive_max_processes =  10 comment= '''' scope=spfile

sql statement: alter system set  fal_server =  ''PRODCDBTNS'', ''STBYTNS'' comment= '''' scope=spfile

sql statement: alter system reset  local_listener scope=spfile

sql statement: alter system set  log_archive_dest_2 =  ''service=PRODCDBTNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=prodcdb'' comment= '''' scope=spfile

sql statement: alter system set  log_archive_dest_3 =  ''service=STBYTNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=stbycdb'' comment= '''' scope=spfile

sql statement: alter system set  log_archive_dest_1 =  ''location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=stby2cdb'' comment= '''' scope=spfile

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area    1191182336 bytes

Fixed Size                     8792104 bytes
Variable Size                452986840 bytes
Database Buffers             721420288 bytes
Redo Buffers                   7983104 bytes

contents of Memory Script:
{
   sql clone "alter system set  control_files =
  ''+DATA/STBY2CDB/CONTROLFILE/current.264.981652083'', ''+FRA/STBY2CDB/CONTROLFILE/current.318.981652083'' comment=
 ''Set by RMAN'' scope=spfile";
   backup as copy current controlfile for standby auxiliary format  '+DATA/STBY2CDB/CONTROLFILE/current.290.981652083';
   restore clone primary controlfile to  '+FRA/STBY2CDB/CONTROLFILE/current.276.981652083' from
 '+DATA/STBY2CDB/CONTROLFILE/current.290.981652083';
   sql clone "alter system set  control_files =
  ''+DATA/STBY2CDB/CONTROLFILE/current.290.981652083'', ''+FRA/STBY2CDB/CONTROLFILE/current.276.981652083'' comment=
 ''Set by RMAN'' scope=spfile";
   shutdown clone immediate;
   startup clone nomount;
}
executing Memory Script

sql statement: alter system set  control_files =   ''+DATA/STBY2CDB/CONTROLFILE/current.264.981652083'', ''+FRA/STBY2CDB/CONTROLFILE/current.318.981652083'' comment= ''Set by RMAN'' scope=spfile

Starting backup at 16-JUL-18
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
copying standby control file
output file name=/opt/app/oracle/product/12.2.0/dbhome_1/dbs/snapcf_prodcdb.f tag=TAG20180716T144207
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
Finished backup at 16-JUL-18

Starting restore at 16-JUL-18
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=137 device type=DISK

channel ORA_AUX_DISK_1: copied control file copy
Finished restore at 16-JUL-18

sql statement: alter system set  control_files =   ''+DATA/STBY2CDB/CONTROLFILE/current.290.981652083'', ''+FRA/STBY2CDB/CONTROLFILE/current.276.981652083'' comment= ''Set by RMAN'' scope=spfile

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area    1191182336 bytes

Fixed Size                     8792104 bytes
Variable Size                452986840 bytes
Database Buffers             721420288 bytes
Redo Buffers                   7983104 bytes

contents of Memory Script:
{
   sql clone 'alter database mount standby database';
}
executing Memory Script

sql statement: alter database mount standby database
RMAN-05529: warning: DB_FILE_NAME_CONVERT resulted in invalid ASM names; names changed to disk group only.

contents of Memory Script:
{
   set newname for tempfile  1 to
 "+DATA";
   set newname for tempfile  2 to
 "+DATA";
   set newname for tempfile  3 to
 "+DATA";
   switch clone tempfile all;
   set newname for datafile  1 to
 "+DATA";
   set newname for datafile  2 to
 "+DATA";
   set newname for datafile  3 to
 "+DATA";
   set newname for datafile  4 to
 "+DATA";
   set newname for datafile  5 to
 "+DATA";
   set newname for datafile  6 to
 "+DATA";
   set newname for datafile  7 to
 "+DATA";
   set newname for datafile  33 to
 "+DATA";
   set newname for datafile  34 to
 "+DATA";
   set newname for datafile  35 to
 "+DATA";
   set newname for datafile  121 to
 "+DATA";
   backup as copy reuse
   datafile  1 auxiliary format
 "+DATA"   datafile
 2 auxiliary format
 "+DATA"   datafile
 3 auxiliary format
 "+DATA"   datafile
 4 auxiliary format
 "+DATA"   datafile
 5 auxiliary format
 "+DATA"   datafile
 6 auxiliary format
 "+DATA"   datafile
 7 auxiliary format
 "+DATA"   datafile
 33 auxiliary format
 "+DATA"   datafile
 34 auxiliary format
 "+DATA"   datafile
 35 auxiliary format
 "+DATA"   datafile
 121 auxiliary format
 "+DATA"   ;
   sql 'alter system archive log current';
}
executing Memory Script

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

renamed tempfile 1 to +DATA in control file
renamed tempfile 2 to +DATA in control file
renamed tempfile 3 to +DATA in control file

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting backup at 16-JUL-18
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00003 name=+DATA/PRODCDB/DATAFILE/sysaux.265.965841035
output file name=+DATA/STBY2CDB/DATAFILE/sysaux.296.981652125 tag=TAG20180716T144248
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:35
channel ORA_DISK_1: starting datafile copy
input datafile file number=00001 name=+DATA/PRODCDB/DATAFILE/system.259.965841027
output file name=+DATA/STBY2CDB/DATAFILE/system.298.981652161 tag=TAG20180716T144248
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile file number=00005 name=+DATA/PRODCDB/DATAFILE/undotbs1.262.965841045
output file name=+DATA/STBY2CDB/DATAFILE/undotbs1.289.981652175 tag=TAG20180716T144248
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile copy
input datafile file number=00004 name=+DATA/PRODCDB/6325282AC695380EE0535500A8C0D89D/DATAFILE/sysaux.263.965841045
output file name=+DATA/STBY2CDB/6325282AC695380EE0535500A8C0D89D/DATAFILE/sysaux.285.981652183 tag=TAG20180716T144248
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile file number=00034 name=+DATA/PRODCDB/6EFD65D00A9340E9E0535500A8C09D38/DATAFILE/sysaux.277.979213603
output file name=+DATA/STBY2CDB/6EFD65D00A9340E9E0535500A8C09D38/DATAFILE/sysaux.306.981652199 tag=TAG20180716T144248
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile file number=00002 name=+DATA/PRODCDB/6325282AC695380EE0535500A8C0D89D/DATAFILE/system.258.965841031
output file name=+DATA/STBY2CDB/6325282AC695380EE0535500A8C0D89D/DATAFILE/system.275.981652213 tag=TAG20180716T144248
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile copy
input datafile file number=00006 name=+DATA/PRODCDB/6325282AC695380EE0535500A8C0D89D/DATAFILE/undotbs1.261.965841047
output file name=+DATA/STBY2CDB/6325282AC695380EE0535500A8C0D89D/DATAFILE/undotbs1.282.981652221 tag=TAG20180716T144248
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile copy
input datafile file number=00033 name=+DATA/PRODCDB/6EFD65D00A9340E9E0535500A8C09D38/DATAFILE/system.276.979213603
output file name=+DATA/STBY2CDB/6EFD65D00A9340E9E0535500A8C09D38/DATAFILE/system.268.981652227 tag=TAG20180716T144248
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile copy
input datafile file number=00035 name=+DATA/PRODCDB/6EFD65D00A9340E9E0535500A8C09D38/DATAFILE/undotbs1.275.979213603
output file name=+DATA/STBY2CDB/6EFD65D00A9340E9E0535500A8C09D38/DATAFILE/undotbs1.262.981652235 tag=TAG20180716T144248
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile copy
input datafile file number=00121 name=+DATA/PRODCDB/DATAFILE/test.303.969113983
output file name=+DATA/STBY2CDB/DATAFILE/test.283.981652243 tag=TAG20180716T144248
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting datafile copy
input datafile file number=00007 name=+DATA/PRODCDB/DATAFILE/users.269.965841065
output file name=+DATA/STBY2CDB/DATAFILE/users.311.981652243 tag=TAG20180716T144248
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
Finished backup at 16-JUL-18

sql statement: alter system archive log current

contents of Memory Script:
{
   switch clone datafile all;
}
executing Memory Script

datafile 1 switched to datafile copy
input datafile copy RECID=1 STAMP=981652244 file name=+DATA/STBY2CDB/DATAFILE/system.298.981652161
datafile 2 switched to datafile copy
input datafile copy RECID=2 STAMP=981652244 file name=+DATA/STBY2CDB/6325282AC695380EE0535500A8C0D89D/DATAFILE/system.275.981652213
datafile 3 switched to datafile copy
input datafile copy RECID=3 STAMP=981652245 file name=+DATA/STBY2CDB/DATAFILE/sysaux.296.981652125
datafile 4 switched to datafile copy
input datafile copy RECID=4 STAMP=981652245 file name=+DATA/STBY2CDB/6325282AC695380EE0535500A8C0D89D/DATAFILE/sysaux.285.981652183
datafile 5 switched to datafile copy
input datafile copy RECID=5 STAMP=981652245 file name=+DATA/STBY2CDB/DATAFILE/undotbs1.289.981652175
datafile 6 switched to datafile copy
input datafile copy RECID=6 STAMP=981652245 file name=+DATA/STBY2CDB/6325282AC695380EE0535500A8C0D89D/DATAFILE/undotbs1.282.981652221
datafile 7 switched to datafile copy
input datafile copy RECID=7 STAMP=981652245 file name=+DATA/STBY2CDB/DATAFILE/users.311.981652243
datafile 33 switched to datafile copy
input datafile copy RECID=8 STAMP=981652245 file name=+DATA/STBY2CDB/6EFD65D00A9340E9E0535500A8C09D38/DATAFILE/system.268.981652227
datafile 34 switched to datafile copy
input datafile copy RECID=9 STAMP=981652245 file name=+DATA/STBY2CDB/6EFD65D00A9340E9E0535500A8C09D38/DATAFILE/sysaux.306.981652199
datafile 35 switched to datafile copy
input datafile copy RECID=10 STAMP=981652245 file name=+DATA/STBY2CDB/6EFD65D00A9340E9E0535500A8C09D38/DATAFILE/undotbs1.262.981652235
datafile 121 switched to datafile copy
input datafile copy RECID=11 STAMP=981652245 file name=+DATA/STBY2CDB/DATAFILE/test.283.981652243
Finished Duplicate Db at 16-JUL-18
9. Start the managed recovery on the new standby and verify applying of the archived logs
alter database recover managed standby database using current logfile disconnect;

SQL> select sequence#,applied from v$archived_log;

 SEQUENCE# APPLIED
---------- ---------
       497 YES
       498 YES
       499 IN-MEMORY


10. Next step is to add the new standby to the Oracle restart. Update the oracle binary permission on the Oracle home of the new standby to avoid the ORA-27303 issue. Once done add the new standby to Oracle restart configuration. Run the following as Oracle user
srvctl add database -db stby2cdb -oraclehome $ORACLE_HOME  -spfile "/opt/app/oracle/product/12.2.0/dbhome_1/dbs/spfilestby2cdb.ora" -role physical_standby -startoption mount -diskgroup "DATA,FRA"

srvctl  config database -db stby2cdb
Database unique name: stby2cdb
Database name:
Oracle home: /opt/app/oracle/product/12.2.0/dbhome_1
Oracle user: oracle
Spfile: /opt/app/oracle/product/12.2.0/dbhome_1/dbs/spfilestby2cdb.ora
Password file:
Domain:
Start options: mount
Stop options: immediate
Database role: PHYSICAL_STANDBY
Management policy: AUTOMATIC
Disk Groups: DATA,FRA
Services:
OSDBA group:
OSOPER group:
Database instance: stby2cdb

srvctl  stop database -db stby2cdb
srvctl  start database -db stby2cdb
11. Final step is to add the new standby to the existing data guard broker configuration. On the new standby cancel the managed recovery and set dg_broker_start to true. It is assumed that dg_broker_config_file* parameter will be set to default value.
alter database recover managed standby database cancel;

alter system set dg_broker_start=true scope=both sid='*';
12. To avoid the ORA-16575, clear the log_archive_dest values added during the earlier steps. On all databases, including the new standby, run the following to clear the new log_archive_dest value
alter system reset log_archive_dest_3 scope=both;
Additionally on the new standby run the following to clear the second log archive dest as well.
alter system reset log_archive_dest_2 scope=both;
13. Once the log archive dest are cleared add the new standby to the existing data guard configuration and enable it.
DGMGRL> add database stby2cdb as connect identifier is stby2tns;
Database "stby2cdb" added

DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  prodcdb  - Primary database
    Warning: ORA-16792: configurable property value is inconsistent with member setting

    stbycdb  - Physical standby database
      Warning: ORA-16792: configurable property value is inconsistent with member setting

    stby2cdb - Physical standby database (disabled)

Fast-Start Failover: DISABLED

Configuration Status:
WARNING   (status updated 39 seconds ago)

DGMGRL> enable database stby2cdb;
Enabled.

DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  prodcdb  - Primary database
    Warning: ORA-16792: configurable property value is inconsistent with member setting

    stbycdb  - Physical standby database
      Warning: ORA-16792: configurable property value is inconsistent with member setting

    stby2cdb - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
WARNING   (status updated 36 seconds ago)
14. The warning on the existing databases is down to mismatch of db/log file name convert values. The Oracle doc states the following (only db_file_name convert text is quoted here) "when a database is added to the configuration, the broker sets the initial value of this property to the in-memory value of the DB_FILE_NAME_CONVERT initialization parameter. It is possible that the in-memory value and server parameter file (SPFILE) value of this parameter will differ. If you want to use the parameter's in-memory value, then enable the database and the broker will ensure that the SPFILE value of the parameter is set to the in-memory value. If you want to use the SPFILE value, then set the property value to be the parameter's value stored in the SPFILE. Then enable the database". Listing inconsistent properties for each database list these values
DGMGRL> show database prodcdb inconsistentProperties
INCONSISTENT PROPERTIES
   INSTANCE_NAME        PROPERTY_NAME         MEMORY_VALUE         SPFILE_VALUE         BROKER_VALUE
         prodcdb    DbFileNameConvert /stbycdb/, /prodcdb/ /stbycdb/,/prodcdb/,/stby2cdb/,/prodcdb/ /stbycdb/, /prodcdb/
         prodcdb   LogFileNameConvert /stbycdb/, /prodcdb/ /stbycdb/,/prodcdb/,/stby2cdb/,/prodcdb/ /stbycdb/, /prodcdb/

DGMGRL> show database stbycdb inconsistentProperties
INCONSISTENT PROPERTIES
   INSTANCE_NAME        PROPERTY_NAME         MEMORY_VALUE         SPFILE_VALUE         BROKER_VALUE
         stbycdb    DbFileNameConvert /prodcdb/, /stbycdb/ /prodcdb/,/stbycdb/,/stby2cdb/,/stbycdb/ /prodcdb/, /stbycdb/
         stbycdb   LogFileNameConvert /prodcdb/, /stbycdb/ /prodcdb/,/stbycdb/,/stby2cdb/,/stbycdb/ /prodcdb/, /stbycdb/
15. To fix this manually update each property for each database that report inconsistency. Following is executed on current production
DGMGRL>  edit database prodcdb set property 'DbFileNameConvert'='/stbycdb/,/prodcdb/,/stby2cdb/,/prodcdb/';
Warning: ORA-16675: database instance restart required for property value modification to take effect

Property "DbFileNameConvert" updated

show database prodcdb DbFileNameConvert
  DbFileNameConvert = '/stbycdb/,/prodcdb/,/stby2cdb/,/prodcdb/'

edit database prodcdb set property 'LogFileNameConvert'='/stbycdb/,/prodcdb/,/stby2cdb/,/prodcdb/';
Warning: ORA-16675: database instance restart required for property value modification to take effect

Property "LogFileNameConvert" updated

show database prodcdb  LogFileNameConvert
  LogFileNameConvert = '/stbycdb/,/prodcdb/,/stby2cdb/,/prodcdb/'
Following is executed on the existing standby
DGMGRL>  edit database stbycdb set property 'DbFileNameConvert'='/prodcdb/,/stbycdb/,/stby2cdb/,/stbycdb/' ;
Warning: ORA-16675: database instance restart required for property value modification to take effect

Property "DbFileNameConvert" updated

DGMGRL>  edit database stbycdb set property 'LogFileNameConvert'='/prodcdb/,/stbycdb/,/stby2cdb/,/stbycdb/' ;
Warning: ORA-16675: database instance restart required for property value modification to take effect

Property "LogFileNameConvert" updated

DGMGRL> show database stbycdb LogFileNameConvert
  LogFileNameConvert = '/prodcdb/,/stbycdb/,/stby2cdb/,/stbycdb/'
DGMGRL> show database stbycdb DbFileNameConvert
  DbFileNameConvert = '/prodcdb/,/stbycdb/,/stby2cdb/,/stbycdb/'
16. As highlighted above, for this change to take effect restart of the databases, both primary and exiting standby is required incurring down time. Configuration status after restart of the existing standby (stbycdb)
DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  prodcdb  - Primary database
    Warning: ORA-16809: multiple warnings detected for the member

    stbycdb  - Physical standby database (disabled)
    stby2cdb - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
WARNING   (status updated 16 seconds ago)

DGMGRL> enable database stbycdb
Enabled.
DGMGRL>  show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  prodcdb  - Primary database
    Warning: ORA-16792: configurable property value is inconsistent with member setting

    stbycdb  - Physical standby database
    stby2cdb - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
WARNING   (status updated 1 second ago)
17.After the restart of primary
DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  prodcdb  - Primary database
    stbycdb  - Physical standby database
    stby2cdb - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 27 seconds ago)
18. The existing DG configuration has the protection mode set to maximum availability. However the new standby was added with log transport mode of ASYNC. Change this to SYNC to keep the same protection level even if one standby goes down.
DGMGRL> show database stby2cdb LogXptMode
  LogXptMode = 'ASYNC'
DGMGRL> edit database stby2cdb set property LogXptMode='SYNC';
Property "logxptmode" updated

DGMGRL> show database stby2cdb LogXptMode
  LogXptMode = 'SYNC'
DGMGRL>  show database stbycdb LogXptMode
  LogXptMode = 'SYNC'
19. If possible switchover to the newly added standby and verify DG configuration is working as expected.
DGMGRL> switchover to stby2cdb
Performing switchover NOW, please wait...
New primary database "stby2cdb" is opening...
Oracle Clusterware is restarting database "prodcdb" ...
Switchover succeeded, new primary is "stby2cdb"
DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  stby2cdb - Primary database
    prodcdb  - Physical standby database
    stbycdb  - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 58 seconds ago)
20. Rotate the primary among all databases to verify file name conversion and log transport is working as expected.
DGMGRL> switchover to prodcdb
Performing switchover NOW, please wait...
Operation requires a connection to database "prodcdb"
Connecting ...
Connected to "prodcdb"
Connected as SYSDBA.
New primary database "prodcdb" is opening...
Oracle Clusterware is restarting database "stby2cdb" ...
Switchover succeeded, new primary is "prodcdb"
DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  prodcdb  - Primary database
    stbycdb  - Physical standby database
    stby2cdb - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 61 seconds ago)

DGMGRL> switchover to stbycdb
Performing switchover NOW, please wait...
Operation requires a connection to database "stbycdb"
Connecting ...
Connected to "stbycdb"
Connected as SYSDBA.
New primary database "stbycdb" is opening...
Oracle Clusterware is restarting database "prodcdb" ...
Switchover succeeded, new primary is "stbycdb"
DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  stbycdb  - Primary database
    prodcdb  - Physical standby database
    stby2cdb - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 65 seconds ago)
This concludes the adding of new physical standby to an existing data guard configuration.

Related Posts
Creating Data Guard Broker for an Existing 12.2 Data Guard Setup with CDB
Oracle Data Guard on 12.2 CDB with Oracle Restart
Creating Data Guard Broker on 12c
Adding Far Sync Instances to Existing Data Guard Configuration
11gR2 RAC to RAC Data Guard
Data Guard Broker for 11gR2 RAC
11gR2 Standalone Data Guard (with ASM and Role Separation)
Setting Up Data Guard Broker for an Existing Data Guard Configuration with Far Sync
Enable Active Dataguard on 11gR2 RAC Standby