Start of Service Fails with ORA-16000 on Physical Standby Open for Read Only

Start of database service failed with ora-16000 on physical standby where both CDB and PDB are open read only mode. The DG setup is same one mentioned in earlier post Data Guard on 12.2 CDB.

SQL> show con_name

CON_NAME
----------
CDB$ROOT

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ ONLY WITH APPLY

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDBAPP1                        READ ONLY  NO

srvctl add service -db stbycdb -pdb pdbapp1 -service abc -role PHYSICAL_STANDBY -notification TRUE -failovertype NONE -failovermethod NONE -failoverdelay 0 -failoverretry 0
srvctl start service -db stbycdb -s abc
PRCD-1084 : Failed to start service abc
PRCR-1079 : Failed to start resource ora.stbycdb.abc.svc
CRS-5017: The resource action "ora.stbycdb.abc.svc start" encountered the following error:
ORA-16000: database or pluggable database open for read-only access
ORA-06512: at "SYS.DBMS_SERVICE", line 5
ORA-06512: at "SYS.DBMS_SERVICE", line 288
ORA-06512: at line 1
. For details refer to "(:CLSN00107:)" in "/opt/app/oracle/diag/crs/city7s/crs/trace/ohasd_oraagent_grid.trc".

CRS-2674: Start of 'ora.stbycdb.abc.svc' on 'city7s' failed

Reason seems to be creating the service with physical standby role is trying to add some rows to the read only database. To fix problem first add the service to the primary with role as physical standby and start the service. It may seems odd to start a database service that it is defined for a physical standby role but that's what needed to resolve this. Once the service is started stop it on the primary before the next steps.

srvctl add service -db prodcdb -pdb pdbapp1 -service abc -role PHYSICAL_STANDBY -notification TRUE -failovertype NONE -failovermethod NONE -failoverdelay 0 -failoverretry 0
srvctl start service -db prodcdb -s abc

lsnrctl status

...
  Instance "prodcdb", status READY, has 1 handler(s) for this service...
Service "prodcdb_DGB" has 1 instance(s).
  Instance "prodcdb", status READY, has 1 handler(s) for this service...
Service "prodcdb_DGMGRL" has 1 instance(s).
  Instance "prodcdb", status UNKNOWN, has 1 handler(s) for this service...
Service "abc" has 1 instance(s).
  Instance "prodcdb", status READY, has 1 handler(s) for this service...
The command completed successfully

srvctl stop service -db prodcdb -s abc

Do few log switches and wait until these logs are applied on the standby. Once logs are applied on standby create and start the service on standby.

srvctl add service -db stbycdb -pdb pdbapp1 -service abc -role PHYSICAL_STANDBY -notification TRUE -failovertype NONE -failovermethod NONE -failoverdelay 0 -failoverretry 0
srvctl start service -db stbycdb -s abc

lsnrctl status

...
  Instance "stbycdb", status READY, has 1 handler(s) for this service...
Service "stbycdb_DGB" has 1 instance(s).
  Instance "stbycdb", status READY, has 1 handler(s) for this service...
Service "stbycdb_DGMGRL" has 1 instance(s).
  Instance "stbycdb", status UNKNOWN, has 1 handler(s) for this service...
Service "abc" has 1 instance(s).
  Instance "stbycdb", status READY, has 1 handler(s) for this service...
The command completed successfully

Useful metalink notes
ORA-16000 Cannot Enable Auto Open of PDB On Physical Standby [ID 2377174.1]
How to create a RAC Database Service With Physical Standby Role Option? [ID 1129143.1]