Friday, March 29, 2019

Backup and Recover Using DBCLI on OCI PaaS DB

OCI Paas DBs could be configured with automatic backup in which case OCI would take care of the backup. The database gets backed up to an object storage. But the backup files or the object storage bucket they are stored in is not visible from the console. This post shows the steps for backing up a PaaS DB to a user created object storage bucket using DBCLI. It is assumed a object store bucket is already created. For this post an object store bucket called "orabaks" was created. This will be used as the backup storage.

Backup using DBCLI
1. In order to use this object storage bucket it must be associated with an object store swift end point. Create object store swift end point as shown below. This uses tenancy and username used to login to the tenancy and the authentication token as the password for the "object store swift password". The end point URL must be changed to reflect the region where the DB resides. Refer previous post on dbcli command to find the region.
# dbcli create-objectstoreswift -n orabackupstore -t tenancyName -u asanga@email.com -e https://swiftobjectstorage.eu-frankfurt-1.oraclecloud.com/v1 -p
object store swift password:
{
  "jobId" : "3fa9e4fd-9da1-413d-9bca-e2434afb9229",
  "status" : "Created",
  "message" : "Create object store swift",
  "reports" : [ ],
  "createTimestamp" : "March 13, 2019 11:12:26 AM UTC",
  "resourceList" : [ {
    "resourceId" : "cb78ef29-a447-4b2d-af17-5b110a6717cb",
    "resourceType" : "ObjectStoreSwift",
    "jobId" : "3fa9e4fd-9da1-413d-9bca-e2434afb9229",
    "updatedTime" : "March 13, 2019 11:12:26 AM UTC"
  } ],
  "description" : "create object store:orabackupstore",
  "updatedTime" : "March 13, 2019 11:12:26 AM UTC"
}
# dbcli list-objectstoreswifts

ID                                       Name                 UserName                       TenantName                Url
---------------------------------------- -------------------- ------------------------------ ------------------------- ---------------------------------------------------------------------------
c9ffa569-7f15-4220-8524-0e6f294871a4     bnCGRmStQJ1DnEMR1p4a bnCGRmStQJ1DnEMR1p4a           dbbackupfra               https://swiftobjectstorage.eu-frankfurt-1.oraclecloud.com/v1
cb78ef29-a447-4b2d-af17-5b110a6717cb     orabackupstore       asanga@email.com               tenancyName               https://swiftobjectstorage.eu-frankfurt-1.oraclecloud.com/v1

2. Create a backup configuration using the object store swift ID created in the previous step and the object store name (orabaks). The backup destination is listed as object store and this could be changed to disk for disk backups, in which case backups will be created in the FRA.
# dbcli create-backupconfig -d OBJECTSTORE -c orabaks -o cb78ef29-a447-4b2d-af17-5b110a6717cb -w 1 -n mybakconfig1
{
  "jobId" : "0ecd1ed3-658a-4374-ad42-105f80e5480d",
  "status" : "Created",
  "message" : "backup config creation",
  "reports" : [ ],
  "createTimestamp" : "March 13, 2019 11:21:05 AM UTC",
  "resourceList" : [ {
    "resourceId" : "09c01611-6a58-49d1-b9de-553594b73cfc",
    "resourceType" : "BackupConfig",
    "jobId" : "0ecd1ed3-658a-4374-ad42-105f80e5480d",
    "updatedTime" : "March 13, 2019 11:21:05 AM UTC"
  } ],
  "description" : "create backup config:mybakconfig1",
  "updatedTime" : "March 13, 2019 11:21:05 AM UTC"
}

dbcli list-backupconfigs

ID                                       Name                 RecoveryWindow   CrosscheckEnabled   BackupDestination
---------------------------------------- -------------------- ---------------- ------------------- --------------------
2e6e46e9-9dca-4906-a2e1-27b8df727152     bnCGRmStQJ1DnEMR1p4a_BC 30               true                ObjectStore
09c01611-6a58-49d1-b9de-553594b73cfc     mybakconfig1         1                   true                ObjectStore


dbcli describe-backupconfig -i 09c01611-6a58-49d1-b9de-553594b73cfc
Backup Config details
----------------------------------------------------------------
                     ID: 09c01611-6a58-49d1-b9de-553594b73cfc
                   Name: mybakconfig1
      CrosscheckEnabled: true
         RecoveryWindow: 1
      BackupDestination: ObjectStore
         BackupLocation: orabaks
          ObjectStoreId: cb78ef29-a447-4b2d-af17-5b110a6717cb
            CreatedTime: March 13, 2019 11:21:05 AM UTC
            UpdatedTime: March 13, 2019 11:21:34 AM UTC

3. Next step is to associate the newly created database configuration with the database. For this the database id is needed. Use the following command to find the DB id and the ID of the current backup configuration associated with the DB.
dbcli list-databases

ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
690f6c09-7c48-4332-ab65-b86f932c1a33     apcdb      Si       18.5.0.0.190115      true       Oltp              ASM        Configured   0f59f70a-167b-4d8a-84e3-6e7d7529b093

dbcli describe-database -i 690f6c09-7c48-4332-ab65-b86f932c1a33

Database details
----------------------------------------------------------------
                     ID: 690f6c09-7c48-4332-ab65-b86f932c1a33
            Description: apcdb
                DB Name: apcdb
             DB Version: 18.5.0.0.190115
                DB Type: Si
             DB Edition: EE
                   DBID: 1068299060
 Instance Only Database: false
                    CDB: true
               PDB Name: cgpdb
    PDB Admin User Name: pdbuser
                  Class: Oltp
                  Shape:
                Storage: ASM
          DB Redundancy:
           CharacterSet: AL32UTF8
  National CharacterSet: AL16UTF16
               Language: AMERICAN
              Territory: AMERICA
                Home ID: 0f59f70a-167b-4d8a-84e3-6e7d7529b093
        Console Enabled: false
     Level 0 Backup Day: Sunday
     AutoBackup Enabled: false
         BackupConfigId: 2e6e46e9-9dca-4906-a2e1-27b8df727152
                Created: March 11, 2019 9:06:44 AM UTC
         DB Domain Name: dbsubnet.fravcn.oraclevcn.com

4. To associate the new backup configuration with the database use the update database command.
dbcli update-database -i 690f6c09-7c48-4332-ab65-b86f932c1a33 -bi 09c01611-6a58-49d1-b9de-553594b73cfc
{
  "jobId" : "597671db-4db5-4a1e-91af-325d8c9ae39e",
  "status" : "Created",
  "message" : "update database",
  "reports" : [ ],
  "createTimestamp" : "March 13, 2019 11:26:50 AM UTC",
  "resourceList" : [ {
    "resourceId" : "690f6c09-7c48-4332-ab65-b86f932c1a33",
    "resourceType" : "DB",
    "jobId" : "597671db-4db5-4a1e-91af-325d8c9ae39e",
    "updatedTime" : "March 13, 2019 11:26:50 AM UTC"
  } ],
  "description" : "update database : apcdb",
  "updatedTime" : "March 13, 2019 11:26:50 AM UTC"
}

 dbcli describe-database -i 690f6c09-7c48-4332-ab65-b86f932c1a33

Database details
----------------------------------------------------------------
                     ID: 690f6c09-7c48-4332-ab65-b86f932c1a33
            Description: apcdb
                DB Name: apcdb
             DB Version: 18.5.0.0.190115
                DB Type: Si
             DB Edition: EE
                   DBID: 1068299060
 Instance Only Database: false
                    CDB: true
               PDB Name: cgpdb
    PDB Admin User Name: pdbuser
                  Class: Oltp
                  Shape:
                Storage: ASM
          DB Redundancy:
           CharacterSet: AL32UTF8
  National CharacterSet: AL16UTF16
               Language: AMERICAN
              Territory: AMERICA
                Home ID: 0f59f70a-167b-4d8a-84e3-6e7d7529b093
        Console Enabled: false
     Level 0 Backup Day: Sunday
     AutoBackup Enabled: false
         BackupConfigId: 09c01611-6a58-49d1-b9de-553594b73cfc
                Created: March 11, 2019 9:06:44 AM UTC
         DB Domain Name: dbsubnet.fravcn.oraclevcn.com
5. Create a database backup by specifying the DB id. The backups files will be stored in the user created object store created (orabaks). The following command shows creating an archive log backup.
dbcli create-backup -bt ArchiveLog -i 690f6c09-7c48-4332-ab65-b86f932c1a33
{
  "jobId" : "047bbf1e-2ef2-4efd-a0dd-663d0b2c4456",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "March 13, 2019 11:34:01 AM UTC",
  "resourceList" : [ ],
  "description" : "Create ArchiveLog Backup with TAG-auto for Db:apcdb in OSS:orabaks",
  "updatedTime" : "March 13, 2019 11:34:01 AM UTC"
}

dbcli describe-job -i 047bbf1e-2ef2-4efd-a0dd-663d0b2c4456

Job details
----------------------------------------------------------------
                     ID:  047bbf1e-2ef2-4efd-a0dd-663d0b2c4456
            Description:  Create ArchiveLog Backup with TAG-auto for Db:apcdb in OSS:orabaks
                 Status:  Success
                Created:  March 13, 2019 11:34:01 AM UTC
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validate backup config                   March 13, 2019 11:34:04 AM UTC      March 13, 2019 11:34:04 AM UTC      Success
Database container validation            March 13, 2019 11:34:04 AM UTC      March 13, 2019 11:34:05 AM UTC      Success
libopc existence check                   March 13, 2019 11:34:05 AM UTC      March 13, 2019 11:34:05 AM UTC      Success
Backup Validations                       March 13, 2019 11:34:05 AM UTC      March 13, 2019 11:34:54 AM UTC      Success
Recovery Window validation               March 13, 2019 11:34:54 AM UTC      March 13, 2019 11:35:17 AM UTC      Success
Archivelog deletion policy configuration March 13, 2019 11:35:17 AM UTC      March 13, 2019 11:35:37 AM UTC      Success
Archive log backup                       March 13, 2019 11:35:37 AM UTC      March 13, 2019 11:38:41 AM UTC      Success
Backup TDE Wallet files to ObjectStore   March 13, 2019 11:38:41 AM UTC      March 13, 2019 11:38:42 AM UTC      Success
The created backup files are visible inside the object storage bucket.

6. A RMAN backup report could be created using the DB id. The rman report will list the log files which would have finer details of the backup.
dbcli create-rmanbackupreport -i 690f6c09-7c48-4332-ab65-b86f932c1a33 -w detailed -rn archlogreport
{
  "jobId" : "20f74edf-dcbf-4bd7-89cd-51358d2ab7d6",
  "status" : "Created",
  "message" : "Rman BackupReport creation.",
  "reports" : [ ],
  "createTimestamp" : "March 13, 2019 11:44:45 AM UTC",
  "resourceList" : [ {
    "resourceId" : "34dbed6a-4182-4fb5-a1b8-8918a799485f",
    "resourceType" : "Report",
    "jobId" : "20f74edf-dcbf-4bd7-89cd-51358d2ab7d6",
    "updatedTime" : "March 13, 2019 11:44:45 AM UTC"
  } ],
  "description" : "Create detailed Backup Report ",
  "updatedTime" : "March 13, 2019 11:44:45 AM UTC"
}


dbcli list-rmanbackupreports

ID                                       Name                 ReportType DbId                                     createTime                          updatedTime
---------------------------------------- -------------------- ---------- ---------------------------------------- ----------------------------------- -----------------------------------
34dbed6a-4182-4fb5-a1b8-8918a799485f     archlogreport        detailed   690f6c09-7c48-4332-ab65-b86f932c1a33     March 13, 2019 11:44:45 AM UTC      March 13, 2019 11:45:36 AM UTC

describe-rmanbackupreport -i 34dbed6a-4182-4fb5-a1b8-8918a799485f
Backup Report details
----------------------------------------------------------------
                     ID: 34dbed6a-4182-4fb5-a1b8-8918a799485f
            Report Type: detailed
               Location: Node apdb: /opt/oracle/dcs/log/apdb/rman/bkup/apcdb_fra2g6/rman_list_backup_detail/2019-03-13/rman_list_backup_detail_2019-03-13_11-45-20.0508.log
            Database ID: 690f6c09-7c48-4332-ab65-b86f932c1a33
            CreatedTime: March 13, 2019 11:44:45 AM UTC

ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
b5d96197-91ec-4319-abf7-09b39d70bafb     Create Regular-L0 Backup with TAG-auto for Db:apcdb in OSS:orabaks          March 13, 2019 12:01:36 PM UTC      Success
8bc91a59-2bee-4b9e-aa59-b73048adca38     Create Archivelog Backup with TAG-auto for Db:apcdb in OSS:orabaks          March 13, 2019 12:53:59 PM UTC      Success

Backup report could be deleted using the following command.
dbcli delete-rmanbackupreport -i 34dbed6a-4182-4fb5-a1b8-8918a799485f
To remove a backup configuration, if its assign to the DB then update the DB with a different backup configuration. Use the following command to delete the backup configuration by specifying backup configuration id.
dbcli delete-backupconfig -i 09c01611-6a58-49d1-b9de-553594b73cfc
To delete a object store swift use the following command, specifying object store swift id.
dbcli delete-objectstoreswift -i cb78ef29-a447-4b2d-af17-5b110a6717cb



Recover using DBCLI
Following steps shows point in time recovery using DBCLI to simulate a recovery from accidental drop of a table. DBCLI also provides other recovery types as well (Latest|SCN).
1. The table is droped using purge option.
SQL> conn asanga/asanga#1234$ABC@pdbsrvx
Connected.
SQL> ! date
Wed Mar 13 12:58:37 UTC 2019

SQL> drop table x purge;
Table dropped.
2. Use recover database command to execute a PITR. The recovery time stamp (specified by -r) is just before the table was dropped.
dbcli recover-database -i 690f6c09-7c48-4332-ab65-b86f932c1a33 -r 03/13/2019 12:58:35 -t PITR
{
  "jobId" : "037b6bfd-1bc3-4c2e-af5e-1a22dbc05189",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "March 13, 2019 13:01:57 PM UTC",
  "resourceList" : [ ],
  "description" : "Create recovery-pitr : time '03/13/2019 12:58:35' for db : apcdb",
  "updatedTime" : "March 13, 2019 13:01:57 PM UTC"
}
Monitor the recovery using the recovery job id given above.
dbcli describe-job -i 037b6bfd-1bc3-4c2e-af5e-1a22dbc05189

Job details
----------------------------------------------------------------
                     ID:  037b6bfd-1bc3-4c2e-af5e-1a22dbc05189
            Description:  Create recovery-pitr : time '03/13/2019 12:58:35' for db : apcdb
                 Status:  Success
                Created:  March 13, 2019 1:01:57 PM UTC
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Database recovery validation             March 13, 2019 1:02:08 PM UTC       March 13, 2019 1:03:05 PM UTC       Success
Database recovery                        March 13, 2019 1:03:05 PM UTC       March 13, 2019 1:05:54 PM UTC       Success
Enable block change tracking             March 13, 2019 1:05:54 PM UTC       March 13, 2019 1:05:56 PM UTC       Success
Database opening                         March 13, 2019 1:05:56 PM UTC       March 13, 2019 1:06:53 PM UTC       Success
Database restart                         March 13, 2019 1:06:53 PM UTC       March 13, 2019 1:08:55 PM UTC       Success
Recovery metadata persistance            March 13, 2019 1:08:55 PM UTC       March 13, 2019 1:08:55 PM UTC       Success
Verify table is recovered
SQL> desc x
 Name     Null?    Type
 -------- ------- ----------------------------
 A                NUMBER