Tuesday, March 26, 2019

Updating the OS and Patching OCI PaaS DB Using DBCLI

OCI provide a command line interface for managing the PaaS DB (DBCLI) beside the console. This post shows the steps for updating the OS and patching the DB System and the DB using DBCLI.

Updating the OS
To update the OS the database server needs access to the YUM repository. If the DB system is in a private subnet then configure a NAT gateway to access the YUM repository. It is assumed that NAT gateway is configured and DB server can access the internet or egress traffic is allowed only to YUM repository location.
1. Check the region where the DB server resides.
# curl -s http://169.254.169.254/opc/v1/instance/ |grep region
  "region" : "eu-frankfurt-1",
2. In this case the region is eu-frankfurt-1. Replace the region name in below URL with actual region and get the yum repos file.
https://swiftobjectstorage.region_name.oraclecloud.com/v1/dbaaspatchstore/DBaaSOSPatches/oci_dbaas_ol6repo

# wget https://swiftobjectstorage.eu-frankfurt-1.oraclecloud.com/v1/dbaaspatchstore/DBaaSOSPatches/oci_dbaas_ol6repo -O /tmp/oci_dbaas_ol6repo
--2019-03-12 10:14:05--  https://swiftobjectstorage.eu-frankfurt-1.oraclecloud.com/v1/dbaaspatchstore/DBaaSOSPatches/oci_dbaas_ol6repo
...
Saving to: `/tmp/oci_dbaas_ol6repo'
3. Get the version lock file. Version lock file is needed to control the update. An update will occur only if a versionlock file has a valid update available to apply to the DB system.
# wget https://swiftobjectstorage.eu-frankfurt-1.oraclecloud.com/v1/dbaaspatchstore/DBaaSOSPatches/versionlock.list -O /tmp/versionlock.list
--2019-03-12 10:14:37--  https://swiftobjectstorage.eu-frankfurt-1.oraclecloud.com/v1/dbaaspatchstore/DBaaSOSPatches/versionlock.list
...
Saving to: `/tmp/versionlock.list'
4. Copy the downloaded files to their respective locations in the DB server
cp /tmp/oci_dbaas_ol6repo /etc/yum.repos.d/ol6.repo

cp /tmp/versionlock.list /etc/yum/pluginconf.d/versionlock.list
5. Open the yum repository file and enable the repository for the region. During this testing it was found that Frankfurt region was enabled by default.

6. Note the current kernel version and run the update command.
uname -r
4.1.12-124.20.3.el6uek.x86_64

yum update
Any warning similar to following could be ignored.
Warning: RPMDB altered outside of yum.
** Found 3 pre-existing rpmdb problem(s), 'yum check' output follows:
oda-hw-mgmt-18.3.1.0.0_LINUX.X64_181212-1.x86_64 has missing requires of libnfsodm18.so()(64bit)
oda-hw-mgmt-18.3.1.0.0_LINUX.X64_181212-1.x86_64 has missing requires of perl(GridDefParams)
oda-hw-mgmt-18.3.1.0.0_LINUX.X64_181212-1.x86_64 has missing requires of perl(s_GridSteps)
Also towards the end of the update following error.
ol6_UEKR4/filelists_db                                                                                                                                                                                                | 1.6 MB     00:00
Uploading /boot/vmlinuz-4.1.12-124.25.1.el6uek.x86_64 to http://169.254.0.3/kernel

Error uploading /kernel: 502 - <html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
</body>
</html>
In spite of this error the kernel gets updated as shown below

6. Once the update is complete reboot the server and check the kernel version.
# reboot
# uname -r
4.1.12-124.25.1.el6uek.x86_64



Patching the PaaS DB
Patching the DB is of two parts similar to patching using console. First is to patch the DB system (Grid Infrastructure) and then the DB.

Updating the DBCLI
Before the patching could begin check if there's a newer version of DBCLI and update DBCLI.
cliadm update-dbcli

Job details
----------------------------------------------------------------
                     ID:  d0a0e701-3c5a-4f72-a97e-455efd02cefd
            Description:  DcsCli patching
                 Status:  Created
                Created:  March 12, 2019 10:59:31 AM UTC
                Message:  Dcs cli will be updated
Wait couple of minutes and check the outcome of the update, (use the job id given)
dbcli describe-job -i d0a0e701-3c5a-4f72-a97e-455efd02cefd

Job details
----------------------------------------------------------------
                     ID:  d0a0e701-3c5a-4f72-a97e-455efd02cefd
            Description:  DcsCli patching
                 Status:  Success
                Created:  March 12, 2019 10:59:31 AM UTC
                Message:

Task Name           Start Time                          End Time                            Status
------------------- -------------------------------     ----------------------------------- ----------
dcs-cli upgrade     March 12, 2019 10:59:32 AM UTC      March 12, 2019 10:59:34 AM UTC      Success
Check the currently available patches. This command list available patches for all versions.
# dbcli describe-latestpatch
componentType   availableVersion
--------------- --------------------
gi              12.2.0.1.190115
gi              12.1.0.2.190115
gi              18.5.0.0.190115
db              11.2.0.4.190115
db              12.2.0.1.190115
db              12.1.0.2.190115
db              18.5.0.0.190115
To list the patching available for installed GI and DB run the following command.
# dbcli describe-component
System Version
---------------
18.3.3.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
GI                                        18.3.0.0.180717       18.5.0.0.190115
DB                                        18.3.0.0.180717       18.5.0.0.190115
Patching the DB System
To patch the DB system run update server command.
# dbcli update-server
{
  "jobId" : "058707c8-a9ec-48f0-ba47-47e5aa4f65c6",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "March 12, 2019 11:05:00 AM UTC",
  "resourceList" : [ ],
  "description" : "Server Patching",
  "updatedTime" : "March 12, 2019 11:05:01 AM UTC"
}
The dcs agent log could be used to monitor the patch apply in progress
cd /opt/oracle/dcs/log
tail -100f dcs-agent.log
Another method is to use the describe job option with the job id.
dbcli describe-job -i "058707c8-a9ec-48f0-ba47-47e5aa4f65c6"

Job details
----------------------------------------------------------------
                     ID:  058707c8-a9ec-48f0-ba47-47e5aa4f65c6
            Description:  Server Patching
                 Status:  Running
                Created:  March 12, 2019 11:05:00 AM UTC
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Create Patching Repository Directories   March 12, 2019 11:05:01 AM UTC      March 12, 2019 11:05:01 AM UTC      Success
Download latest patch metadata           March 12, 2019 11:05:01 AM UTC      March 12, 2019 11:05:01 AM UTC      Success
Update Patching Repository               March 12, 2019 11:05:02 AM UTC      March 12, 2019 11:05:02 AM UTC      Running
Once the patching is complete the status will change to success.
dbcli describe-job -i "058707c8-a9ec-48f0-ba47-47e5aa4f65c6"

Job details
----------------------------------------------------------------
                     ID:  058707c8-a9ec-48f0-ba47-47e5aa4f65c6
            Description:  Server Patching
                 Status:  Success
                Created:  March 12, 2019 11:05:00 AM UTC
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Create Patching Repository Directories   March 12, 2019 11:05:01 AM UTC      March 12, 2019 11:05:01 AM UTC      Success
Download latest patch metadata           March 12, 2019 11:05:01 AM UTC      March 12, 2019 11:05:01 AM UTC      Success
Update Patching Repository               March 12, 2019 11:05:02 AM UTC      March 12, 2019 11:08:24 AM UTC      Success
Opatch updation                          March 12, 2019 11:08:25 AM UTC      March 12, 2019 11:08:30 AM UTC      Success
Patch conflict check                     March 12, 2019 11:08:31 AM UTC      March 12, 2019 11:10:31 AM UTC      Success
apply clusterware patch                  March 12, 2019 11:10:31 AM UTC      March 12, 2019 11:32:08 AM UTC      Success
Updating GiHome version                  March 12, 2019 11:32:08 AM UTC      March 12, 2019 11:32:35 AM UTC      Success
Update System version                    March 12, 2019 11:32:52 AM UTC      March 12, 2019 11:32:52 AM UTC      Success
Use the component describe command again to see the patched version. Since the latest patch was applied the available column will state "up-to-date"
dbcli describe-component
System Version
---------------
18.3.3.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
GI                                        18.5.0.0.190115       up-to-date
DB                                         18.3.0.0.180717       18.5.0.0.190115
Patching the DB
Patching of the DB happens by specifying the DB home to patch. For this the DB home ID is needed. To find the DB home ID use the list home command.
dbcli list-dbhomes

ID                                       Name                 DB Version             Home Location                                 Status
---------------------------------------- -------------------- ---------------------- --------------------------------------------- ----------
0f59f70a-167b-4d8a-84e3-6e7d7529b093     OraDB18000_home1     18.3.0.0.180717        /u01/app/oracle/product/18.0.0.0/dbhome_1     Configured
Run the home patching command specifying the DB Home ID.
# dbcli update-dbhome -i 0f59f70a-167b-4d8a-84e3-6e7d7529b093
{
  "jobId" : "dae78453-e3a4-4b6e-ba6b-baaeb8902f2f",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "March 12, 2019 11:37:09 AM UTC",
  "resourceList" : [ ],
  "description" : "DB Home Patching: Home Id is 0f59f70a-167b-4d8a-84e3-6e7d7529b093",
  "updatedTime" : "March 12, 2019 11:37:09 AM UTC"
}
Use the job describe command to find the status of the job.
# dbcli describe-job -i "dae78453-e3a4-4b6e-ba6b-baaeb8902f2f"

Job details
----------------------------------------------------------------
                     ID:  dae78453-e3a4-4b6e-ba6b-baaeb8902f2f
            Description:  DB Home Patching: Home Id is 0f59f70a-167b-4d8a-84e3-6e7d7529b093
                 Status:  Success
                Created:  March 12, 2019 11:37:09 AM UTC
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Create Patching Repository Directories   March 12, 2019 11:37:09 AM UTC      March 12, 2019 11:37:09 AM UTC      Success
Download latest patch metadata           March 12, 2019 11:37:09 AM UTC      March 12, 2019 11:37:09 AM UTC      Success
checking GiHome version                  March 12, 2019 11:37:09 AM UTC      March 12, 2019 11:37:09 AM UTC      Success
Update System version                    March 12, 2019 11:37:09 AM UTC      March 12, 2019 11:37:09 AM UTC      Success
Update Patching Repository               March 12, 2019 11:37:09 AM UTC      March 12, 2019 11:37:55 AM UTC      Success
Validating dbHome available space        March 12, 2019 11:37:55 AM UTC      March 12, 2019 11:37:55 AM UTC      Success
Opatch updation                          March 12, 2019 11:37:56 AM UTC      March 12, 2019 11:38:00 AM UTC      Success
Patch conflict check                     March 12, 2019 11:38:00 AM UTC      March 12, 2019 11:41:00 AM UTC      Success
db upgrade                               March 12, 2019 11:41:00 AM UTC      March 12, 2019 11:59:06 AM UTC      Success
SqlPatch upgrade                         March 12, 2019 11:59:06 AM UTC      March 12, 2019 12:02:14 PM UTC      Success
updating the Database version            March 12, 2019 12:02:29 PM UTC      March 12, 2019 12:02:43 PM UTC      Success
The list component will show the DB is also up-to-date.
# dbcli describe-component
System Version
---------------
18.3.3.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
GI                                        18.5.0.0.190115       up-to-date
DB                                        18.5.0.0.190115       up-to-date
Querying the DB will show that sql patch registry is updated.
select PATCH_ID,PATCH_TYPE,ACTION,STATUS,SOURCE_VERSION,TARGET_VERSION,ACTION_TIME from dba_registry_sqlpatch;

  PATCH_ID PATCH_TYPE ACTION          STATUS                    SOURCE_VERSION  TARGET_VERSION  ACTION_TIME
---------- ---------- --------------- ------------------------- --------------- --------------- ------------------------------
  28090523 RU         APPLY           SUCCESS                   18.1.0.0.0      18.3.0.0.0      06-AUG-18 07.10.59.424957 PM
  27923415 INTERIM    APPLY           SUCCESS                   18.1.0.0.0      18.1.0.0.0      06-AUG-18 07.10.59.428991 PM
  28822489 RU         APPLY           SUCCESS                   18.3.0.0.0      18.5.0.0.0      12-MAR-19 11.55.48.034138 AM
  29143230 INTERIM    APPLY           SUCCESS                   18.5.0.0.0      18.5.0.0.0      12-MAR-19 12.00.42.612755 PM
The patched version of the DB System and DB was not updated on the console view. The console showed the pre-patch version for the DB system and DB. One way to rectify this was to run the patch from the console. This will not reapply the patch. But during the pre-patch check the applied patch will be detected and console will update itself with the current patch information.