Getting DB Passwords from Vault Secrets for OKE Deployments

OCI allows storing of DB passwords as secrets in the vault. These secrets could be retreived using various SDKs, CLI and etc. This post shows how DB password could be retreived from a vault for JDBC Connection pools when a java application is deployed in OKE.
1. In order to retreivew the secrets from vault the user making the request must be authenticated. Instance principal is used to avoid using a password based authentication for this. OCI allows dynamic group be created based compartment, instance id, tag and tag value. As a first step create a dynamic group specifying the compartment where OKE worker node resides and some qualifying tags. The dynamic group is called "test dynamic group".

Compartment is used instead of instance ID in the dynamic group creation. This is due the fact that new worker nodes could be created and old ones destroyed as part of the life cycle of the OKE cluster. Tag values have been used to further reduce the number of instances that qualify for the dynamic group. All worker nodes would have "oke" as the created-by tag value. So this tag could be used to distinguish between OKE cluster service created instances vs other compute instnaces. Further reduction could be made based on project, enviornment and etc.
2. Create the secrets in the vault. Secrets could be created with a prefix which would allow writing of policy capturing only the secrets with the specified prefix text. Below two secrets have been created both with prefix "acme_test_prod".

3. Write a policy allowing dynamic group to access the secret bundles. Use the vault id and prefix of the secret to restrict the dynamic group to specific set of secrets and vaults. Below policy would allow test_dynamic_group to get all the secret bundles with names begining with "acme_test_prod" in the specified vault id.

Policy is written for secret bundles as that's what the java API expect. If this is done for OCI CLI then policy could be written for secrets instead of secret bundles. Java API access fails when policy only allow access to secrets instead of secret bundles.



4. Final step set is to write the java code that would be deployed as part of the application into OKE cluster. Download the java SDK from the link here. Below is an example java code that uses instance principal provider to create a secrets client that could be used to retreive the secrets from the vault.

final InstancePrincipalsAuthenticationDetailsProvider provider;
        try {
            provider = InstancePrincipalsAuthenticationDetailsProvider.builder().build();
        } catch (Exception e) {
            
            throw e;
        }
    
        SecretsClient secretsDpClient  = new SecretsClient(provider);
                
        GetSecretBundleByNameResponse getSecretBundleByNameResponse = secretsDpClient.getSecretBundleByName(GetSecretBundleByNameRequest.builder()
        .secretName("acme_test_prod_schema1").vaultId("ocid1.vault.oc1.vault id here...").build());
        
        Base64SecretBundleContentDetails details = (Base64SecretBundleContentDetails) getSecretBundleByNameResponse.getSecretBundle().getSecretBundleContent();
        byte[] content = Base64.getDecoder().decode(details.getContent());
        //System.out.println("Password : "new String(content));
        
        PoolDataSource ds = PoolDataSourceFactory.getPoolDataSource();
        ds.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource");
        ds.setURL("jdbc:oracle:thin:@test");
        ds.setUser("asanga");
        ds.setPassword(new String(content));

Accessing OCI Bucket Using Instance Principal

This post shows how to access a OCI bucket using instance principal. Being able to access a bucket based on instance pricipal is useful as it eliminate the need for sharing passwords. Access to OCI storage buckets may be needed for example when importing a data dump into ATP. With instance princiapl a compute instance could be designated as the uploader of the dump files to bucket which could be imported into ATP later.
1. If not already done create a storage bucket.

2. Have a compute instance that would be used to create the instance principal

3. If the compute instance is in a private subnet then create a service gateway and have a rourting rule to sending service related traffic through it.

4. Create a dynamic group using the compute instance's ID as the value to match.

5. Creaet a policy allowing the dynamic group to read the bucket and manage the objects in the bucket.

6. Install OCI client. This only needs to be installed. No need to configure it. If the compute instance is OEL then this would be done via yum.

7. To check if the instance principal works and bucket is accessible, set the OCI_CLI_AUTH=instance_principal and run a list command against tbe bucket using oci cli. Specify the namespace of the OCI tenancy after -ns (omitted in screenshow below).

Output above shows the content of the bucket. It contain just one file called asanga.dmp.
Another way to specify the authentication in the oci cli is to use the --auth. This eliminate the need to set an environment variable.

ORA-00800: soft external error, arguments: [Set Priority Failed], [VKTM], [Check traces and OS configuration]

While adding a new standby node to the existing DG configuration noticed the following in the new standby databases's alert log.

Starting background process VKTM
2021-03-30T00:08:42.781044+10:00
Errors in file /opt/app/oracle/diag/rdbms/dbx12/dbx12/trace/dbx12_vktm_32410.trc  (incident=41):
ORA-00800: soft external error, arguments: [Set Priority Failed], [VKTM], [Check traces and OS configuration], [Check Oracle document and MOS notes], []
Incident details in: /opt/app/oracle/diag/rdbms/dbx12/dbx12/incident/incdir_41/dbx12_vktm_32410_i41.trc
2021-03-30T00:08:42.782979+10:00
Error attempting to elevate VKTM's priority: no further priority changes will be attempted for this process
VKTM started with pid=5, OS id=32410

MOS Doc 2718971.1 gives a workaround for this issue (seems this doc has now been made internal).
The problme was related to cgroup setup. In a database setup where everythig is working fine the cgroup for the VKTM would have /. For example if VKTM PID is 5207 then following could be be used to find out cgroup setting

cat /proc/5207/cgroup | grep cpu
10:cpu,cpuacct:/
6:cpuset:/

Any other setting would mean VKTM would run into above issue.
One of the solutions is to set the hidden parameter _high_priority_processes='VKTM'. But this was already set to TRUE so not going to be a solution.
The problem server had the following cgroup setting.

# ps -eaf | grep -i vktm | grep -v grep
oracle    3315     1  0 17:53 ?        00:00:00 ora_vktm_dbx12
oracle    3357     1  0 14:19 ?        00:01:40 asm_vktm_+ASM
# cat /proc/3315/cgroup | grep cpu
11:cpuset:/
6:cpu,cpuacct:/user.slice

So the next workround was to set the kernel parameter as below.

# echo 0 > /sys/fs/cgroup/cpu,cpuacct/system.slice/cpu.rt_runtime_us
# echo 950000 > /sys/fs/cgroup/cpu,cpuacct/user.slice/cpu.rt_runtime_us

This seem to resolve the situation and was able stop and start the standby instance without the above error message appaering on the alert log.



However, the question remains why did the cgroup change. This was the first incident of facing this error. Since this is adding a standby database server to exisitng setup there was a reference point to check against any changes. So first the OS. The "good" servers had OEL 7.9 while the "problem" server had OEL 7.7. Both servers are Azure VMs
Then decided to check inside cgroup setting. The good server had following (no user.slice)

 ls -l /sys/fs/cgroup/cpu,cpuacct/
drwxr-xr-x. 3 root root 0 Apr  7 09:32 WALinuxAgent
-rw-r--r--. 1 root root 0 Apr  7 09:32 tasks
-rw-r--r--. 1 root root 0 Apr  7 09:32 cgroup.procs
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.cfs_period_us
-r--r--r--. 1 root root 0 Apr  7 09:32 cgroup.sane_behavior
-r--r--r--. 1 root root 0 Apr  7 09:32 cpu.stat
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_percpu_sys
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.shares
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_percpu
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.stat
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.cfs_quota_us
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_sys
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_all
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_percpu_user
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.rt_runtime_us
-rw-r--r--. 1 root root 0 Apr  7 09:32 notify_on_release
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.rt_period_us
-rw-r--r--. 1 root root 0 Apr  7 09:32 release_agent
-rw-r--r--. 1 root root 0 Apr  7 09:32 cgroup.clone_children
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_user

While the problem server had the following

drwxr-xr-x.  2 root root 0 Apr  7 09:29 auoms
drwxr-xr-x.  2 root root 0 Apr  7 09:29 auomscollect
-rw-r--r--.  1 root root 0 Apr  7 09:29 cgroup.clone_children
-rw-r--r--.  1 root root 0 Apr  7 09:29 cgroup.procs
-r--r--r--.  1 root root 0 Apr  7 09:29 cgroup.sane_behavior
-r--r--r--.  1 root root 0 Apr  7 09:29 cpuacct.stat
-rw-r--r--.  1 root root 0 Apr  7 09:29 cpuacct.usage
-r--r--r--.  1 root root 0 Apr  7 09:29 cpuacct.usage_all
-r--r--r--.  1 root root 0 Apr  7 09:29 cpuacct.usage_percpu
-r--r--r--.  1 root root 0 Apr  7 09:29 cpuacct.usage_percpu_sys
-r--r--r--.  1 root root 0 Apr  7 09:29 cpuacct.usage_percpu_user
-r--r--r--.  1 root root 0 Apr  7 09:29 cpuacct.usage_sys
-r--r--r--.  1 root root 0 Apr  7 09:29 cpuacct.usage_user
-rw-r--r--.  1 root root 0 Apr  7 09:29 cpu.cfs_period_us
-rw-r--r--.  1 root root 0 Apr  7 09:29 cpu.cfs_quota_us
-rw-r--r--.  1 root root 0 Apr  7 09:29 cpu.rt_period_us
-rw-r--r--.  1 root root 0 Apr  7 09:29 cpu.rt_runtime_us
-rw-r--r--.  1 root root 0 Apr  7 09:29 cpu.shares
-r--r--r--.  1 root root 0 Apr  7 09:29 cpu.stat
-rw-r--r--.  1 root root 0 Apr  7 09:29 notify_on_release
-rw-r--r--.  1 root root 0 Apr  7 09:29 release_agent
drwxr-xr-x. 69 root root 0 Apr  7 09:29 system.slice
-rw-r--r--.  1 root root 0 Apr  6 14:35 tasks
drwxr-xr-x.  2 root root 0 Apr  7 09:29 user.slice
drwxr-xr-x.  2 root root 0 Apr  7 09:29 WALinuxAgent

Beside user.slice the auoms* seem to be the different between the two. Auoms is Azure management agent plugin. Could this be the reason why cgroup has a user.slice? To test this out disableed auoms and restarted the server.

# systemctl stop auoms.service
# systemctl disable auoms.service
Removed symlink /etc/systemd/system/multi-user.target.wants/auoms.service.
Removed symlink /etc/systemd/system/auoms.service.
# /sbin/reboot

When the server restart the cgroup didn't have a user.slice

drwxr-xr-x. 3 root root 0 Apr  7 09:32 WALinuxAgent
-rw-r--r--. 1 root root 0 Apr  7 09:32 tasks
-rw-r--r--. 1 root root 0 Apr  7 09:32 cgroup.procs
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.cfs_period_us
-r--r--r--. 1 root root 0 Apr  7 09:32 cgroup.sane_behavior
-r--r--r--. 1 root root 0 Apr  7 09:32 cpu.stat
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_percpu_sys
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.shares
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_percpu
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.stat
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.cfs_quota_us
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_sys
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_all
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_percpu_user
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.rt_runtime_us
-rw-r--r--. 1 root root 0 Apr  7 09:32 notify_on_release
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.rt_period_us
-rw-r--r--. 1 root root 0 Apr  7 09:32 release_agent
-rw-r--r--. 1 root root 0 Apr  7 09:32 cgroup.clone_children
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_user

DB was starting without the VKTM related error on alert log (earlier set kernel parameters were not set to be persistent). VKTM had the / for cgroup.

 ps ax  | grep vktm
 3314 ?        Ss     0:03 asm_vktm_+ASM
 3506 ?        Ss     0:03 ora_vktm_dbx12
 4664 pts/0    S+     0:00 grep --color=auto vktm

# cat /proc/3506/cgroup | grep cpu
6:cpu,cpuacct:/
4:cpuset:/

There were several instances to be added to the DG and on other servers the OS was upgraded to 7.9 from 7.7. This upgrade seem to be remove the auoms and there were no cgroup issues.
If facing similar issues first check what has caused the cgroup change before attempting hidden parameter or kernel parameter workarounds.