Showing posts with label audit. Show all posts
Showing posts with label audit. Show all posts

Wednesday, October 22, 2025

Unified Audit Relink is looking for gcc after applying RU 19.29

Following error was observed while relinking for unified audit option after applying RU 19.29.
echo $ORACLE_HOME
/opt/app/oracle/product/19.29.0/dbhome_1

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk uniaud_on ioracle ORACLE_HOME=$ORACLE_HOME
/bin/sh: /usr/bin/gcc: No such file or directory
expr: missing operand
Try 'expr --help' for more information.
/bin/sh: line 0: test: -gt: unary operator expected
/usr/bin/ar cr /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/libknlopt.a /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/kzaiang.o
chmod 755 /opt/app/oracle/product/19.29.0/dbhome_1/bin
cd /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/;\
/usr/bin/ar r  /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/libknlopt.a `/usr/bin/ar t /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/libknlopt.a` ;

 - Linking Oracle
rm -f /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/oracle
/opt/app/oracle/product/19.29.0/dbhome_1/bin/orald  -o /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/oracle -m64 -z noexecstack -Wl,--disable-new-dtags -L/opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/ -L/opt/app/oracle/product/19.29.0/dbhome_1/lib/ -L/opt/app/oracle/product/19.29.0/dbhome_1/lib/stubs/   -Wl,-E /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/opimai.o /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/ssoraed.o /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/ttcsoi.o -Wl,--whole-archive -lperfsrv19 -Wl,--no-whole-archive /opt/app/oracle/product/19.29.0/dbhome_1/lib/nautab.o /opt/app/oracle/product/19.29.0/dbhome_1/lib/naeet.o /opt/app/oracle/product/19.29.0/dbhome_1/lib/naect.o /opt/app/oracle/product/19.29.0/dbhome_1/lib/naedhs.o /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/config.o  -ldmext -lserver19 -lodm19 -lofs -lcell19 -lnnet19 -lskgxp19 -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lclient19  -lvsnst19 -lcommon19 -lgeneric19 -lknlopt -loraolap19 -lskjcx19 -lslax19 -lpls19  -lrt -lplp19 -ldmext -lserver19 -lclient19  -lvsnst19 -lcommon19 -lgeneric19 `if [ -f /opt/app/oracle/product/19.29.0/dbhome_1/lib/libavserver19.a ] ; then echo "-lavserver19" ; else echo "-lavstub19"; fi` `if [ -f /opt/app/oracle/product/19.29.0/dbhome_1/lib/libavclient19.a ] ; then echo "-lavclient19" ; fi` -lknlopt -lslax19 -lpls19  -lrt -lplp19 -ljavavm19 -lserver19  -lwwg  `cat /opt/app/oracle/product/19.29.0/dbhome_1/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnro19 `cat /opt/app/oracle/product/19.29.0/dbhome_1/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnnzst19 -lzt19 -lztkg19 -lmm -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lztkg19 `cat /opt/app/oracle/product/19.29.0/dbhome_1/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnro19 `cat /opt/app/oracle/product/19.29.0/dbhome_1/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnnzst19 -lzt19 -lztkg19   -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 `if /usr/bin/ar tv /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo19 -lserver19"; fi` -L/opt/app/oracle/product/19.29.0/dbhome_1/ctx/lib/ -lctxc19 -lctx19 -lzx19 -lgx19 -lctx19 -lzx19 -lgx19 -lclscest19 -loevm -lclsra19 -ldbcfg19 -lhasgen19 -lskgxn2 -lnnzst19 -lzt19 -lxml19 -lgeneric19 -locr19 -locrb19 -locrutl19 -lhasgen19 -lskgxn2 -lnnzst19 -lzt19 -lxml19 -lgeneric19  -lgeneric19 -lorazip -loraz -llzopro5 -lorabz2 -lorazstd -loralz4 -lipp_z -lipp_bz2 -lippdc -lipps -lippcore  -lippcp -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lsnls19 -lunls19  -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lasmclnt19 -lcommon19 -lcore19  -ledtn19 -laio -lons  -lmql1 -lipc1    -lfthread19    `cat /opt/app/oracle/product/19.29.0/dbhome_1/lib/sysliblist` -Wl,-rpath,/opt/app/oracle/product/19.29.0/dbhome_1/lib -lm    `cat /opt/app/oracle/product/19.29.0/dbhome_1/lib/sysliblist` -ldl -lm   -L/opt/app/oracle/product/19.29.0/dbhome_1/lib `test -x /usr/bin/hugeedit -a -r /usr/lib64/libhugetlbfs.so && test -r /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/shugetlbfs.o && echo -Wl,-zcommon-page-size=2097152 -Wl,-zmax-page-size=2097152 -lhugetlbfs`
rm -f /opt/app/oracle/product/19.29.0/dbhome_1/bin/oracle
mv /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/oracle /opt/app/oracle/product/19.29.0/dbhome_1/bin/oracle
chmod 6751 /opt/app/oracle/product/19.29.0/dbhome_1/bin/oracle
(if [ ! -f /opt/app/oracle/product/19.29.0/dbhome_1/bin/crsd.bin ]; then \
    getcrshome="/opt/app/oracle/product/19.29.0/dbhome_1/srvm/admin/getcrshome" ; \
    if [ -f "$getcrshome" ]; then \
        crshome="`$getcrshome`"; \
        if [ -n "$crshome" ]; then \
            if [ $crshome != /opt/app/oracle/product/19.29.0/dbhome_1 ]; then \
                oracle="/opt/app/oracle/product/19.29.0/dbhome_1/bin/oracle"; \
                $crshome/bin/setasmgidwrap oracle_binary_path=$oracle; \
            fi \
        fi \
    fi \
fi\
);

gcc and gcc-c++ are not required on Oracle Linux and Red Hat Enterprise Linux (refer 2668780.1 and 2982833.1).


This appears to be an issue introduced with 19.29. No such issue with RU 19.28.
echo $ORACLE_HOME
/opt/app/oracle/product/19.28.0/dbhome_1

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk uniaud_on ioracle ORACLE_HOME=$ORACLE_HOME
/usr/bin/ar cr /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/libknlopt.a /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/kzaiang.o
chmod 755 /opt/app/oracle/product/19.28.0/dbhome_1/bin
cd /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/;\
/usr/bin/ar r  /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/libknlopt.a `/usr/bin/ar t /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/libknlopt.a` ;

 - Linking Oracle
rm -f /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/oracle
/opt/app/oracle/product/19.28.0/dbhome_1/bin/orald  -o /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/oracle -m64 -z noexecstack -Wl,--disable-new-dtags -L/opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/ -L/opt/app/oracle/product/19.28.0/dbhome_1/lib/ -L/opt/app/oracle/product/19.28.0/dbhome_1/lib/stubs/   -Wl,-E /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/opimai.o /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/ssoraed.o /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/ttcsoi.o -Wl,--whole-archive -lperfsrv19 -Wl,--no-whole-archive /opt/app/oracle/product/19.28.0/dbhome_1/lib/nautab.o /opt/app/oracle/product/19.28.0/dbhome_1/lib/naeet.o /opt/app/oracle/product/19.28.0/dbhome_1/lib/naect.o /opt/app/oracle/product/19.28.0/dbhome_1/lib/naedhs.o /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/config.o  -ldmext -lserver19 -lodm19 -lofs -lcell19 -lnnet19 -lskgxp19 -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lclient19  -lvsnst19 -lcommon19 -lgeneric19 -lknlopt -loraolap19 -lskjcx19 -lslax19 -lpls19  -lrt -lplp19 -ldmext -lserver19 -lclient19  -lvsnst19 -lcommon19 -lgeneric19 `if [ -f /opt/app/oracle/product/19.28.0/dbhome_1/lib/libavserver19.a ] ; then echo "-lavserver19" ; else echo "-lavstub19"; fi` `if [ -f /opt/app/oracle/product/19.28.0/dbhome_1/lib/libavclient19.a ] ; then echo "-lavclient19" ; fi` -lknlopt -lslax19 -lpls19  -lrt -lplp19 -ljavavm19 -lserver19  -lwwg  `cat /opt/app/oracle/product/19.28.0/dbhome_1/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnro19 `cat /opt/app/oracle/product/19.28.0/dbhome_1/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnnzst19 -lzt19 -lztkg19 -lmm -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lztkg19 `cat /opt/app/oracle/product/19.28.0/dbhome_1/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnro19 `cat /opt/app/oracle/product/19.28.0/dbhome_1/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnnzst19 -lzt19 -lztkg19   -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 `if /usr/bin/ar tv /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo19 -lserver19"; fi` -L/opt/app/oracle/product/19.28.0/dbhome_1/ctx/lib/ -lctxc19 -lctx19 -lzx19 -lgx19 -lctx19 -lzx19 -lgx19 -lclscest19 -loevm -lclsra19 -ldbcfg19 -lhasgen19 -lskgxn2 -lnnzst19 -lzt19 -lxml19 -lgeneric19 -locr19 -locrb19 -locrutl19 -lhasgen19 -lskgxn2 -lnnzst19 -lzt19 -lxml19 -lgeneric19  -lgeneric19 -lorazip -loraz -llzopro5 -lorabz2 -lorazstd -loralz4 -lipp_z -lipp_bz2 -lippdc -lipps -lippcore  -lippcp -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lsnls19 -lunls19  -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lasmclnt19 -lcommon19 -lcore19  -ledtn19 -laio -lons  -lmql1 -lipc1    -lfthread19    `cat /opt/app/oracle/product/19.28.0/dbhome_1/lib/sysliblist` -Wl,-rpath,/opt/app/oracle/product/19.28.0/dbhome_1/lib -lm    `cat /opt/app/oracle/product/19.28.0/dbhome_1/lib/sysliblist` -ldl -lm   -L/opt/app/oracle/product/19.28.0/dbhome_1/lib `test -x /usr/bin/hugeedit -a -r /usr/lib64/libhugetlbfs.so && test -r /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/shugetlbfs.o && echo -Wl,-zcommon-page-size=2097152 -Wl,-zmax-page-size=2097152 -lhugetlbfs`
rm -f /opt/app/oracle/product/19.28.0/dbhome_1/bin/oracle
mv /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/oracle /opt/app/oracle/product/19.28.0/dbhome_1/bin/oracle
chmod 6751 /opt/app/oracle/product/19.28.0/dbhome_1/bin/oracle
(if [ ! -f /opt/app/oracle/product/19.28.0/dbhome_1/bin/crsd.bin ]; then \
    getcrshome="/opt/app/oracle/product/19.28.0/dbhome_1/srvm/admin/getcrshome" ; \
    if [ -f "$getcrshome" ]; then \
        crshome="`$getcrshome`"; \
        if [ -n "$crshome" ]; then \
            if [ $crshome != /opt/app/oracle/product/19.28.0/dbhome_1 ]; then \
                oracle="/opt/app/oracle/product/19.28.0/dbhome_1/bin/oracle"; \
                $crshome/bin/setasmgidwrap oracle_binary_path=$oracle; \
            fi \
        fi \
    fi \
fi\
);

Sunday, October 31, 2021

Huge Amount of Audit Records Generated After ASH Package Update

After a patch update on EM 13.4 cloud control the performance hub page showed the "ASH Package Version Status" warning (refer 2784094.1). Warning itself shows what needs to be done to address the issue.

If the CIS standard for 19c DB is strictly followed which asks to revoke execute privilege on dbms_lob from public then execute privilge on dbms_lob must be granted to either public of dbsnmp user (for all containers) for ash package update to work (refer 2699059.1).
Once the ash package is updated, activities on ash analytics page (simply monitroing and automatic page refreshes) results in huge amount of audit records being generated. For 24 hour period nearly 32GB of audit records were generated.
Majority of records are as below.
EVENT_TIMESTAMP                ACTION_NAM OBJECT_NAME          DBUSERNAME           OBJECT_SCHEMA        OS_USERNAM SQL_TEXT
------------------------------ ---------- -------------------- -------------------- -------------------- ---------- ----------------------------------------------------------------------------------------------------
                                                                                                                        FROM TABLE(GV$(CURSOR(

19-AUG-21 02.51.57.585441 PM   SELECT     USER_NAME_XML        C##MONITOR         DBSNMP               oracle     WITH  unified_ash AS (
                                                                                                                        SELECT /*+ NO_MERGE */ *
                                                                                                                        FROM TABLE(GV$(CURSOR(

19-AUG-21 02.51.57.594962 PM   SELECT     MAP_WAITCLASS_XML    C##MONITOR         DBSNMP               oracle     WITH  unified_ash AS (
                                                                                                                        SELECT /*+ NO_MERGE */ *
                                                                                                                        FROM TABLE(GV$(CURSOR(

19-AUG-21 02.51.57.606929 PM   SELECT     MAP_XID_XML          C##MONITOR         DBSNMP               oracle     WITH  unified_ash AS (
                                                                                                                        SELECT /*+ NO_MERGE */ *
                                                                                                                        FROM TABLE(GV$(CURSOR(

Everytime the performance page is refreshed (manual or auto) new set of audit records are added to the unified audit table.



The main cause of the audit record generation is the auditing of privilege "SELECT ANY DICTIONARY", which is also part of CIS standard. The auditing of this privilege was present even before the ash package update. What seem to have changed is now the user used for monitoring activity (c##monitor) seem to access the dbsnmp objects. c##montior user is granted only one role and that is EM_EXPRESS_BASIC.
To reduce the audit records and keep the ash package update, modify the audit policy by dropping the auditing of select any dictionary privilege. This will break the strict adherence to CIS.
Or simply do not update the ash package and ignore the warning. This was the case before patching and warning was shown. The number of audit records generated was low even with the use of select any dictionary privilege was being audited.
Only way to have the update and reduce the number of audit records being generated is to login to EM with dbsnmp user for monitoring instead of another user. DBSNMP user has more roles and privileges granted to it than EM_EXPRESS_BASIC. As such it may not be an option that is always be feasible.

Saturday, May 22, 2021

Importing Unified Audit Data From Standby

In a primary database importing audit data involves a export job with INCLUDE=AUDIT_TRAILS parameter. However, situation is different on standby as audit records are written to file system location. In a CDB the root container audit data on standby is available in a folder path similar to $ORACLE_BASE/audit/$ORACLE_SID while the audit data for the PDB is available in a path similar to $ORACLE_BASE/audit/$ORACLE_SID/GUID where GUID is the PDB's generic unique identifier.
The audit files are not text file but binary files. The strings command could be used to get glance at the content of the audit file.
strings ora_audit_0629.bin
ANG Spillover Audit File
ORAAUDNG
oracle
ip-172-31-11-54.eu-west-1.compute.internal
pts/0
(TYPE=(DATABASE));(CLIENT ADDRESS=((ADDRESS=(PROTOCOL=tcp)(HOST=172.31.11.54)(PORT=23959))));
LOCKTEST
sqlplus@ip-172-31-11-54.eu-west-1.compute.intern
9837
ORA_LOGON_FAILURES
LOCKTEST
However, this doesn't give all the information as seen from above there's no date information.
Also same audit file would be appended with audit data for multiple sessions. For example above audit record generated for login failure for session originating in sever 172.31.11.54. However, after a while more records could be seen on the same file such as below. Now there exists another login failure for client originating from server 172.31.15.132.
strings ora_audit_0629.bin
ANG Spillover Audit File
ORAAUDNG
oracle
ip-172-31-11-54.eu-west-1.compute.internal
pts/0
(TYPE=(DATABASE));(CLIENT ADDRESS=((ADDRESS=(PROTOCOL=tcp)(HOST=172.31.11.54)(PORT=23983))));
LOCKTEST
sqlplus@ip-172-31-11-54.eu-west-1.compute.intern
10309
ORA_LOGON_FAILURES
LOCKTESTq
ORAAUDNG
oracle
ip-172-31-15-132.eu-west-1.compute.internal
pts/0
(TYPE=(DATABASE));(CLIENT ADDRESS=((ADDRESS=(PROTOCOL=tcp)(HOST=172.31.15.132)(PORT=30643))));
LOCKTEST
sqlplus@ip-172-31-15-132.eu-west-1.compute.inter
10478
ORA_LOGON_FAILURES
LOCKTEST
Therefore audit file must be queried via unified_audit_trail to get fine details of audit records.



If the unified audit table is queried on the standby DB (if it is open in read only mode) this will show records from both primary DB (available via redo apply) and from standby DB (earlier post on this causing high file waits).
To view the audit data of just the standby DB, the easiest and simplest way is to copy the required audit files (either from the CDB root or PDB location) to a different (test or temporary) database that doesn't have any audit data of its own and query the unified_audit_trail. The location where files are copied to must be similar to $ORACLE_BASE/audit/$ORACLE_SID or $ORACLE_BASE/audit/$ORACLE_SID/GUID. With audit files in above the pre-defined location, they are picked up automatically from their external location and exposed via the unified_audit_trail.
However, if the audit records must be kept in the DB then to improt the audit records into the database use the following.
EXEC DBMS_AUDIT_MGMT.LOAD_UNIFIED_AUDIT_FILES;
This will import all the audit files records in the pre-defined audit file location into the database. At the end of this the audit files are removed from the file system.

Monday, November 30, 2020

High Disk file operations I/O Waits When Querying Unified Audit Trail

DB monitoring showed high waits for Disk file operations I/O event.
The select query originated from the EM console.
The database is running pure unified auditing (not mix mode) as such the dba_common_audit_trail part was of no concern. A simple select query on the unified audit trail also showed the same high wait times for Disk file operations I/O.
The issue was happening only on the CDB$root unified audit trail. The PDB audit trail had no issue.
Gathering statistics on audsys didn't help either.
exec dbms_STATS.GATHER_TABLE_STATS(OWNNAME=>'AUDSYS',TABNAME=>'AUD$UNIFIED',degree=>4);
Finally ran a 10046 trace and it showed the following for the Disk file operations I/O wait.
WAIT #140628544496888: nam='Disk file operations I/O' ela= 2352 FileOperation=8 fileno=1 filetype=36 obj#=-1 tim=5526698301528
It was starge objection id is -1 but stranger was there was no filetype 36 listed.
select file_no,filetype_id,FILETYPE_NAME from v$iostat_file where file_no=1 and filetype_id=36;

no rows selected 


At this stage looked at the historical data for the disk file operation I/O wait event on the AWR. It showed the first high waits occured soon after the swtichover to standby (circled in black). Even after switching back to original primary the wait events continued (circled in blue). Prior the switchover there was very low wait events and wait times on the primary.
When unified auditing is enabled on a standby it creates audit files on the local disk in the $ORACLE_BASE/audit/$ORACLE_SID directory. Based on above pattern a hypothesis was devised that unified audit trail could be getting populated from records in the audit files on the local disk that were craeted when it was a standby.
This hypothesis was tested out by archiving and removing the audit records created in the $ORACLE_BASE/audit/$ORACLE_SID. Afterwards querying the unified audit trail did not result in high disk file operation I/O.
Comparing the audit record count before the audit files on the local disks were removed
select count(*) from unified_audit_trail;

COUNT(*)
----------
761793 
and after
select count(*) from unified_audit_trail;

COUNT(*)
----------
516385
shows that some records from audit files on the local disk were used to populat the unified audit trail view.

Thursday, May 21, 2020

DBCA Reports ORA-46385 When Oracle Home Has Unified Auditing Enabled

Running dbca in silent mode reports ORA-46385 if the Oracle home has unified auditing is enabled. Below is the output on the shell prompt (only relevant section of the output is shown). This was on 19c.
Creating data dictionary views
33% complete
38% complete
39% complete
40% complete
[WARNING] ORA-46385: DML and DDL operations are not allowed on table

41% complete
46% complete
49% complete
52% complete
57% complete
Oracle Text
Trace files shows the same.
[Thread-242] [ 2020-02-06 12:23:53.087 UTC ] [BasicStep.handleNonIgnorableError:541]  oracle.sysman.assistants.util.SilentMessageHandler@3a6532cd:messageHandler
[Thread-242] [ 2020-02-06 12:23:53.087 UTC ] [BasicStep.handleNonIgnorableError:542]  ORA-46385: DML and DDL operations are not allowed on table
:msg
WARNING: Feb 06, 2020 12:23:53 PM oracle.assistants.common.base.util.AssistantAdvisor logMessage
WARNING: [ 2020-02-06 12:23:53.088 UTC ] [WARNING] ORA-46385: DML and DDL operations are not allowed on table


The dbca can run till the end despite the error. However, if a clean error free installation is preferred then turn off the unified auditing on the Oracle home before running dbca. There's no error in this case (output shown around same % mark where error happened before). Enable unified auditing once database is created.
Creating data dictionary views
33% complete
38% complete
39% complete
40% complete
41% complete
46% complete
49% complete
52% complete
57% complete
Oracle Text

Wednesday, December 4, 2013

Excessive Audit File (*.aud) Generation

Excessive amount of audit file generation was observed in a 11gR2 RAC environment. The audit trail parameter was set to OS and adump directory was getting populated with *.aud files at a rate of 60 per second. Audit files were cleaned out using cron (1298957.1) and schedule frequency wasn't enough to reduce the amount of *.aud file to an acceptable level. With 11gR2 (also available in 10.2.0.5 and 11.1.0.7) it is possible to use DBMS_AUDIT_MGMT to purge audit records including OS audit files (731908.1).
Starting with 11gR1 audit file generation has changed to create a new audit file for new session instead of appending to an existing audit file (1474823.1). This would generate lot of small (in size) audit files and could see burst of audit file generation for 3 tier application when connection pools are initiated. Also by default audit is enable of several actions. These could be found with the scripts on 1019552.6 or 287436.1. Below is the output from running script (1019552.6) on a new 11.2.0.4 RAC DB (only audit trail is changed to OS).
SQL> @tstaudit
Press return to see the audit related parameters...

NAME                 DISPLAY_VALUE
-------------------- --------------------
audit_file_dest      /opt/app/oracle/admin/std11g2/adump

audit_sys_operations FALSE
audit_syslog_level
audit_trail          OS

System auditing options across the system and by user

User name    Proxy name   Audit Option                   SUCCESS    FAILURE
------------ ------------ ------------------------------ ---------- ----------
                          ALTER ANY PROCEDURE            BY ACCESS  BY ACCESS
                          ALTER ANY TABLE                BY ACCESS  BY ACCESS
                          ALTER DATABASE                 BY ACCESS  BY ACCESS
                          ALTER PROFILE                  BY ACCESS  BY ACCESS
                          ALTER SYSTEM                   BY ACCESS  BY ACCESS
                          ALTER USER                     BY ACCESS  BY ACCESS
                          CREATE ANY JOB                 BY ACCESS  BY ACCESS
                          CREATE ANY LIBRARY             BY ACCESS  BY ACCESS
                          CREATE ANY PROCEDURE           BY ACCESS  BY ACCESS
                          CREATE ANY TABLE               BY ACCESS  BY ACCESS
                          CREATE EXTERNAL JOB            BY ACCESS  BY ACCESS
                          CREATE PUBLIC DATABASE LINK    BY ACCESS  BY ACCESS
                          CREATE SESSION                 BY ACCESS  BY ACCESS
                          CREATE USER                    BY ACCESS  BY ACCESS
                          DATABASE LINK                  BY ACCESS  BY ACCESS
                          DIRECTORY                      BY ACCESS  BY ACCESS
                          DROP ANY PROCEDURE             BY ACCESS  BY ACCESS
                          DROP ANY TABLE                 BY ACCESS  BY ACCESS
                          DROP PROFILE                   BY ACCESS  BY ACCESS
                          DROP USER                      BY ACCESS  BY ACCESS
                          EXEMPT ACCESS POLICY           BY ACCESS  BY ACCESS
                          GRANT ANY OBJECT PRIVILEGE     BY ACCESS  BY ACCESS
                          GRANT ANY PRIVILEGE            BY ACCESS  BY ACCESS
                          GRANT ANY ROLE                 BY ACCESS  BY ACCESS
                          PROFILE                        BY ACCESS  BY ACCESS
                          PUBLIC SYNONYM                 BY ACCESS  BY ACCESS
                          ROLE                           BY ACCESS  BY ACCESS
                          SYSTEM AUDIT                   BY ACCESS  BY ACCESS
                          SYSTEM GRANT                   BY ACCESS  BY ACCESS

29 rows selected.

Press return to see auditing options on all objects...

no rows selected
Press return to see audit trail... Note that the query returns the audit data for the last day only

no rows selected
Press return to see system privileges audited across the system and by user...
As seen from above output one of the audit option is "CREATE SESSION". As changing audit trail requires a restart of the database (RAC allows rolling restarts) it was decided to remove the audit on create session. This reduced the amount of audit file generated but still could see burst of audit files being generated every 5 second. Having examined the audit file it was now clear no audit files were generated for non sys users after removing audit on create session. Only audit files now generated are for sys users and only had following content on them
Thu Nov 28 14:32:12 2013 +01:00
LENGTH : '155'
ACTION :[7] 'CONNECT'
DATABASE USER:[1] '/'
PRIVILEGE :[6] 'SYSDBA'
CLIENT USER:[6] 'oracle'
CLIENT TERMINAL:[0] ''
STATUS:[1] '0'
DBID:[10] '2765679112'



Audit for various sys user operations are created by default irrespective of the configuration setting (308066.1). Audit files could also be generated by grid control agent (1196323.1) which wasn't applicable in this case as there was no grid control or agent running. Another possible cause is due to the resource health check by the grid infrastructure (1378881.1). Resource check interval was set to 1 second and if this had been the cause the audit file generation frequency would have been every 1 second instead of every 5 second. So this wasn't the cause either. Metalink note 1171314.1 list possible causes for excessive audit file generation and classify them as expected behavior, due to audit setting or due to bug but it still didn't provide any help in diagnosing the cause of this audit file generation.
There was no application or cron jobs that were executed on the OS as sys user. So it was decided to check where the sys connections originating from.
SQL>  select inst_id,sid,serial#,program,machine,USERNAME ,sql_id from gv$session where username='SYS' order by 1;

   INST_ID        SID    SERIAL# PROGRAM                               MACHINE      USERN   SQL_ID
---------- ---------- ---------- -----------------------------------   ---------    ------- -------------
         1        569          5 oraagent.bin@dm-db1.hps (TNS V1-V3)   dm-db1.hps   SYS
         1       2566       6317 oracle@dm-db1.hps (O001)              dm-db1.hps   SYS
         1       3683        561 oraagent.bin@dm-db1.hps (TNS V1-V3)   dm-db1.hps   SYS    4qm8a3w6a1rfd
         1       3967          1 oraagent.bin@dm-db1.hps (TNS V1-V3)   dm-db1.hps   SYS
         2        569          3 oraagent.bin@dm-db2.hps (TNS V1-V3)   dm-db2.hps   SYS
         2       3682          1 oraagent.bin@dm-db2.hps (TNS V1-V3)   dm-db2.hps   SYS

6 rows selected.
Sys sessions with program oraagent.bin@hostname are expected in RAC environment (except for bug in 11.2.0.2, refer 1307139.1) and does not cause excessive audit file generation. Only thing out of place was O001 process. Looking at the logon time for this session it showed been logon while back and similar systems (11.2.0.3 RAC) didn't have this process logon for long periods as in this case. From the Oracle reference document states "Onnn slave processes are spawned on demand. These processes communicate with the ASM instance. Maintains a connection to the ASM instance for metadata operations". According to 1556564.1 Onnn processes are spawned and terminated automatically and could be killed of if required and system will re-spawn it when needed and killing it will not effect database operations.
However it was decided to do a cluster stack restart on the node 1 during system maintenance. After the restart the Onnn process was not there and audit file generation went back to normal, burst of audit file generation every 5 second wasn't there anymore.

Useful metalink note
How to Setup Auditing [ID 1020945.6]
11g: Possible reasons for many OS audit trail (.aud) files, <1KB in size [ID 1474823.1]
Large Number of Audit Files Generated by Oracle Restart or Grid Infrastructure [ID 1378881.1]
Large Number Of Audit Files Generated During Rman Backups [ID 1211396.1]
AUDIT_SYS_OPERATIONS Set To FALSE Yet Audit Files Are Generated [ID 308066.1]
Huge/Large/Excessive Number Of Audit Records Are Being Generated In The Database [ID 1171314.1]
SCRIPT: Generate AUDIT and NOAUDIT Statements for Current Audit Settings [ID 287436.1]
How does the NOAUDIT option work (Disabling AUDIT commands)[ID 1068714.6]
A Lot of Audit Files in ASM Home [ID 813416.1]
Many OS Audit Files Produced By The Grid Control Agent Connections [ID 1196323.1]
New Feature DBMS_AUDIT_MGMT To Manage And Purge Audit Information [ID 731908.1]
Manage Audit File Directory Growth with cron [ID 1298957.1]
Script to Show Audit Options/Audit Trail [ID 1019552.6]
AUDIT_TRAIL Set to DB yet Some Audited Entries for non-Sysdba Users Are Created in the OS Trail. [ID 1279934.1]
The Column DBA_PRIV_AUDIT_OPTS Has Rows With USER_NAME 'ANY CLIENT' and PROXY_NAME NULL [ID 455565.1]
High Load On Server from Process Ora_onnn [ID 1556564.1]