Thursday, September 5, 2013

Change in Shared Memory Segment Creation on 12c

It appears that with 12c there's a change in the way shared memory segments are allocated when using automatic shared memory management (ASMM not AMM). Metalink note 15566.1 describe when shmmax parameter is large enough to contain the whole of SGA then a single shared memory segment will be allocated. However this behavior was changed on 11.2.0.3 and when a SR was raised and it was confirmed this was the intended behavior going forward from 11.2.0.3 with an explanation on 1399908.1.
11.2.0.3 created 3 shard memory segment irrespective of the size of the shmmax (even if whole of sga could fit within it). Now on 12c it seems there are some more changes and deviation from the behavior seen on 11.2.0.3. Instead of 3 segment it only creates 2 segments. This behavior is seen for both CDB and non-CDB and nattach on the smaller segment is twice the size of the large segment.
Current memory parameters in the database using ASMM (not AMM)
show parameter sga
sga_max_size big integer 1360M
sga_target big integer 1360M

show parameter memory
memory_max_target big integer 0
memory_target big integer 0
Limits for the share memory segments in the kerenl
ipcs -lm

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 67108864
max total shared memory (kbytes) = 17179869184
min seg size (bytes) = 1
Shmmax is 64GB. So creating an shared memory segment of 1360MB is not an issue and no need to break it up to multiple segments.
Created segments for the datbase
ipcs -m

------ Shared Memory Segments --------
key        shmid    owner  perms bytes      nattch status
0x7b917ec8 12189696 grid   640   4096       0
0x00000000 12222465 grid   640   4096       0
0x11380e5c 23756802 oracle 640   18874368   146
0x00000000 23789571 oracle 640   1409286144 73
Only two segments (owned by Oracle) and nattch on the smaller segment is twice that of the larger segment. Nattach value on the larger segments represents the total number of sessions in the database. If a new session is created then nattach value for both segments reflect the change in the session count. On 11.2.0.3 all segments had the same value for nattch.
To link the shared memory seen to the database use sysresv
$ORACLE_HOME/bin/sysresv

IPC Resources for ORACLE_SID "cdb12c1" :
Shared Memory:
ID KEY
23756802 0x11380e5c
23789571 0x00000000
Semaphores:
ID KEY
294915 0xc6ed03c4
Oracle Instance alive for sid "cdb12c1"
Looking at the size of the shared memory segments created shmid 23789571 is the largest segment which is 1409286144 bytes (1344 MB) is not same size as the SGA. The small segment with shmid 23756802 is 18874368 bytes (18MB) and together both segments are 1362 MB (which is > 1360MB SGA size).
If it's assumed that both segments together make up the SGA then it opens up the question why the smaller segment shows twice the number of process attached to it than the larger segment. So far nothing on metalink about this. Awaiting for answers from Oracle...


Tuesday, September 3, 2013

ORA-1691: unable to extend lobsegment is expected behavior?

ORA-1691 happens when a lob segment cannot extend itself to accommodate the growth due to new data being inserted. Usually remedies are extending the data file and adding a data file. However there's a special situation where this is the expected behavior. ORA-1691 is reported on alert log (only on alert log no client side error) when space taken by the lob segment is allocated for reuse, before the undo retention period is expired.
From the Admin Guide "Automatic tuning of undo retention is not supported for LOBs. This is because undo information for LOBs is stored in the segment itself and not in the undo tablespace. For LOBs, the database attempts to honor the minimum undo retention period specified by UNDO_RETENTION. However, if space becomes low, unexpired LOB undo information may be overwritten."
From Database Concept "The database manages read consistency for LOB segments differently from other data. Instead of using undo data to record changes, the database stores the before images in the segment itself. When a transaction changes a LOB, the database allocates a new chunk and leaves the old data in place. If the transaction rolls back, then the database rolls back the changes to the index, which points to the old chunk."
Below is the test case that illustrate the reporting of ora-1691 on the alert log. The test case create the LOB as BasicFile. But the observed behavior is same for SecureFile as well. The test was done on 11.2.0.3.7 and 11.1.0.16 on Linux.
1. Create two tablespaces one for data and one for lob segments
SQL> create tablespace datatbs datafile '+data(datafile)' size 5m autoextend on next 1m maxsize 10m;
SQL>  create tablespace lobtbs datafile '+data(datafile)' size 5m autoextend on next 1m maxsize 10m;
2. Create the table with a LOB column and make sure data segments and lob segments are created in the correct tablespaces
CREATE TABLE lobtest ( ID number,  "OBJECT" BLOB ) SEGMENT CREATION IMMEDIATE TABLESPACE datatbs
  LOB ( "OBJECT" )
  STORE AS object_lob_seg ( TABLESPACE lobtbs
        DISABLE STORAGE IN ROW
        CHUNK 8K
        CACHE
        INDEX object_lob_idx (TABLESPACE lobtbs)
    );
3. The java code given at the end of the post is used to populate the table. The java code will insert a integer and a 50K BLOB.
java LobReuse 192.168.0.66 1521 ent11g2 asanga asa 141
Here (192.168.0.66 1521 ent11g2) is the (DB Server name or IP, listener port, SID) and next two parameters (asanga asa) is (username, password) and last parameter 141 is number of inserts. Verify 141 rows are inserted
SQL> select count(*) from lobtest;

  COUNT(*)
----------
       141
141 is the maximum number of inserts that could be made before the 10m maximum size is reached on the LOBTBS tablespace. Trying to insert any more rows would result in ora-1691 on client side which is the expected behavior (this is not the reason for the post).
java LobReuse 192.168.0.66 1521 ent11g2 asanga asa 1

java.sql.SQLException: ORA-01691: unable to extend lob segment ASANGA.OBJECT_LOB_SEG by 128 in tablespace LOBTBS

        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
        at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:879)
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:450)
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
and following lines will be reported on the alert log
ORA-1691: unable to extend lobsegment ASANGA.OBJECT_LOB_SEG by 128 in tablespace              LOBTBS
ORA-1691: unable to extend lobsegment ASANGA.OBJECT_LOB_SEG by 128 in tablespace              LOBTBS
ORA-1691: unable to extend lobsegment ASANGA.OBJECT_LOB_SEG by 128 in tablespace              LOBTBS
4. Delete all the inserted rows and verify no rows are in the table.
SQL> delete from lobtest;
141 rows deleted.

SQL> commit;
Commit complete.

SQL> select count(*) from lobtest;

  COUNT(*)
----------
         0
Deleting rows will not reduce the segment size (unless shrink space other space reclaimable mechanism are used to reduce the size). Once space is allocated to a segment it will remain allocated and reused if become free due to row deletion. At this stage the segment is occupying the full tablespace and there's no place to grow and for new inserts the log segment space must be reused.
5. Run the java code to insert another 141 rows as before.
java LobReuse 192.168.0.66 1521 ent11g2 asanga asa 141
This will execute without any errors and at the end of the execution 141 rows would have been inserted to the table

SQL> select count(*) from lobtest;

  COUNT(*)
----------
       141
However if alert log was monitored during this time following could be seen
ORA-1691: unable to extend lobsegment ASANGA.OBJECT_LOB_SEG by 128 in tablespace              LOBTBS
ORA-1691: unable to extend lobsegment ASANGA.OBJECT_LOB_SEG by 128 in tablespace              LOBTBS
ORA-1691: unable to extend lobsegment ASANGA.OBJECT_LOB_SEG by 128 in tablespace              LOBTBS
ORA-1691: unable to extend lobsegment ASANGA.OBJECT_LOB_SEG by 128 in tablespace              LOBTBS
ORA-1691: unable to extend lobsegment ASANGA.OBJECT_LOB_SEG by 128 in tablespace              LOBTBS
ORA-1691: unable to extend lobsegment ASANGA.OBJECT_LOB_SEG by 128 in tablespace              LOBTBS
This is misleading as no error was shown on the client side and 141 rows got inserted successful. But there's no way to verify this looking only at the alert log which looks like the segment is unable to expand and no rows got inserted. Having the Event 44951 set seem to reduce the number of lines reporting ora-1691 on alert log but still it does get reported. (Event 44951 sets number of chunks to clean up). Beside the error it enq: HW - contention waits were also observed.



According to Oracle this is expected behavior and not a bug. For system where automated alert log monitoring is in place it would be difficult to distinguish if there's a space issue or not without looking into free space within the lobsegment.
One way to overcome this observed behavior is to change the undo related parameters of the LOB segment, which are pctversion and retention. Creating the table with pctversion 0 eliminate the reporting of ora-1691 on the alert log when log segment space is reused.
CREATE TABLE lobtest ( ID number,  "OBJECT" BLOB ) SEGMENT CREATION IMMEDIATE TABLESPACE datatbs
  LOB ( "OBJECT" )
  STORE AS object_lob_seg ( TABLESPACE lobtbs
        DISABLE STORAGE IN ROW
        CHUNK 8K
        CACHE
        PCTVERSION 0
        INDEX object_lob_idx (TABLESPACE lobtbs)
    );
However this may not suite every application and could lead to ora-1555 snapshot too old errors. Oracle documentation provide some guide lines on how to set pctversion value based on update/read patterns of the application.
If space is not reused within the the undo_retention then again the ora-1691 is not observed on the alert log. There's no explicit way to change the retention value on LOB segment. It's taken from the undo_retention parameter.

Related Post
ORA-1691: unable to extend lobsegment Message Only on Alert Log When Inserting to a Table with SecureFile

Java code used for LOB insert
import java.sql.Connection;
import oracle.jdbc.OraclePreparedStatement;
import oracle.jdbc.pool.OracleDataSource;

/**
 *
 * @author Asanga
 */
public class LobReuse {

    public static void main(String[] args) {

      try {

            OracleDataSource pool = new OracleDataSource();
            pool.setURL("jdbc:oracle:thin:@"+args[0]+":"+args[1]+":"+args[2]);
            pool.setUser(args[3]);
            pool.setPassword(args[4]);

            int inserts = Integer.parseInt(args[5]);
            for(int i = 0 ; i < inserts ; i++){
            Connection con = pool.getConnection();

            con.setAutoCommit(false);
            byte[] x = new byte[50 * 1024];
            x[1]=10; x[40 * 1024] = 20;

            OraclePreparedStatement pr = (OraclePreparedStatement) con.prepareStatement("insert into lobtest values(?,?)");
            pr.setInt(1, i );
            pr.setBytes(2, x);
            pr.execute();
            con.commit();
            pr.close();
            con.close();
            }

        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }}

Saturday, August 24, 2013

Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Creating CDB & PDB

After installing clusterware and database software remaining step is to create the database. In this post a container database (CDB) and a pluggable database (PDB) will be created using DBCA. CDB and PDB are part of 12c new multitenant feature.
1. Before starting DBCA change the permission on cfgtoollogs and admin folders. cfgtoolslogs permission was an issue in 11.2 when installed with role separation as well(see step 15). In 12c admin directory in ORACLE_BASE also causes issue as it doesn't have write permission for the oinstall group thus making oracle user unable to create necessary directories inside it.

To fix it change to ORACLE_BASE as grid user (or root) and execute the following on all nodes
cd $ORACLE_BASE
chmod 770 cfgtoollogs
chmod 770 admin
2. Verify the database installation pre-req check with cluvfy -pre dbcfg
[oracle@rhel12c1 ~]$ cluvfy stage -pre dbcfg -n rhel12c1,rhel12c2 -d /opt/app/oracle/product/12.1.0/dbhome_1

Performing pre-checks for database configuration

Checking node reachability...
Node reachability check passed from node "rhel12c1"


Checking user equivalence...
User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Check: Node connectivity using interfaces on subnet "192.168.0.0"
Node connectivity passed for subnet "192.168.0.0" with node(s) rhel12c2,rhel12c1
TCP connectivity check passed for subnet "192.168.0.0"


Check: Node connectivity using interfaces on subnet "192.168.1.0"
Node connectivity passed for subnet "192.168.1.0" with node(s) rhel12c2,rhel12c1
TCP connectivity check passed for subnet "192.168.1.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.

Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.
Group existence check passed for "oper"
Group existence check passed for "dba"
Group existence check passed for "backupdba"
Group existence check passed for "dgdba"
Group existence check passed for "kmdba"
Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "rhel12c2:/tmp"
Free disk space check passed for "rhel12c1:/tmp"
Check for multiple users with UID value 500 passed
User existence check passed for "oracle"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Membership check for user "oracle" in group "oinstall" [as Primary] passed
Membership check for user "oracle" in group "dba" passed
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Package existence check passed for "binutils"
Package existence check passed for "compat-libcap1"
Package existence check passed for "compat-libstdc++-33(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check passed for "gcc"
Package existence check passed for "gcc-c++"
Package existence check passed for "ksh"
Package existence check passed for "make"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "glibc-devel(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check passed for "libaio-devel(x86_64)"
Check for multiple users with UID value 0 passed
Current group ID check passed

Starting check for consistency of primary group of root user

Check for consistency of root user's primary group passed

Checking CRS integrity...

Clusterware version consistency passed.

CRS integrity check passed

Checking node application existence...

Checking existence of VIP node application (required)
VIP node application check passed

Checking existence of NETWORK node application (required)
NETWORK node application check passed

Checking existence of ONS node application (optional)
ONS node application check passed

Time zone consistency check passed

Pre-check for database configuration was successful.



3.Start DBCA and create the CDB first.
It was noted that the time to create database using custom option without the use of OLTP and warehouse templates takes longer compared to 11.2
It is possible to create both CDB and PDB in one step but here only the CDB is created initially.
With 12c database vault and label security could be configured with DBCA at the time of database creation.

4. PDBs could be created with DBCA as well as using command "create pluggable database". Time it takes to provision a PDB is much shorter compared to creating a database using DBCA in previous releases. This is one of the reasons why oracle is "marketing/advertising/touting" that 12c is the database for the cloud.
Start DBCA to create the PDB.
Each PDB needs a CDB and earlier created CDB is specified as the CDB for this PDB being created.

SRVCTL could be used to manage PDBs in a RAC environment. Dynamic services must be created for each PDBs. In this for the admin managed CDB and PDBs service is created with following command
srvctl add service -d cdb12c -pdb pdb12c -s pdbsvc -preferred "cdb12c1,cdb12c2"

srvctl config service -d cdb12c -s pdbsvc
Service name: pdbsvc
Service is enabled
Server pool: cdb12c_pdbsvc
Cardinality: 2
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type:
Failover method:
TAF failover retries:
TAF failover delay:
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name: pdb12c
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Session State Consistency:
Preferred instances: cdb12c1,cdb12c2
Available instances:
Starting the service will open the PDBs in read,write mode.
SQL>  select inst_id,name,open_mode from gv$pdbs;

   INST_ID NAME                           OPEN_MODE
---------- ------------------------------ ----------
         2 PDB12C                         MOUNTED
         1 PDB12C                         MOUNTED

srvctl start service -d cdb12c -s pdbsvc

SQL> select inst_id,name,open_mode from gv$pdbs;

   INST_ID NAME                           OPEN_MODE
---------- ------------------------------ ----------
         2 PDB12C                         READ WRITE
         1 PDB12C                         READ WRITE
on alert log following could be seen
Mon Oct 14 14:14:09 2013
ALTER PLUGGABLE DATABASE pdb12c OPEN  /* svc agent *//* {1:23451:950} */
Mon Oct 14 14:14:09 2013
This instance was first to open pluggable database PDB12C (container=3)
Opening pdb PDB12C (3) with no Resource Manager plan active
Pluggable database PDB12C opened read write
Completed: ALTER PLUGGABLE DATABASE pdb12c OPEN  /* svc agent *//* {1:23451:950} */
Modify the service's auto start status and PDBs will be open in read write mode each time the service is started
# crsctl modify resource ora.cdb12c.pdbsvc.svc -attr "AUTO_START=always"
However stopping the service using srvctl has no effect on the open mode of the PDB. It will stop the service and prevent any new connection coming in but the PDB will remain in read write mode. Even though on Oracle documentation it says stopping the service will stop the PDB.
This text could lead to confusion if one would expect the PDB will be closed when it is stopped. The real behavior of the stop is mentioned in another document which reads (from Admin Guide)"When your database is being managed by Oracle Restart or Oracle Clusterware, and you use the SRVCTL utility to start a service with a PDB property for a PDB that is closed, the PDB is opened in read/write mode on the nodes where the service is started. However, stopping a PDB service does not change the open mode of the PDB."
To stop the PDBs in RAC uses alter pluggable database command with "instances=all".

Creation of the PDB concludes the installing of 12c RAC.

Related Posts
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)