Sunday, November 22, 2015

Relink Fails due to Requirement Checks Failure

Oracle recommends re-linking oracle binaries after OS upgrade. There are few earlier posts on this topic. In this case the server in question was a Amazon EC2 server and didn't have any swap configured.
$ cat /proc/meminfo | grep "Swap"
SwapCached:          0 kB
SwapTotal:           0 kB
SwapFree:            0 kB
After the OS upgrade , relinking of oracle binaries failed due to insufficient swap space.
$ ./relink all
writing relink log to: /opt/app/oracle/product/testbox/11.2.0/install/relink.log

$ more ../install/relink.log
Starting Oracle Universal Installer...

Checking swap space: 0 MB available, 500 MB required.    Failed <<<<

Some requirement checks failed. You must fulfill these requirements before

continuing with the installation,

Exiting Oracle Universal Installer, log for this session can be found at /opt/app/oracle/oraInventory/logs/installActions2015-09-25_12-11-54PM.log
Only parameters that could be passed on to relink is all and as_installed. However looking inside relink script it could be seen that it is using runInstaller.
#-----------------------------------
# location of runInstaller executable
RUNINSTALLER=$ORACLE_HOME/oui/bin/runInstaller

#--------------------------
Runinstaller command gets set of arguments.
#-----------------------------------
# full argument list for runInstaller
#
ARGS="-relink -waitForCompletion -maketargetsxml $MAKEORDER $LOGDIR_ARG ORACLE_HOME=$ORACLE_HOME"

#--------------------------
Adding -ignoreSysPrereqs as the first argument resolved the issue.
ARGS="-ignoreSysPrereqs -relink -waitForCompletion -maketargetsxml $MAKEORDER $LOGDIR_ARG ORACLE_HOME=$ORACLE_HOME"
Relink completes successfully afterwards and relink.log will end with the following for oracle home relinks
test ! -f /opt/app/oracle/product/testbox/11.2.0/bin/oracle ||\
           mv -f /opt/app/oracle/product/testbox/11.2.0/bin/oracle /opt/app/oracle/product/testbox/11.2.0/bin/oracleO
mv /opt/app/oracle/product/testbox/11.2.0/rdbms/lib/oracle /opt/app/oracle/product/testbox/11.2.0/bin/oracle
chmod 6751 /opt/app/oracle/product/testbox/11.2.0/bin/oracle
and with the following for GI home relinking
test ! -f /grid/oracle/product/11.2.0/bin/oracle ||\
           mv -f /grid/oracle/product/11.2.0/bin/oracle /grid/oracle/product/11.2.0/bin/oracleO
mv /grid/oracle/product/11.2.0/rdbms/lib/oracle /grid/oracle/product/11.2.0/bin/oracle
chmod 6751 /grid/oracle/product/11.2.0/bin/oracle
Once the relinking is completed remove the ignoreSysPrereqs parameter from the relink file.
If the server already has swap created and want to recreate the same scenario as above for testing, use
swapoff -a
swapon -a
to disable and enable swapping
Also if the environment uses role separation then after the relink it may be necessary to set the correct permission on the oracle binary using setasmgidwrap.

Related Posts
Upgrading RHEL 6 OS in a 11gR2 RAC Environment
Upgrading OS in 11gR2 RAC Environment (RHEL 5)
Upgrading ASMLib and OS in 11gR1 RAC Environment


Sunday, November 15, 2015

12c Encryption

This post gives a highlight of using TDE in 12c. For detail information refer advance security guide. In 12c the orapki/alter system commands related to key management has been replaced with ADMINISTER KEY MANAGEMENT commands.
The first set of steps shows setting a non-CDB for use of TDE. 1. Create a location for wallet files (key store location).
mkdir -p /opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde
chmod 700 /opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde
2. Add ENCRYPTION_WALLET_LOCATION to sqlnet.ora file
ENCRYPTION_WALLET_LOCATION =
  (SOURCE =
   (METHOD = FILE)
    (METHOD_DATA =
     (DIRECTORY = /opt/app/oracle/product/12.1.0/dbhome_2/network/admin/tde)
    )
  )
3. Create the software key store (wallet) by specifying key store location and password or key store (asanga123 in this case)
SQL> ADMINISTER KEY MANAGEMENT CREATE KEYSTORE '/opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde' identified by asanga123;
Result of this is a key store file (wallet file).
SQL> ! ls
ewallet.p12
At this stage the status of the key store would be closed
SQL> select WRL_TYPE,STATUS,WALLET_TYPE,WALLET_ORDER,FULLY_BACKED_UP,CON_ID,WRL_PARAMETER from v$ENCRYPTION_WALLET;

WRL_TYPE   STATUS   WALLET_TYPE   WALLET_OR FULLY_BAC     CON_ID WRL_PARAMETER
---------- -------  ------------- --------- --------- ---------- -----------------------------------------------------------
FILE       CLOSED   UNKNOWN       SINGLE    UNDEFINED          0 /opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde/ 
4. Open the key store by providing the key store password
SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY asanga123;
keystore altered.

SQL> select WRL_TYPE,STATUS,WALLET_TYPE,WALLET_ORDER,FULLY_BACKED_UP,CON_ID,WRL_PARAMETER from v$ENCRYPTION_WALLET;

WRL_TYPE   STATUS             WALLET_TYPE   WALLET_OR FULLY_BAC     CON_ID WRL_PARAMETER
---------- ------------------ ------------- --------- --------- ---------- -----------------------------------------------------------
FILE       OPEN_NO_MASTER_KEY PASSWORD      SINGLE    NO                 0 /opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde/ 
5. Usually at this stage auto login key store is also created. But this lead to an issue (refer 1944507.1).
SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY asanga123 WITH BACKUP;
ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY asanga123 WITH BACKUP
*
ERROR at line 1:
ORA-28417: password-based keystore is not open

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE open identified by asanga123;
ADMINISTER KEY MANAGEMENT SET KEYSTORE open identified by asanga123
*
ERROR at line 1:
ORA-28354: Encryption wallet, auto login wallet, or HSM is already open
Therefore master encryption key is created before creating auto login wallet.
SQL>  ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY asanga123 with backup ;
keystore altered.
This will change wallet status from open_no_master_key to open.
SQL> select WRL_TYPE,STATUS,WALLET_TYPE,WALLET_ORDER,FULLY_BACKED_UP,CON_ID,WRL_PARAMETER from v$ENCRYPTION_WALLET;

WRL_TYPE   STATUS  WALLET_TYPE   WALLET_OR FULLY_BAC     CON_ID WRL_PARAMETER
---------- ------- ------------- --------- --------- ---------- -----------------------------------------------------------
FILE       OPEN    PASSWORD      SINGLE    NO                 0 /opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde/
On the database alert log following could be observed
Creating new database key for new master key and wallet
Creating new database key with the new master key
Retiring: ena 2 flag 6 mkloc 0
   encrypted key 8dd09c987ef966198af992379477f13900000000000000000000000000000000
   mkid b81a02de82664fbcbf2c9bcdcec4a3ae
Creating: ena 2 flag e mkloc 1
   encrypted key 92c72aeada0197dda6da3e4d64ac875c00000000000000000000000000000000
   mkid ac6f7e2c97ff4fdfbf9d900a653e2c21
Switching out all online logs for the new master key
6. Finally create the auto login key store so that key store is auto opened when the database starts
SQL> ADMINISTER KEY MANAGEMENT CREATE local AUTO_LOGIN KEYSTORE FROM KEYSTORE '/opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde' identified by asanga123;
keystore altered.
This will create the cwallet.sso file and wallet type will be changed to local_autologin
SQL> ! ls
cwallet.sso  ewallet_2015101517371593.p12  ewallet.p12  ewallet.p12.bak

select WRL_TYPE,STATUS,WALLET_TYPE,WALLET_ORDER,FULLY_BACKED_UP,CON_ID,WRL_PARAMETER from v$ENCRYPTION_WALLET;

WRL_TYPE   STATUS WALLET_TYPE       WALLET_OR FULLY_BAC     CON_ID WRL_PARAMETER
---------- ------ ----------------- --------- --------- ---------- -----------------------------------------------------------
FILE       OPEN   LOCAL_AUTOLOGIN   SINGLE    NO                 0 /opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde/ 
Encryption could be enabled on CDB with PDB same way as above. Below is an example of enabling TDE for CDB with two PDBs.
SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDBONE                         READ WRITE NO
         4 PDBTWO                         READ WRITE NO
Create key store, open and set master encryption key.
SQL> ADMINISTER KEY MANAGEMENT CREATE KEYSTORE '/opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde' identified by asanga123;
keystore altered.

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY asanga123 container=all;
keystore altered.

SQL>  ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY asanga123 with backup container=all;
keystore altered.

SQL> select WRL_TYPE,STATUS,WALLET_TYPE,WALLET_ORDER,FULLY_BACKED_UP,CON_ID,WRL_PARAMETER from v$ENCRYPTION_WALLET;

WRL_TYPE   STATUS  WALLET_TYPE   WALLET_OR FULLY_BAC     CON_ID WRL_PARAMETER
---------- ------- ------------- --------- --------- ---------- -----------------------------------------------------------
FILE       OPEN    PASSWORD      SINGLE    NO                 0 /opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde/
Enable auto login
SQL> ADMINISTER KEY MANAGEMENT CREATE local AUTO_LOGIN KEYSTORE FROM KEYSTORE '/opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde' identified by asanga123;
keystore altered.
Login to a PDB and check the wallet status
SQL> show con_name

CON_NAME
-----------
PDBTWO

select WRL_TYPE,STATUS,WALLET_TYPE,WALLET_ORDER,FULLY_BACKED_UP,CON_ID,WRL_PARAMETER from v$ENCRYPTION_WALLET;

WRL_TYPE   STATUS WALLET_TYPE       WALLET_OR FULLY_BAC     CON_ID WRL_PARAMETER
---------- ------ ----------------- --------- --------- ---------- -----------------------------------------------------------
FILE       OPEN   LOCAL_AUTOLOGIN   SINGLE    NO                 0 /opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde/ 
v$encryption_keys view could be used to identify each containers key ids even though container id remains 0 for all.
SQL> select CON_ID,KEY_ID,KEYSTORE_TYPE,CREATOR_DBNAME,CREATOR_PDBNAME from v$encryption_keys;

    CON_ID KEY_ID                                                 KEYSTORE_TYPE     CREATOR_DBNAME  CREATOR_PDBNAME
---------- ------------------------------------------------------ ----------------- --------------- ------------------------------
         0 AQugdpFHIk9yv1tQiZj0EhUAAAAAAAAAAAAAAAAAAAAAAAAAAAAA   SOFTWARE KEYSTORE PDBENC          CDB$ROOT
         0 Af8ZFXnsWk+/v9Z4RFalEHwAAAAAAAAAAAAAAAAAAAAAAAAAAAAA   SOFTWARE KEYSTORE PDBENC          PDBONE
         0 AU3e99wEOk8lv4QEBp4Ow3AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA   SOFTWARE KEYSTORE PDBENC          PDBTWO
Following test could be used to verify TDE is working. In this case a tablespace is created with encryption enabled. Few rows are inserted to tables created in those table spaces. At times buffer cache flushing or manual check points may be required to force database writer to write to data files
SQL>  create tablespace enctest datafile size 10m ENCRYPTION DEFAULT STORAGE(ENCRYPT);
Tablespace created.
SQL>  create tablespace nonenctest datafile size 10m ;
Tablespace created.

SQL>select TABLESPACE_NAME,ENCRYPTED from dba_tablespaces;

TABLESPACE_NAME                ENC
------------------------------ ---
ENCTEST                        YES
NONENCTEST                     NO

create table enctable (a varchar2(100)) tablespace ENCTEST;
create table nonecntbale (a varchar2(100)) tablespace NONENCTEST;

begin
  for i in 1 .. 100
  loop
  insert into enctable values ('top secret text');
  insert into nonecntbale values ('top secret text');
  end loop;
  commit;
  end;
/
alter system checkpoint;
Strings command is used to look into the data file content.First the data file belonging to non-encrypted tablespace
strings /opt/app/oracle/oradata/ENT12C1/datafile/o1_mf_nonencte_c1zsml08_.dbf
top secret text,
top secret text,
Secondly the data file belonging to the encrypted tablespace
strings /opt/app/oracle/oradata/ENT12C1/datafile/o1_mf_enctest_c1zsl0d9_.dbf
Vj#>{
O+l;2
c1ax|
        drl
fzc1
Lbqby%}
u;Fa
=B]Wv
~/th
9hHW
=Jc;
@s|J84
|3M*
2ATG
As seen above encrypted tablespace doesn't show it's content in clear text.



Encryption requires additional CPU. Next is a minor test that was done to compare the CPU usage for inserting to a encryption enabled tablespace vs non-encrypted tablespace.
create table NOencryptiontbl(a number, b varchar2(100)) tablespace NONENCTEST;
create table encryptiontbl(a number, b varchar2(100)) tablespace ENCTEST;
  
  begin
    for i in 1 .. 1000000
    loop
    --insert into encryptiontbl values (i,'asdfghjkllqwertyuiopzxcvbnm134567890'||i);
    insert into NOencryptiontbl values (i,'asdfghjkllqwertyuiopzxcvbnm134567890'||i);
    commit;
    end loop;
  end;
  /
The inserts were first done on table created on non-encrypted and then on table created on encryption enabled tablespace. Value of CPU used by this session statistic was used to compare the CPU usage of each session once the inserts have completed. Graph below shows the CPU used, with TDE enabled it took on average 51 CPU seconds more for inserts to complete.

If TDE is used in a data guard environment then the standby must have the key store (wallet) copied over from the primary. Without wallet open the recovery at standby will fail. In the below output datafile 17 was part of a encrypted tablespace created at primary.
Datafile #17: '/opt/app/oracle/oradata/ENT12C1S/datafile/o1_mf_t2_bv215f2m_.dbf'
kcrf_decrypt_redokey: wallet is not opened..(err 28365)
Errors with log /opt/app/oracle/fast_recovery_area/ENT12C1S/archivelog/2015_07_23/o1_mf_1_2726_bv214gg4_.arc
MRP0: Background Media Recovery terminated with error 28365
Thu Jul 23 16:13:49 2015
Errors in file /opt/app/oracle/diag/rdbms/ent12c1s/ent12c1s/trace/ent12c1s_pr00_2541.trc:
ORA-28365: wallet is not open
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovered data files to a consistent state at change 41627576
Thu Jul 23 16:13:50 2015
Errors in file /opt/app/oracle/diag/rdbms/ent12c1s/ent12c1s/trace/ent12c1s_pr00_2541.trc:
ORA-28365: wallet is not open
Thu Jul 23 16:13:51 2015
MRP0: Background Media Recovery process shutdown (ent12c1s)
Another thing to watch out for is exporting using exp as oppose to expdp. Tables residing in encrypted tablespaces cannot be exported using exp.
exp asanga/asa file=asa.dmp tables=p1,p2
Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
About to export specified tables via Conventional Path ...
EXP-00111: Table P1 resides in an Encrypted Tablespace T2 and will not be exported
EXP-00111: Table P2 resides in an Encrypted Tablespace T2 and will not be exported
Export terminated successfully with warnings.
Useful metalink notes
TDE Wallet Problem in 12c: Cannot do a Set Key operation when an auto-login wallet is present [ID 1944507.1]
Master Note For Transparent Data Encryption ( TDE ) [ID 1228046.1]
How to migrate a non pluggable database that uses TDE to pluggable database ? [ID 1678525.1]
Known TDE Wallet Issues [ID 1301365.1]
Auto Login Wallet Cannot Be Closed [ID 1204604.1]
Removing TDE, but V$ENCRYPTION_WALLET / GV$ENCRYPTION_WALLET still has rows. [ID 2003528.1]
Step by Step Troubleshooting Guide for TDE Error ORA-28374 [ID 1541818.1]
How To Import Encrypted Datapump Data from 11G To 12c DB [ID 1642059.1]

Related Post
19c Encryption

Sunday, November 8, 2015

Moving Grid Infrastructure Management Repository (GIMR) Database (MGMTDB) Out of Default Disk Group

With 12.1.0.2 creation of Grid Infrastructure Management Repository (GIMR) is mandatory. This results in a CDB called -MGMTDB being created at the end of the GI installation. This CDB (-MGMTDB) contains a single PDB which has the same name as the cluster name. If ASM is used for storing OCR/Vote then, when creating the cluster, by default this database is created on the same disk group where ocr/vote resides. As of 12.1.0.2 there's no way to specify a different disk group for GIMR alone. This post shows steps to moving the GIMR to a different disk group from the default disk group.
1. Create a new ASM disk group to store GIMR. In this case this new disk group is called GIMR and has external redundancy. The GIMR currently reside in a disk group called CLUSFS with normal redundancy. It's important that newly created disk group has compatible.asm and compatible.rdbms set to 12.1.
SQL> select name,type total_mb,free_mb,REQUIRED_MIRROR_FREE_MB,USABLE_FILE_MB from v$asm_diskgroup;

NAME                           TOTAL_    FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB
------------------------------ ------ ---------- ----------------------- --------------
CLUSFS                         NORMAL      17400                   10236           3582
GIMR                           EXTERN      10108                       0          10108

SQL> select name,os_mb,total_mb,free_mb from v$asm_disk where name is not null;

NAME                                OS_MB   TOTAL_MB    FREE_MB
------------------------------ ---------- ---------- ----------
GIMR_0000                           10236      10236      10108
CLUSFS_0002                         10236      10236       5804
CLUSFS_0000                         10236      10236       5812
CLUSFS_0001                         10236      10236       5800
2. Stop and disable ora.crf resource
# crsctl stop res ora.crf -init
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded

# crsctl modify res ora.crf -attr ENABLED=0 -init
3. Find the node GIMR database is running and run the delete database command from that node. This could drop the current GIMR databse.
$ srvctl status mgmtdb
Database is enabled
Instance -MGMTDB is running on node rhel12c1

[grid@rhel12c1 grid2]$ dbca -silent -deleteDatabase -sourceDB -MGMTDB
Connecting to database
4% complete
9% complete
14% complete
19% complete
23% complete
28% complete
47% complete
Updating network configuration files
48% complete
52% complete
Deleting instance and datafiles
76% complete
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb.log" for further details.
4. Create the GIMR with the name "-MGMTDB" and specifying new ASM diskgroup
$ dbca -silent -createDatabase -sid -MGMTDB -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc 
-gdbName _mgmtdb -storageType ASM -diskGroupName +GIMR -datafileJarLocation $ORACLE_HOME/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck
Registering database with Oracle Grid Infrastructure
5% complete
Copying database files
7% complete
9% complete
16% complete
23% complete
30% complete
37% complete
41% complete
Creating and starting Oracle instance
43% complete
48% complete
49% complete
50% complete
55% complete
60% complete
61% complete
64% complete
Completing Database Creation
68% complete
79% complete
89% complete
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb/_mgmtdb0.log" for further details.


5. Before creating the PDB connected with the GIMR find out the cluster name. There are many ways to find out the cluster name simplest is to use cemutlo.
$ cemutlo -n
rhel12c-cluster
6. Trying to create PDB with above cluster name will fail.
$ dbca -silent -createPluggableDatabase -sourceDB -MGMTDB -pdbName rhel12c-cluster -createPDBFrom RMANBACKUP -PDBBackUpfile $GI_HOME/assistants/dbca/templates/mgmtseed_pdb.dfb -PDBMetadataFile $GI_HOME/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true -internalSkipGIHomeCheck
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb0.log" for further details.

$ more /opt/app/oracle/cfgtoollogs/dbca/_mgmtdb0.log
The Container database selected is in the open state
rhel12c-cluster: PDB Name must be at least 1 character and at most 30 characters, 
should start with an alphabetical character and must include only alphabetical characters, numbers or the '_' character.
The reason is database names cannot contain - (dash) and cluster name cannot contain _ (underscore). "The cluster name is case-insensitive, must be unique across your enterprise, must be at least one character long and no more than 15 characters in length, must be alphanumeric, cannot begin with a numeral, and may contain hyphens (-). Underscore characters (_) are not allowed.".
So how did the GIMR got created in the first place? It seems if the cluster name contains (-) installer implicitly replace them with (_). This could be verified by looking in the $ORACLE_BASE/cfgtoollogs/dbca/_mgmtdb folder. This will contain a folder called rhel12c_cluster which has all the log files related to original PDB creation. Therefore if the cluster name contains (-) then replace it with (_) and run the PDB create command again.
$ dbca -silent -createPluggableDatabase -sourceDB -MGMTDB -pdbName rhel12c_cluster -createPDBFrom RMANBACKUP -PDBBackUpfile $GI_HOME/assistants/dbca/templates/mgmtseed_pdb.dfb -PDBMetadataFile $GI_HOME/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true -internalSkipGIHomeCheck
Creating Pluggable Database
4% complete
12% complete
21% complete
38% complete
55% complete
85% complete
Completing Pluggable Database Creation
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb/rhel12c_cluster/_mgmtdb0.log" for further details.
7. Find out on which node the -MGMTDB database is running and run mgmtca from that node to secure the GIMR. Running mgmtca doesn't produce any output.
$ srvctl status MGMTDB
Database is enabled
Instance -MGMTDB is running on node rhel12c1
[grid@rhel12c1 _mgmtdb]$ mgmtca
8. Finally enable and start the ora.crf resource.
# crsctl modify res ora.crf -attr ENABLED=1 -init
# crsctl start res ora.crf -init
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
Useful metalink notes
Managing the Cluster Health Monitor Repository [ID 1921105.1]
FAQ: 12c Grid Infrastructure Management Repository (GIMR) [ID 1568402.1]
How to Move GI Management Repository to Different Shared Storage (Diskgroup, CFS or NFS etc) [ID 1589394.1]
12.2: How to Create GI Management Repository [ID 2246123.1]
MDBUtil: GI Management Repository configuration tool [ID 2065175.1]
How to relocate CHM repository and increase retention time [ID 2062234.1]

Sunday, November 1, 2015

Convert Single Instance DB to RAC DB - Manual Method

There are many ways to convert a single database to RAC. Using database template, rconfig or the manual method. This post shows the steps for manual method of converting a single instance database to RAC (tested for both 11.2.0.4 and 12.1.0.2). The single database is running out of a non-rac oracle home. Backup of this would be restored using a rac enabled oracle home and then converted to a RAC database.
The database is called "asanga" and will retain the same name when converted to RAC.
1. Create a pfile of the single instance database. The pfile entries are shown below
 more pfile.ora
*.audit_file_dest='/opt/app/oracle/admin/asanga/adump'
*.audit_trail='NONE'
*.compatible='11.2.0.4.0'
*.db_block_size=8192
*.db_create_file_dest='+DATA'
*.db_domain=''
*.db_name='ASANGA'
*.db_recovery_file_dest_size=5218762752
*.db_recovery_file_dest='+FLASH'
*.diagnostic_dest='/opt/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=asangaXDB)'
*.open_cursors=300
*.pga_aggregate_target=524288000
*.processes=3000
*.remote_login_passwordfile='EXCLUSIVE'
*.sessions=4524
*.sga_target=2147483648
*.undo_tablespace='UNDOTBS1'
2. Create the adump directories in all nodes of the cluster (in this case it is a two node cluster)
mkdir -p /opt/app/oracle/admin/asanga/adump
3. Copy the pfile created earlier to one of the RAC nodes and restore the datababase.
SQL> startup nomount pfile='/home/oracle/backup/pfile.ora';

SQL> create spfile from pfile='/home/oracle/backup/pfile.ora';
SQL> startup force nomount;

RMAN> restore controlfile from '/home/oracle/backup/ctlbkp04qfl69p_1_1.ctl';

Starting restore at 28-AUG-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=2275 device type=DISK

channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07
output file name=+DATA/asanga/controlfile/current.270.888927491
output file name=+FLASH/asanga/controlfile/current.447.888927493
Finished restore at 28-AUG-15

RMAN> alter database mount;

RMAN> catalog start with '/home/oracle/backup/';

RMAN> run {
2> restore database;
3> recover database;
4> }

Starting restore at 28-AUG-15
using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to +DATA/asanga/datafile/system.309.888838053
channel ORA_DISK_1: restoring datafile 00002 to +DATA/asanga/datafile/sysaux.308.888838053
channel ORA_DISK_1: restoring datafile 00003 to +DATA/asanga/datafile/undotbs1.307.888838053
channel ORA_DISK_1: restoring datafile 00004 to +DATA/asanga/datafile/users.310.888838053
channel ORA_DISK_1: reading from backup piece /home/oracle/backup/full_bkp_ASANGA_20150827_02qfl697_1_1
channel ORA_DISK_1: piece handle=/home/oracle/backup/full_bkp_ASANGA_20150827_02qfl697_1_1 tag=FULL_BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:25
Finished restore at 28-AUG-15

Starting recover at 28-AUG-15
using channel ORA_DISK_1

starting media recovery

channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2
channel ORA_DISK_1: reading from backup piece /home/oracle/backup/full_arc_ASANGA_20150827_03qfl69n_1_1
channel ORA_DISK_1: piece handle=/home/oracle/backup/full_arc_ASANGA_20150827_03qfl69n_1_1 tag=FULL_ARC_BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
archived log file name=+FLASH/asanga/archivelog/2015_08_28/thread_1_seq_2.444.888927577 thread=1 sequence=2
channel default: deleting archived log(s)
archived log file name=+FLASH/asanga/archivelog/2015_08_28/thread_1_seq_2.444.888927577 RECID=3 STAMP=888927577
unable to find archived log
archived log thread=1 sequence=3
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 08/28/2015 12:19:38
RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence 3 and starting SCN of 1483282

RMAN> alter database open resetlogs;

database opened
4. Shutdown and restart the database to see if it opens without any issues.

5. The single instance will have only one redo thread. Add anther redo thread (or more if RAC has more nodes) and enable it.
SQL> select * from v$log;

    GROUP#    THREAD#  SEQUENCE#      BYTES  BLOCKSIZE    MEMBERS ARC STATUS           FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ ---------
         1          1          1   52428800        512          2 NO  CURRENT                1483283 28-AUG-15   2.8147E+14
         2          1          0   52428800        512          2 YES UNUSED                       0                      0
         3          1          0   52428800        512          2 YES UNUSED                       0                      0


alter database add logfile thread 2 group 4 ('+DATA','+FLASH') size 50m ;
alter database add logfile thread 2 group 5 ('+DATA','+FLASH') size 50m ;
alter database add logfile thread 2 group 6 ('+DATA','+FLASH') size 50m ;

SQL> alter database enable public thread 2;

Database altered.

SQL>  select * from v$log;

    GROUP#    THREAD#  SEQUENCE#      BYTES  BLOCKSIZE    MEMBERS ARC STATUS           FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ ---------
         1          1          1   52428800        512          2 NO  CURRENT                1483283 28-AUG-15   2.8147E+14
         2          1          0   52428800        512          2 YES UNUSED                       0                      0
         3          1          0   52428800        512          2 YES UNUSED                       0                      0
         4          2          1   52428800        512          2 YES INACTIVE               1484502 28-AUG-15      1484512 28-AUG-15
         5          2          0   52428800        512          2 YES UNUSED                       0                      0
         6          2          0   52428800        512          2 YES UNUSED                       0                      0

6 rows selected.
6. The single instance would have had one undo tablespace. Create undo tablespace for each additional instance
create undo tablespace UNDOTBS2 datafile '+DATA(datafile)' SIZE 600M AUTOEXTEND ON NEXT 64M MAXSIZE UNLIMITED;



7. Create a pfile from the spfile and edit it by removing the *.undo_tablespace='UNDOTBS1' and add instance specific entries
*.cluster_database_instances=2
*.cluster_database=true
asanga1.instance_number=1
asanga2.instance_number=2
asanga1.thread=1
asanga2.thread=2
asanga1.undo_tablespace='UNDOTBS1'
asanga2.undo_tablespace='UNDOTBS2'
Also make sure log archive format has thread number
log_archive_format                   string      %t_%s_%r.dbf
8. Shutdown the database and copy the new pfile to all nodes and start each instance using the pfile.
[oracle@rhel6m1 ~]$ export ORACLE_SID=asanga1

[oracle@rhel6m1 backup]$ sqlplus  / as sysdba

SQL> startup pfile='/home/oracle/backup/initasanga.ora';
ORACLE instance started.

Total System Global Area 2137886720 bytes
Fixed Size                  2254952 bytes
Variable Size            1811941272 bytes
Database Buffers          318767104 bytes
Redo Buffers                4923392 bytes
Database mounted.
Database opened.

SQL> select instance_name,status from v$instance;

INSTANCE_NAME    STATUS
---------------- ------------
asanga1          OPEN

[oracle@rhel6m2 ~]$ export ORACLE_SID=asanga2
[oracle@rhel6m2 ~]$ sqlplus  / as sysdba

SQL> startup pfile='/home/oracle/initasanga.ora';
ORACLE instance started.

Total System Global Area 2137886720 bytes
Fixed Size                  2254952 bytes
Variable Size            1811941272 bytes
Database Buffers          318767104 bytes
Redo Buffers                4923392 bytes
Database mounted.
Database opened.

SQL> select instance_name,status from v$instance;

INSTANCE_NAME    STATUS
---------------- ------------
asanga2          OPEN

SQL> select HOST_NAME,INSTANCE_NUMBER,INSTANCE_NAME,STATUS,THREAD# from gv$instance;

HOST_NAME                 INSTANCE_NUMBER INSTANCE_NAME    STATUS          THREAD#
------------------------- --------------- ---------------- ------------ ----------
rhel6m2.domain.net                     2 asanga2          OPEN                  2
rhel6m1.domain.net                     1 asanga1          OPEN                  1
9. Once confirmed that all instances are opening without any issue, create a spfile in a shared location.
SQL> create spfile='+data' from pfile='/home/oracle/backup/initasanga.ora';
Make an alias in ASM for the spfile
ASMCMD> mkalias spfile.283.888929113 spfileasanga.ora
ASMCMD> ls -l
Type           Redund  Striped  Time             Sys  Name
PARAMETERFILE  UNPROT  COARSE   AUG 28 12:00:00  Y    spfile.283.888929113
                                                 N    spfileasanga.ora => +DATA/ASANGA/PARAMETERFILE/spfile.283.888929113
Create instance specific pfile with entries to the spfile alias. Also remove the spfile created earlier from the local node
cat initasanga1.ora
spfile='+DATA/ASANGA/PARAMETERFILE/spfileasanga.ora'

scp initasanga1.ora rhel6m2:`pwd`/initasanga2.ora
10. Create oracle password files for each node

11. Run the following script to create cluster related views
@?/rdbms/admin/catclust.sql
After running this scrip the database registry will show RAC component entry
COMP_ID    COMP_NAME                           STATUS     VERSION
---------- ----------------------------------- ---------- ----------
RAC        Oracle Real Application Clusters    VALID      11.2.0.4.0
12.Add the database to the cluster
srvctl add database -d asanga -o $ORACLE_HOME -p "+DATA/ASANGA/PARAMETERFILE/spfileasanga.ora" 
srvctl add instance -d asanga -i asanga1 -n rhel6m1
srvctl add instance -d asanga -i asanga2 -n rhel6m2

[oracle@rhel6m1 dbs]$ srvctl config database -d asanga -a
Database unique name: asanga
Database name: rhel6m1
Oracle home: /opt/app/oracle/product/11.2.0/dbhome_4
Oracle user: oracle
Spfile: +DATA/ASANGA/PARAMETERFILE/spfileasanga.ora
Domain: local
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: asanga
Database instances: asanga1,asanga2
Disk Groups:
Mount point paths:
Services:
Type: RAC
Database is enabled
Database is administrator managed
13. Shutdown both instances and start the database using srvctl.
srvctl start database -d asanga
srvctl status database -d asanga
Instance asanga1 is running on node rhel6m1
Instance asanga2 is running on node rhel6m2

srvctl config database -d asanga -a
Database unique name: asanga
Database name: rhel6m1
Oracle home: /opt/app/oracle/product/11.2.0/dbhome_4
Oracle user: oracle
Spfile: +DATA/ASANGA/PARAMETERFILE/spfileasanga.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: asanga
Database instances: asanga1,asanga2
Disk Groups: DATA,FLASH
Mount point paths:
Services:
Type: RAC
Database is enabled
Database is administrator managed
14. Verify the database is also listed as a cluster resource
crsctl stat res ora.asanga.db
NAME=ora.asanga.db
TYPE=ora.database.type
TARGET=ONLINE           , ONLINE
STATE=ONLINE on rhel6m1, ONLINE on rhel6m2