Showing posts with label spfile. Show all posts
Showing posts with label spfile. Show all posts

Friday, November 11, 2011

ASM for Standalone Server in 11gR2 with Role Separation

One of the major differences in 11gR2 compared to previous version is the moving of ASM configuration to Grid Infrastructure. Therefore to setup a standalone database with ASM, grid infrastructure is needed.
In this blog the grid user will be used for installing and managing grid infrastructure and oracle user will be used for installing and managing the database software. This is not a comprehensive step by step guide, some general database installation steps are not shown here focusing mainly on the steps that are specific to the topic at hand. The software used here are 11gR2 (11.2.0.3)

1. Create the following groups in the OS
groupadd oinstall
groupadd dba
groupadd oper
groupadd asmadmin
groupadd asmdba
groupadd asmoper
and create the grid user and oracle user as follows
useradd -g oinstall -G asmadmin,asmdba,asmoper grid
useradd -g oinstall -G dba,oper,asmdba oracle
If Oralce user does not have the asmdba then ASM disk groups created by grid user won't be visible to Oracle user.

2. Create the grid infrastructure location and make the ownership as grid:oinstall
cd /opt/app/11.2.0/
ls -l
drwxr-xr-x 2 grid oinstall 4096 Nov  3 17:41 grid
3. Install and configure oracleasm libs. User grid user and asmdba as the oracleasm interface user and group
/etc/init.d/oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting  without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmdba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
/etc/init.d/oracleasm createdisk DATA /dev/sdb1
Marking disk "DATA" as an ASM disk:                        [  OK  ]
 /etc/init.d/oracleasm createdisk FLASH /dev/sdc1
Marking disk "FLASH" as an ASM disk:                       [  OK  ]
4. Login as grid user and start the grid infrastructure installation and select grid infrastrcuture for standalone server.

5. Create a ASM diskgroup to store data files. On this step only one diskgroup could be created. Once the grid infrastrcuture is installed use ASMCA to create any additional diskgroups needed.

6. Select the OS group related ASM activities. If the OS groups mentioned in step 1 are created this step would come pre-selected as shown below.





7. When grid infrastructure is installed for RAC it is advised not to install it as a sub directory of ORACLE BASE. But in this case trying to install out side oracle base would give a warning as shown below.



The grid infrastructure path is selected as below.


8. Installing summary


9. Execute the root scripts when prompted
/opt/app/oracle/product/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/oracle/product/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node rhel5 successfully pinned.
Adding Clusterware entries to inittab

rhel5     2011/11/03 18:15:01     /opt/app/oracle/product/11.2.0/grid/cdata/rhel5/backup_20111103_181501.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server
10. Once the root scripts are finished listener and ASM instance will be created.


11. Following commands could be used to stop and start the grid infrastructure related services
crsctl stop has
crsctl start has
12. ASM's SPFile will be residing inside ASM itself. This could be found out in number of ways, looking at the alert log of ASM when ASM starts
Machine:        x86_64
Using parameter settings in server-side spfile +DATA/asm/asmparameterfile/registry.253.766260991
System parameters with non-default values:
  large_pool_size          = 12M
  instance_type            = "asm"
  remote_login_passwordfile= "EXCLUSIVE"
  asm_diskgroups           = "FLASH"
  asm_diskgroups           = "DATA"
  asm_power_limit          = 1
  diagnostic_dest          = "/opt/app/oracle"
Or using the asmcmd's spget command which shows the spfile location registered with GnP profile
ASMCMD> spget
+DATA/asm/asmparameterfile/registry.253.766260991
To copy the ASM spfile to a file systme location use spcopy (use spmove to move it). Before executing the spcopy stop the diskgroups
srvctl stop diskgroup -g data[flash] 
ASMCMD> spcopy +DATA/asm/asmparameterfile/registry.253.766260991 /opt/app/oracle/product/11.2.0/grid/dbs/spfile+ASM.ora
and update the GnP profile with the new location using spset
ASMCMD> spset /opt/app/oracle/product/11.2.0/grid/dbs/spfile+ASM.ora
ASMCMD> spget
/opt/app/oracle/product/11.2.0/grid/dbs/spfile+ASM.ora
When ASM starts next the new location is listed as below
Machine:        x86_64
Using parameter settings in server-side spfile /opt/app/oracle/product/11.2.0/grid/dbs/spfile+ASM.ora
System parameters with non-default values:
  large_pool_size          = 12M
  instance_type            = "asm"
  remote_login_passwordfile= "EXCLUSIVE"
  asm_diskgroups           = "FLASH"
  asm_diskgroups           = "DATA"
  asm_power_limit          = 1
  diagnostic_dest          = "/opt/app/oracle"
With this concludes the installation of grid infrastructure and creation of ASM.


13. Login as Oracle user and start the database software installation.


14. Some locations ($ORACLE_BASE/admin, $ORACLE_BASE/cfgtoollogs) may not have write permissions for Oracle user, permissions might be set as 755 grid : oinstall. Change permissions as 775 to allow oracle user the access to these location.



15. There is a pre-req check to see if grid user is also a member of the dba group. As per metalink note 1084186.1 "In case of stand alone set up, if the 11.2 Database be managed by Oracle Restart ( srvctl stop/start ), then 'grid' user should also be a part of 'dba' group."



16. Complete the installation.



17. Run dbca as Oracle user from the database software home. Creating database is straightforward. If anything can go wrong then it will be that ASM disk are not being shown when ASM is created for data file locations.


This will be because Oracle user is not in the asmdba group. Once added the disk will appear for selection. This concludes the creation of standalone database with ASM in 11gR2.

The database comes with Oracle restart configured as well as ASM as a target in the EM Console


Some useful metalink notes

DBCA Does Not Display ASM Disk Groups In 11.2 [ID 1177483.1] (issue seems to be fixed in 11.2.0.3)
Patch 11gR2 Grid Infrastructure Standalone (Oracle Restart) [ID 1089476.1]
11gR2 (11.2.0.1) Unable To Create ASM Spfile Standalone [ID 1082332.1]
Database Creation on 11.2 Grid Infracture with Role Separation ( ORA-15025, KFSG-00312, ORA-15081 ) [ID 1084186.1]
How to change ownership and group memberships of Grid home in 11.2 Standalone [ID 1171674.1]
Ora-01115, Ora-01110, Ora-15081 When Connecting DB with non-dba Group Users [ID 1372571.1]
ASM 11.2 Configuration KIT (ASM 11gR2 Installation & Configuration, Deinstallation, Upgrade, ASM Job Role Separation (Standalone Only). [ID 1092213.1]
How To Upgrade Oracle Restart i.e. Single Node Grid Infrastructure/ASM from 11.2.0.2 To 11.2.0.3. [ID 1400185.1]
Listener Log Showing * Service_died * Lsnragt * 12537 Died Message Every Minute. [ID 1349844.1]
WARNING: Subscription for node down event still pending' in Listener Log [ID 372959.1]

Related Post
Installing 11gR2 Standalone Server with ASM and Role Separation in RHEL 6

Friday, September 24, 2010

Restoring OCR & Vote disk due to ASM disk failures - 3

There are few changes to the setup that was used to test various ASM disk failure scenarios and restoring OCR and Vote disks. The main change is, setup used is no longer the RAC setup upgraded from 10gR2 to 11gR1 and then to 11gR2
It's a new installation of 11gR2 grid infrastructure (only GI was installed no RAC database was created. Installing GI was sufficient to test this scenario) and therefore the ASM spfile is also inside the same diskgroup as clusterware files. Metalink note 1082943.1 explains how to move it to another diskgroup. If not a pfile from the spfile should be created to re-create the spfile after the ocr and vote disk restore.

Scenario 3.

1. Both OCR and Vote disks are in ASM diskgroup
2. ASM diskgroup has normal redundancy with only three failure groups
3. All failure groups are affected
4. ASM Spfile is also located in the same diskgroup where clusterware files are located.

1. Current OCR, vote disk and ASM Spfile configuration
# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2272
Available space (kbytes) : 259848
ID : 1242190491
Device/File Name : +clusterdg
Device/File integrity check succeeded
Device/File not configured

Cluster registry integrity check succeeded
Logical corruption check succeeded


$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 7d49533611734f3bbf404d32f1759ed5 (ORCL:CLUS1) [CLUSTERDG]
2. ONLINE 4a8c288d1ade4f8cbf6588c145b27489 (ORCL:CLUS2) [CLUSTERDG]
3. ONLINE ad241f9823cd4fb9bf3412ca67e591df (ORCL:CLUS3) [CLUSTERDG]
Located 3 voting disk(s).

SQL> show parameter spfile

NAME TYPE VALUE
----------- ---------- -----
spfile string +CLUSTERDG/hpc-cluster/asmparameterfile/registry.253.730565167

srvctl config asm -a
ASM home: /opt/app/11.2.0/grid
ASM listener: LISTENER
ASM is enabled.
2. Identify the disks beloging to the ASM diskgroup using oracleasm query -p and corrupt them to simulate disk failure
# dd if=/dev/zero of=/dev/sdc10 count=204800 bs=8192
204800+0 records in
204800+0 records out
1677721600 bytes (1.7 GB) copied, 1.61087 seconds, 1.0 GB/s

# dd if=/dev/zero of=/dev/sdc3 count=204800 bs=8192
204800+0 records in
204800+0 records out
1677721600 bytes (1.7 GB) copied, 1.6883 seconds, 994 MB/s

# dd if=/dev/zero of=/dev/sdc2 count=204800 bs=8192
204800+0 records in
204800+0 records out
1677721600 bytes (1.7 GB) copied, 1.74684 seconds, 960 MB/s
3. ocssd.log will show the detection of vote disk corruption
2010-09-24 15:46:02.893: [    CSSD][1136630080]clssgmDestroyProc: cleaning up proc(0x2aaab02131b0) con(0x8db0) skgpid  ospid 12671 with 0 clients, refcount 0
2010-09-24 15:46:02.893: [ CSSD][1136630080]clssgmDiscEndpcl: gipcDestroy 0x8db0
2010-09-24 15:46:03.009: [ CSSD][1231038784]clssnmvDiskKillCheck: voting disk corrupted (0x00000000,0x00000000) (ORCL:CLUS1)
2010-09-24 15:46:03.009: [ CSSD][1231038784]clssnmvDiskAvailabilityChange: voting file ORCL:CLUS1 now offline
2010-09-24 15:46:03.584: [ CLSF][1241528640]Closing handle:0x2aaab008cbe0
...
2010-09-24 15:46:03.584: [ SKGFD][1241528640]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x2aaab0197570 for disk :ORCL:CLUS1:
2010-09-24 15:46:13.269: [ CSSD][1168099648]clssnmvDiskKillCheck: voting disk corrupted (0x00000000,0x00000000) (ORCL:CLUS3)
2010-09-24 15:46:13.269: [ CSSD][1168099648]clssnmvDiskAvailabilityChange: voting file ORCL:CLUS3 now offline
2010-09-24 15:46:13.431: [ CLSF][1157609792]Closing handle:0x2aaab006ac20
2010-09-24 15:46:13.431: [ SKGFD][1157609792]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x2aaab0062830 for disk :ORCL:CLUS3:
Querying vote disks frequently will enable to spot the state change
$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. PENDOFFL 7d49533611734f3bbf404d32f1759ed5 (ORCL:CLUS1) [CLUSTERDG]
2. ONLINE 4a8c288d1ade4f8cbf6588c145b27489 (ORCL:CLUS2) [CLUSTERDG]
3. ONLINE ad241f9823cd4fb9bf3412ca67e591df (ORCL:CLUS3) [CLUSTERDG]
Located 3 voting disk(s).

$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 7d49533611734f3bbf404d32f1759ed5 (ORCL:CLUS1) [CLUSTERDG]
2. ONLINE 4a8c288d1ade4f8cbf6588c145b27489 (ORCL:CLUS2) [CLUSTERDG]
3. ONLINE ad241f9823cd4fb9bf3412ca67e591df (ORCL:CLUS3) [CLUSTERDG]
Located 3 voting disk(s).
crs_stat (deprecated in 11gR2) shows cluster applications are running but ocrcheck fails
crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....ERDG.dg ora....up.type ONLINE ONLINE hpc1
ora....ER.lsnr ora....er.type ONLINE ONLINE hpc1
ora....N1.lsnr ora....er.type OFFLINE OFFLINE
ora.asm ora.asm.type ONLINE ONLINE hpc1
ora.eons ora.eons.type ONLINE ONLINE hpc1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE hpc1
ora....C1.lsnr application ONLINE ONLINE hpc1
ora.hpc1.gsd application OFFLINE OFFLINE
ora.hpc1.ons application ONLINE OFFLINE
ora.hpc1.vip ora....t1.type ONLINE ONLINE hpc1
ora....network ora....rk.type ONLINE ONLINE hpc1
ora.oc4j ora.oc4j.type OFFLINE OFFLINE
ora.ons ora.ons.type ONLINE OFFLINE
ora....ry.acfs ora....fs.type ONLINE ONLINE hpc1
ora.scan1.vip ora....ip.type OFFLINE OFFLINE

ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage
4. Stop the crs on all nodes and start crs in exclusive mode in one node. Manual shutdown of ASM instance and database instance might be required if the stop command is unable complete these operations.
crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'hpc1'
CRS-2673: Attempting to stop 'ora.crsd' on 'hpc1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'hpc1'
CRS-2673: Attempting to stop 'ora.CLUSTERDG.dg' on 'hpc1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'hpc1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'hpc1'
CRS-2677: Stop of 'ora.registry.acfs' on 'hpc1' succeeded
CRS-4549: Unexpected disconnect while executing shutdown request.
CRS-2677: Stop of 'ora.crsd' on 'hpc1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'hpc1'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'hpc1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'hpc1'
CRS-2673: Attempting to stop 'ora.evmd' on 'hpc1'
CRS-2673: Attempting to stop 'ora.asm' on 'hpc1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'hpc1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'hpc1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.asm' on 'hpc1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'hpc1'
CRS-2677: Stop of 'ora.cssd' on 'hpc1' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'hpc1'
CRS-2673: Attempting to stop 'ora.gipcd' on 'hpc1'
CRS-2677: Stop of 'ora.gipcd' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'hpc1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'hpc1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

crsctl start crs -excl
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.gipcd' on 'hpc1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'hpc1'
CRS-2676: Start of 'ora.gipcd' on 'hpc1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'hpc1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'hpc1'
CRS-2676: Start of 'ora.gpnpd' on 'hpc1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hpc1'
CRS-2676: Start of 'ora.cssdmonitor' on 'hpc1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'hpc1'
CRS-2679: Attempting to clean 'ora.diskmon' on 'hpc1'
CRS-2681: Clean of 'ora.diskmon' on 'hpc1' succeeded
CRS-2672: Attempting to start 'ora.diskmon' on 'hpc1'
CRS-2676: Start of 'ora.diskmon' on 'hpc1' succeeded
CRS-2676: Start of 'ora.cssd' on 'hpc1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'hpc1'
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'hpc1'
CRS-2676: Start of 'ora.drivers.acfs' on 'hpc1' succeeded
CRS-2676: Start of 'ora.ctssd' on 'hpc1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'hpc1'
CRS-2676: Start of 'ora.asm' on 'hpc1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'hpc1'
CRS-2676: Start of 'ora.crsd' on 'hpc1' succeeded
Various log files will show the status of vote disks and ocr disks. ocssd.log
2010-09-24 15:53:43.643: [    CSSD][1147734336]clssnmvDiskVerify: Successful discovery of 0 disks
2010-09-24 15:53:43.643: [ CSSD][1147734336]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2010-09-24 15:53:43.643: [ CSSD][1147734336]clssnmvFindInitialConfigs: No voting files found
2010-09-24 15:53:43.644: [ CSSD][1147734336]clssnmCompleteVFDiscovery: Completing voting file discovery
2010-09-24 15:53:43.644: [ CSSD][1147734336]clssnmvVerifyCommittedConfigVFs: Insufficient voting files found, found 0 of 0 configured, needed 1 voting files
crsd.log
2010-09-24 15:54:30.723: [  OCRASM][21660240]proprasmo: kgfoCheckMount returned [6]
2010-09-24 15:54:30.723: [ OCRASM][21660240]proprasmo: The ASM disk group clusterdg is not found or not mounted
2010-09-24 15:54:30.724: [ OCRRAW][21660240]proprioo: Failed to open [+clusterdg].
Returned proprasmo() with [26]. Marking loc
ion as UNAVAILABLE.
2010-09-24 15:54:30.724: [ OCRRAW][21660240]proprioo: No OCR/OLR devices are usable
2010-09-24 15:54:30.724: [ OCRASM][21660240]proprasmcl: asmhandle is NULL
2010-09-24 15:54:30.724: [ OCRRAW][21660240]proprinit: Could not open raw device
2010-09-24 15:54:30.724: [ OCRASM][21660240]proprasmcl: asmhandle is NULL
2010-09-24 15:54:30.724: [ OCRAPI][21660240]a_init:16!: Backend init unsuccessful : [26]
2010-09-24 15:54:30.724: [ CRSOCR][21660240] OCR context init failure. Error: PROC-26: Error while accessing the physical sto
ge ASM error [SLOS: cat=8, opn=kgfoOpenFile01, dep=15056, loc=kgfokge
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +CLUSTERDG.255.4294967295
ORA-17503: ksfdopn:2 Failed to open file +CLUSTERDG.255.4294967295
ORA-15001: disk] [8]
2010-09-24 15:54:30.724: [ CRSD][21660240][PANIC] CRSD exiting: Could not init OCR, code: 26
2010-09-24 15:54:30.724: [ CRSD][21660240] Done.
ASM alert log shows why the diskgroup containign vote disks and ocr wasn't mounted
SQL> ALTER DISKGROUP ALL MOUNT /* asm agent */
Diskgroup used for OCR is:CLUSTERDG
NOTE: cache registered group CLUSTERDG number=1 incarn=0x6ed824a0
NOTE: cache began mount (first) of group CLUSTERDG number=1 incarn=0x6ed824a0
NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
ERROR: no PST quorum in group: required 2, found 0
5. There's no SPfile for ASM but instance will be up
SQL> show parameter spfile

NAME TYPE VALUE
------- --------- ------
spfile string
6. To test various replace and repair scenarios, a diskgroup was created with a different name. Oracle Clusterware Admin Guide states "If the original OCR location does not exist, then you must create an empty (0 byte) OCR location before you run the ocrconfig -add or ocrconfig -replace commands. The OCR location that you are replacing can be either online or offline."

Replace and repair options could be used to replace the current location and to add, add and replace existing OCR locations. Cluster Admin guide also states "You cannot repair the OCR on a node on which Oracle Clusterware is running.If you run the ocrconfig -add | -repair | -replace command, then the device, file, or Oracle ASM disk group that you are adding must be accessible. This means that a device must exist. You must create an empty (0 byte) OCR location, or the Oracle ASM disk group must exist and be mounted."

There's the question of moutning ASM diskgroup while the crs down. If you try to manually mount the ASM instnace that is part of a cluster you'd get
sqlplus  / as sysasm

SQL> startup
ORA-01078: failure in processing system parameters
ORA-29701: unable to connect to Cluster Synchronization Service
All of the replace and repair option were useless and failed
# ocrconfig -restore /opt/app/11.2.0/grid/cdata/hpc-cluster/backup_20100924_142540.ocr
PROT-16: Internal Error

# ocrconfig -replace +clusterdg -replacement +clusterdgbk
PROT-28: Cannot delete or replace the only configured Oracle Cluster Registry location

# ocrconfig -repair -replace +clusterdg -replacement +clusterdgbk
PROT-21: Invalid parameter

# ocrconfig -repair -replace +clusterdg -replacement +clusterdgbk
PROT-21: Invalid parameter

# ocrconfig -repair -add +clusterdgbk
PROT-21: Invalid parameter

# ocrconfig -add +clusterdgbk
PROT-1: Failed to initialize ocrconfig
Even creating diskgroup with the original name and another one with new name and trying to repair replace also failed
SQL> create diskgroup clusterdg disk 'ORCL:CLUS1' disk 'ORCL:CLUS2' disk 'ORCL:CLUS3' attribute 'compatible.asm'='11.2';
Diskgroup created.

SQL> create diskgroup clusterdg2 disk 'ORCL:RED1' disk 'ORCL:RED2' disk 'ORCL:RED3' attribute 'compatible.asm'='11.2';
Diskgroup created.

# ocrconfig -repair -replace +clusterdg -replacement +clusterdg2
PROT-21: Invalid parameter
7. Finally created a diskgroup with the same name as original and did the OCR restore
SQL> create diskgroup clusterdg disk 'ORCL:RED1' disk 'ORCL:RED2' disk 'ORCL:RED3' attribute 'compatible.asm'='11.2';

ocrconfig -restore /opt/app/11.2.0/grid/cdata/hpc-cluster/backup_20100924_142540.ocr
Unlike the previous scenario there's no crs restart required to restore the vote disks, it was possible to restore the vote disks after restoring ocr
# crsctl query css votedisk
Located 0 voting disk(s).

# crsctl replace votedisk +clusterdg
Successful addition of voting disk 92814292a2ec4fa7bf2d6fe10960cc55
Successful addition of voting disk 57b152766d5f4f3dbf2935c93556b7f5
Successful addition of voting disk 6120792eb1284f64bf958f13a4947ece
Successfully replaced voting disk group with +clusterdg.
CRS-4266: Voting file(s) successfully replaced
8. Stop the crs on the node and start crs on all nodes
crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'hpc1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'hpc1'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'hpc1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'hpc1'
CRS-2673: Attempting to stop 'ora.asm' on 'hpc1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'hpc1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'hpc1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.asm' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'hpc1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'hpc1'
CRS-2677: Stop of 'ora.cssd' on 'hpc1' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'hpc1'
CRS-2673: Attempting to stop 'ora.gipcd' on 'hpc1'
CRS-2677: Stop of 'ora.gipcd' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'hpc1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'hpc1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
9. The last step in this scenario is recreate the ASM SPfile. When ASM instance is started with a spfile or pfile, ASM alert log shows the file location. For spfile
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options.
Using parameter settings in server-side spfile +CLUSTERDG/hpc-cluster/asmparameterfile/registry.253.730565167
System parameters with non-default values:
large_pool_size = 12M
instance_type = "asm"
remote_login_passwordfile= "EXCLUSIVE"
asm_diskstring = "ORCL:CLUS*"
asm_power_limit = 1
diagnostic_dest = "/opt/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance
For pfile
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options.
Using parameter settings in client-side pfile /opt/app/11.2.0/grid/dbs/init+ASM1.ora on machine hpc1
System parameters with non-default values:
large_pool_size = 12M
instance_type = "asm"
remote_login_passwordfile= "EXCLUSIVE"
asm_diskstring = "ORCL:CLUS*"
asm_power_limit = 1
diagnostic_dest = "/opt/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance
But after the corruption of all the disks on the ASM diskgroup, there won't be a server side spfile and even when there's no client side pfile ASM instance starts. ASM alert log shows that it is using default parameter setting without a parameter file
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options.
WARNING: using default parameter settings without any parameter file
Cluster communication is configured to use the following interface(s) for this instance
These default parameters cannot be considered as a spfile or a pfile trying to create one would fail
SQL> create pfile='/home/oracle/pfile.ora' from spfile;
create pfile='/home/oracle/pfile.ora' from spfile
*
ERROR at line 1:
ORA-01565: error in identifying file '?/dbs/spfile@.ora'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

SQL> create spfile from pfile;
create spfile from pfile
*
ERROR at line 1:
ORA-17502: ksfdcre:4 Failed to create file
+CLUSTERDG/hpc-cluster/asmparameterfile/registry.253.730565167
ORA-15177: cannot operate on system aliases
But a pfile could be created from these values in memory using
SQL> create pfile='/home/oracle/pfile.ora' from memory;

File created.
Or if a pfile is available from the time before the diskgroup corruption it could be used to restore the ASM spfile
SQL> create spfile='+clusterdg' from pfile='/home/oracle/asmpfile.ora';

File created.
Last command above would create a new spfile with a different alias than the original and ASM instance would use than during startup.
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options.
Using parameter settings in server-side spfile +CLUSTERDG/hpc-cluster/asmparameterfile/registry.253.730908391
System parameters with non-default values:
large_pool_size = 12M
instance_type = "asm"
remote_login_passwordfile= "EXCLUSIVE"
asm_diskstring = "ORCL:CLUS*"
asm_power_limit = 1
diagnostic_dest = "/opt/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance
Useful Metalink note
How to restore ASM based OCR after complete loss of the CRS diskgroup on Linux/Unix systems [ID 1062983.1]

Friday, July 16, 2010

Oracle 11gR2 RAC SPfile Issue

It seems in Oracle 11gR2 whenever srvctl is used to start the database it actively replaces the init*.ora entry in the ORACLE_HOME/dbs directory. But if the spfile was later recreated in ASM it will have a different filename and entry added during srvctl start (taken from OCR) is nolonger valid and shows a error message that is not relavent to the cause of the error.
Here are some observations.

on 10.2.0.4
srvctl srvctl config database -d livedb -a
tbxdb1 livedb2 /opt/oracle/app/oracle/product/10.2.0/db_1
tbxdb2 livedb1 /opt/oracle/app/oracle/product/10.2.0/db_1
DB_NAME: null
ORACLE_HOME: /opt/oracle/app/oracle/product/10.2.0/db_1
SPFILE: +DATA/livedb/spfilelivedb.ora
DOMAIN: domain.com
DB_ROLE: null
START_OPTIONS: null
POLICY: AUTOMATIC
ENABLE FLAG: DB ENABLED

more initlivedb2.ora
SPFILE='+DATA/livedb/spfilelivedb.ora'
on 11.1.0.7
srvctl config database -d livedb -a
278071-dbclus1 livedb1 /opt/dm/app/oracle/product/11.1.0/db_1
278072-dbclus2 livedb2 /opt/dm/app/oracle/product/11.1.0/db_1
DB_UNIQUE_NAME: livedb
DB_NAME: livedb
ORACLE_HOME: /opt/dm/app/oracle/product/11.1.0/db_1
SPFILE: +DATA/livedb/spfilelivedb.ora
DOMAIN: domain.com
DB_ROLE: null
START_OPTIONS: null
POLICY: AUTOMATIC
ENABLE FLAG: DB ENABLED

more initlivedb1.ora
SPFILE='+DATA/livedb/spfilelivedb.ora'
on 11.2.0.1
srvctl config database -d clusdb
Database unique name: clusdb
Database name:
Oracle home: /opt/app/oracle/product/11.2.0/clusdb
Oracle user: oracle
Spfile: +DATA/clusdb/spfileclusdb.ora_1277909842526
Domain: domain.net
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: clusdb
Database instances: clusdb1
Disk Groups: DATA,FLASH
Services:
Database is administrator managed

more initclusdb1.ora
SPFILE='+DATA/clusdb/spfileclusdb.ora_1277909842526' # line added by Agent
From the outputs above it could be seen that in 11.2 real spfile location is referred not the ASM alias.

But the real spfile name has changed due to recreation
ASMCMD> ls -l
Type Redund Striped Time Sys Name
Y CONTROLFILE/
Y DATAFILE/
Y ONLINELOG/
Y PARAMETERFILE/
Y TEMPFILE/
N spfileclusdb.ora => +DATA/CLUSDB/PARAMETERFILE/
spfile.265.723730605
When trying to start the database with srvctl following error is thrown
srvctl start database -d clusdb
PRCR-1079 : Failed to start resource ora.clusdb.db
ORA-01078: failure in processing system parameters
CRS-2674: Start of 'ora.clusdb.db' on 'hpc1' failed
CRS-2632: There are no more servers to try to place resource 'ora.clusdb.db' on that would satisfy its placement policy
One way to solve the problem is to edit the init*.ora file to point to spfiledbname.ora in the ASM instance and start the database with sqlplus.
vi initclusdb1.ora
SPFILE='+DATA/clusdb/spfileclusdb.ora'
sqlplus / as sysdba
sql>startup
will bring up the database but in subsequent srvctl start will add the original entry to the init*.ora file. Content of the ORACLE_HOME/dbs directory
ls
hc_clusdb1.dat initclusdb1.ora orapwclusdb orapwclusdb1 snapcf_clusdb1.f
more initclusdb1.ora
SPFILE='+DATA/clusdb/spfileclusdb.ora'
Only one init*.ora file and it's content is shown. Stop and start the database with srvctl
srvctl stop database -d clusdb
ls
hc_clusdb1.dat initclusdb1.ora orapwclusdb orapwclusdb1 snapcf_clusdb1.f

srvctl start database -d clusdb
PRCR-1079 : Failed to start resource ora.clusdb.db
ORA-01078: failure in processing system parameters
CRS-2674: Start of 'ora.clusdb.db' on 'hpc1' failed
CRS-2632: There are no more servers to try to place resource 'ora.clusdb.db' on that would satisfy its placement policy

ls
hc_clusdb1.dat initclusdb1.ora initclusdb1.ora.bak.hpc1 orapwclusdb orapwclusdb1 snapcf_clusdb1.f

more initclusdb1.ora
SPFILE='+DATA/clusdb/spfileclusdb.ora_1277909842526' # line added by Agent
As seen from the above outputs the existing init file is backed up and a new one created with the original spfile location by the agent.

Best way to fix it is by using srvctl modify to add the generic spfile location.
srvctl modify database -d clusdb -p +DATA/clusdb/spfileclusdb.ora
srvctl config database -d clusdb
Database unique name: clusdb
Database name:
Oracle home: /opt/app/oracle/product/11.2.0/clusdb
Oracle user: oracle
Spfile: +DATA/clusdb/spfileclusdb.ora
Domain: domain.net
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: clusdb
Database instances: clusdb1
Disk Groups: DATA,FLASH
Services:
Database is administrator managed
Whenever spfile is recreated new spfile could be linked to the ASM alias thus making the init*.ora file content valid all the time.

On metalink the error messages CRS-2632, CRS-2674, ORA-01078 has also been linked to metalink notes 1069254.1, 889845.1, 1071324.1, 1088754.1 with regard to 11gR2 issues.

Thursday, July 1, 2010

Moving OCR, Vote and ASM from Raw Devices to Block Devices

Two methods to move OCR from raw devices to block devices.
This blog uses the 10g cluster upgraded to 11g to move the raw device files (OCR,Vote, ASM files) to block devices.

Online method does not require any cluster downtime but require additional space, a new block device apart from the one already bound to the raw device.
Offline method requires a cluster downtime but does not require any additional space. It uses the existing block devices and removes the binding to the raw device.
This blog uses the Offline method. More on this and other methods on metalink note 428681.1

Moving OCR from raw device to block device with cluster downtime.

1. Shutdown all cluster services
crs_stop -all
and as root
crsctl stop crs
2. Check the status of the ocr
 ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 152364
Used space (kbytes) : 2780
Available space (kbytes) : 149584
ID : 552644455
Device/File Name : /dev/raw/raw1
Device/File integrity check succeeded

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded
3. As root run
ocrconfig -repair ocr /dev/sdc5
ocrconfig -overwrite
second command creates a file in $CRS_HOME/log/hostname/client/ocrconfig_pid.log. If there are multiple files then do a ls -lrt. The above command should have exited with a success status.
Oracle Database 11g CRS Release 11.1.0.7.0 - Production Copyright 1996, 2007 Oracle. All rights reserved.
2010-07-01 11:15:11.193: [ OCRCONF][2976058240]ocrconfig starts...
2010-07-01 11:15:12.883: [ OCRCONF][2976058240]Successfully overwrote OCR configuration on disk
2010-07-01 11:15:12.883: [ OCRCONF][2976058240]Exiting [status=success]...
since this only had on ocr location that's all required to move to block device.
Check the status of the ocr
ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 152364
Used space (kbytes) : 2780
Available space (kbytes) : 149584
ID : 552644455
Device/File Name : /dev/sdc5
Device/File integrity check succeeded

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded
4. Edit the udev rules files connected with raw devices and remove any entries added to bind this block device to raw device and change the permissions of the raw device.

5. Create a new udev rules file to set the permission and ownership of the block device to 640 and root:ointall
vi /etc/udev/rules.d/99-sdc.rules
#OCR
KERNEL=="sdc5*", GROUP="oinstall", MODE="640"
start_udev
Refer Udev under what conditions should start_udev be run?
# /sbin/udevadm trigger --type=subsystems --action=add
# /sbin/udevadm trigger --type=devices --action=add
# /sbin/udevadm trigger --type=subsystems --action=change
# /sbin/udevadm trigger --type=devices --action=change
Verify if the raw device binding to the block device is gone. If not restart the server which will start the cluster and use the new ocr location.

Moving Vote disk from raw devices to block devices

1. Query the currently configured voting disks
crsctl query css votedisk
0. 0 /dev/raw/raw2
Located 1 voting disk(s).
2. For 10g the cluster must be down and for 11.1 moving to a block device is an online operation and no cluster outage is required. 10g has a bug 3972986 which could corrupt the ocr if voting disk is added while cluster is up. More on this bug and workaround on metalink note 390880.1.
If the cluster only has one voting disks then a spare block device or raw device is needed, because cluster must have at least one voting disk inplace at all times, removal of all voting disks is not allowed. If there are multiple voting disks then moving to block device from raw device could be done one voting disk at a time.
Since this cluster only has one vote disk, a new partition was created of same size which is 150MB (10g only requires 20MB and 11g requires 280MB for new installations) and permissions and ownership is set. This spare device is called /dev/sdc10. raw2 is bound to /dev/sdc6

3. Backup the vote disk
dd if=/dev/raw/raw2 of=/home/oracle/votebackup
4. Add the vote disk to the spare block device vote disk, run the command as root
crsctl add css votedisk /dev/sdc10
Now formatting voting disk: /dev/sdc10.
Successful addition of voting disk /dev/sdc10.
Monitor the output on $CRS_HOME/log/hostname/alerthostname.log
2010-07-01 12:16:16.152
[cssd(6403)]CRS-1605:CSSD voting file is online: /dev/sdc10. Details in /opt/app/crs/product/10.2.0/crs/log/hpc1/cssd/ocssd.log.
2010-07-01 12:16:16.169
[cssd(6403)]CRS-1601:CSSD Reconfiguration complete. Active nodes are hpc1 .
Query the new vote disk
crsctl query css votedisk
0. 0 /dev/raw/raw2
1. 0 /dev/sdc10
Located 2 voting disk(s).
5. Remove the raw device vote disk
crsctl delete css votedisk /dev/raw/raw2
Successful deletion of voting disk /dev/raw/raw2.
$CRS_HOME/log/hostname/cssd/ocssd.log should have something similar to
[    CSSD]2010-07-01 12:22:44.983 [1220634944] >TRACE:   clssgmVotediskRemove: requested remove for votedisk 0 /dev/raw/raw2
[ CSSD]2010-07-01 12:22:45.801 [1189165376] >TRACE: clssnmDoSyncUpdate: Wait for 0 vote ack(s)
Query the vote disk is removed from the cluster
crsctl query css votedisk
1. 0 /dev/sdc10
Located 1 voting disk(s).
6. Add the block device that was used for the raw device as a vote disk, change the ownership and permissiosn on the block device before running the below command.
crsctl add css votedisk /dev/sdc6
Now formatting voting disk: /dev/sdc6.
Successful addition of voting disk /dev/sdc6.
From alert log
[cssd(6403)]CRS-1605:CSSD voting file is online: /dev/sdc6. Details in /opt/app/crs/product/10.2.0/crs/log/hpc1/cssd/ocssd.log.
2010-07-01 12:26:50.504
[cssd(6403)]CRS-1601:CSSD Reconfiguration complete. Active nodes are hpc1 .
From ocssd.log
[    CSSD]2010-07-01 12:26:50.489 [1220634944] >TRACE:   clssgmVotediskAdd: requested add for votedisk 0 /dev/sdc6
[ CSSD]2010-07-01 12:26:50.502 [1189165376] >TRACE: clssnmDoSyncUpdate: Wait for 0 vote ack(s)
Query the vote disks
crsctl query css votedisk
0. 0 /dev/sdc6
1. 0 /dev/sdc10
Located 2 voting disk(s).
7. Remove raw device rules from udev rules files and add a new rule for the vote disk in the sdc rule file created in ocr section.
#vote disk
KERNEL=="sdc6", OWNER="oracle", GROUP="oinstall", MODE="660"
8. Remove the spare vote disk added. If following error
 crsctl delete css votedisk /dev/sdc10
Failure 8 with Cluster Synchronization Services while deleting voting disk.
is thrown then restart the cluster and retry
crsctl delete css votedisk /dev/sdc10
Successful deletion of voting disk /dev/sdc10.
9. All done vote disk is moved to use the block device
crsctl query css votedisk
0. 0 /dev/sdc6
Located 1 voting disk(s).




Moving ASM Spfile to block device

If required the ASM spfile which was created in a shared storage raw device could be moved to a block device. Otherwise a init+ASM*.ora file must be created on each $ASM_HOME/dbs/ in the cluster.

1. Before moving create a pfile from the ASM spfile.
create pfile='/home/oracle/asmpfile.ora' from spfile; 
Metalink note 973031.1 describes how to move ASM spfile to shared device. That theory is used here to move from raw dervices to block devices

2. Shutdown database instances and ASM instances

3. Remove ASM spfile entries from the udev rules files, and add a new rule on the udev block device rule file.
# asm spfile
KERNEL=="sdc7", OWNER="oracle", GROUP="oinstall", MODE="660"
4. Unbind the raw device from the block device which will be used to store the ASM spfile. (even if this step is skipped change will still work and raw devices won't be used for ASM spfile)

5. Startup the ASM instance with the pfile created earlier to nomount state.
startup nomount pfile='/home/oracle/asmpfile.ora';
and create the ASM spfile in the new block device location
SQL> create spfile='/dev/sdc7' from pfile='/home/oracle/asmpfile.ora';
6. Edit the $ORACLE_HOME/dbs/init+ASM*.ora file to contain the new spfile location replace
SPFILE='/dev/raw/raw3'
to
SPFILE='/dev/sdc7'
and restart the asm instance. Verify on the asm alert log that new spfile is used
Starting up ORACLE RDBMS Version: 11.1.0.7.0.
Using parameter settings in server-side pfile /opt/app/oracle/product/11.1.0/clusdb/dbs/init+ASM1.ora
System parameters with non-default values:
large_pool_size = 12M
spfile = "/dev/sdc7"
instance_type = "asm"
cluster_database = TRUE
instance_number = 1
asm_diskgroups = "DATA"
asm_diskgroups = "FLASH"
diagnostic_dest = "/opt/app/oracle"
7. Shutdown the ASM instance and update the OCR ASM registry with the location of the new ASM spfile.
Usage: srvctl modify asm -n -i [-o ] [-p ]
-n Node name
-i ASM instance name
-o ORACLE_HOME path
-p Server parameter file path
srvctl modify asm -n hpc1 -i +ASM1 -o $ASM_HOME -p /dev/sdc7
Verify OCR reconfiguration on $CRS_HOME/log/hostname/alert*.log
[crsd(7207)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /opt/app/crs/product/10.2.0/crs/log/hpc1/crsd/crsd.log.
2010-07-01 12:38:49.111
[crsd(7207)]CRS-1012:The OCR service started on node hpc1.
2010-07-01 12:38:49.812
[evmd(7208)]CRS-1401:EVMD started on node hpc1.
2010-07-01 12:38:50.776
[crsd(7207)]CRS-1201:CRSD started on node hpc1.
2010-07-01 12:40:05.470
[cssd(8011)]CRS-1601:CSSD Reconfiguration complete. Active nodes are hpc1 .
8. Start the ASM instances and the database instances

Moving ASM diskgroups from raw devices to block devices


1. View the currently used raw devices
select mount_status,state,name,path from v$asm_disk;
MOUNT_S STATE NAME PATH
------- -------- --------------- -------------
CACHED NORMAL DATA_0000 /dev/raw/raw4
CACHED NORMAL FLASH_0000 /dev/raw/raw5
2. Change the asm_diskstring in spfile to discover the block devices instead of the raw devices
alter system set asm_diskstring='/dev/sdc8','/dev/sdc9' scope=spfile;
3. Shutdown the database instances, ASM instances

4. Remove udev rules for asm disk and add new udev rules for block devices
KERNEL=="sdc[8-9]*", OWNER="oracle", GROUP="oinstall", MODE="660"
5. Restart the udev services and then the asm instnaces. On the ASM alert log verify the block devices are used
NOTE: Assigning number (1,0) to disk (/dev/sdc8)
NOTE: Assigning number (2,0) to disk (/dev/sdc9)
5. New paths on the asm disk view.
MOUNT_ST STATE   NAME            PATH
-------- ------- --------------- ---------
CACHED NORMAL DATA_0000 /dev/sdc8
CACHED NORMAL FLASH_0000 /dev/sdc9

Related Posts
Migrating OCR to ASM in 11gR2 Clusterware
Migrating Voting Disk to ASM in 11gR2 Clusterware
Migrating block devices using ASM instance to ASMLib

Wednesday, April 28, 2010

Replacing spfile in ASM

In a database that is residing in a file system it is easy to replace the spfile with
create spfile from pfile
when the database is started with a pfile. But note the following when replacing a spfile that resides in ASM.

When the DB is created the spfile inside ASM is as below. spfile.260.714060565 is the exact spfile and spfilerac11g.ora is alias refering it

ASMCMD> ls -l
Type Redund Striped Time Sys Name
Y ARCHIVELOG/
Y CONTROLFILE/
Y DATAFILE/
Y ONLINELOG/
Y PARAMETERFILE/
Y TEMPFILE/
N spfilerac11g.ora => +DATA/RAC11G/PARAMETERFILE/spfile.260.714060565
Now create a pfile and start the DB using it
SQL> startup nomount pfile=pfile.ora
ORACLE instance started.

Total System Global Area 626327552 bytes
Fixed Size 2162280 bytes
Variable Size 385876376 bytes
Database Buffers 230686720 bytes
Redo Buffers 7602176 bytes

Create the spfile with the usual command try to start the DB

SQL> create spfile='+DATA' from pfile;

File created.

SQL> startup force;
The startup command will not return. Looking inside the ASM you will see the following

ASMCMD> ls -l
Type Redund Striped Time Sys Name
Y ARCHIVELOG/
Y CONTROLFILE/
Y DATAFILE/
Y ONLINELOG/
Y PARAMETERFILE/
Y TEMPFILE/
N spfilerac11g.ora => +DATA/RAC11G/PARAMETERFILE/spfile.260.714060565
ASMCMD> ls -l parameterfile
Type Redund Striped Time Sys Name
PARAMETERFILE UNPROT COARSE APR 28 23:00:00 Y spfile.260.714060565
PARAMETERFILE UNPROT COARSE APR 29 00:00:00 Y spfile.393.717552031

The alias is still refering the old spfile. Rectifiy this as follows

ASMCMD> rmalias spfilerac11g.ora
ASMCMD> mkalias +DATA/RAC11G/PARAMETERFILE/spfile.393.717552031 spfilerac11g.ora
ASMCMD> ls -l
Type Redund Striped Time Sys Name
Y ARCHIVELOG/
Y CONTROLFILE/
Y DATAFILE/
Y ONLINELOG/
Y PARAMETERFILE/
Y TEMPFILE/
N spfilerac11g.ora => +DATA/RAC11G/PARAMETERFILE/spfile.393.717552031

Try starting the DB and it would still hang. Problem lies how the spfile was created. When creating the spfile specify the full path of the creating pfile
create spfile='+DATA' from pfile='/home/oracle/asanga/pfile.ora';

Replace alias and start the DB

ASMCMD> ls parameterfile
spfile.260.714060565
spfile.393.717552031
spfile.394.717553085
ASMCMD> rmalias spfilerac11g.ora
ASMCMD> mkalias +DATA/RAC11G/PARAMETERFILE/spfile.394.717553085 spfilerac11g.ora
ASMCMD> ls -l
Type Redund Striped Time Sys Name
Y ARCHIVELOG/
Y CONTROLFILE/
Y DATAFILE/
Y ONLINELOG/
Y PARAMETERFILE/
Y TEMPFILE/
N spfilerac11g.ora => +DATA/RAC11G/PARAMETERFILE/spfile.394.717553085

SQL> startup force;
ORACLE instance started.

Total System Global Area 626327552 bytes
Fixed Size 2162280 bytes
Variable Size 385876376 bytes
Database Buffers 230686720 bytes
Redo Buffers 7602176 bytes
Database mounted.
Database opened.