Can't exec "/usr/bin/lsb_release": No such file or directory at GI_HOME/lib/osds_acfslib.pm line 564

Following was observed during the GI PSU 11.2.4.0.6 patch application

2015-06-15 11:19:01: acfs is supported
2015-06-15 11:19:01: Executing '/opt/app/oracle/product/11.2.0/grid_4/bin/acfsroot install'
2015-06-15 11:19:01: Executing cmd: /opt/app/oracle/product/11.2.0/grid_4/bin/acfsroot install
2015-06-15 11:23:00: Command output:
>  Can't exec "/usr/bin/lsb_release": No such file or directory at /opt/app/oracle/product/11.2.0/grid_4/lib/osds_acfslib.pm line 564.
>  Can't exec "/usr/bin/lsb_release": No such file or directory at /opt/app/oracle/product/11.2.0/grid_4/lib/osds_acfslib.pm line 564.
>  Can't exec "/usr/bin/lsb_release": No such file or directory at /opt/app/oracle/product/11.2.0/grid_4/lib/osds_acfslib.pm line 564.
>  Can't exec "/usr/bin/lsb_release": No such file or directory at /opt/app/oracle/product/11.2.0/grid_4/lib/osds_acfslib.pm line 564.
>  Can't exec "/usr/bin/lsb_release": No such file or directory at /opt/app/oracle/product/11.2.0/grid_4/lib/osds_acfslib.pm line 564.
>  Can't exec "/usr/bin/lsb_release": No such file or directory at /opt/app/oracle/product/11.2.0/grid_4/lib/osds_acfslib.pm line 564.
>  Can't exec "/usr/bin/lsb_release": No such file or directory at /opt/app/oracle/product/11.2.0/grid_4/lib/osds_acfslib.pm line 564.
>  Can't exec "/usr/bin/lsb_release": No such file or directory at /opt/app/oracle/product/11.2.0/grid_4/lib/osds_acfslib.pm line 564.
>  Can't exec "/usr/bin/lsb_release": No such file or directory at /opt/app/oracle/product/11.2.0/grid_4/lib/osds_acfslib.pm line 564.
>  ACFS-9300: ADVM/ACFS distribution files found.
>  ACFS-9312: Existing ADVM/ACFS installation detected.
>  ACFS-9314: Removing previous ADVM/ACFS installation.
>  Can't exec "/usr/bin/lsb_release": No such file or directory at /opt/app/oracle/product/11.2.0/grid_4/lib/osds_acfslib.pm line 564.
>  ACFS-9315: Previous ADVM/ACFS components successfully removed.
>  ACFS-9307: Installing requested ADVM/ACFS software.
>  ACFS-9308: Loading installed ADVM/ACFS drivers.
>  ACFS-9321: Creating udev for ADVM/ACFS.
>  ACFS-9323: Creating module dependencies - this may take some time.
>  Can't exec "/usr/bin/lsb_release": No such file or directory at /opt/app/oracle/product/11.2.0/grid_4/lib/osds_acfslib.pm line 564.
>  Can't exec "/usr/bin/lsb_release": No such file or directory at /opt/app/oracle/product/11.2.0/grid_4/lib/osds_acfslib.pm line 564.
>  ACFS-9154: Loading 'oracleoks.ko' driver.
>  Can't exec "/usr/bin/lsb_release": No such file or directory at /opt/app/oracle/product/11.2.0/grid_4/lib/osds_acfslib.pm line 564.
>  ACFS-9154: Loading 'oracleadvm.ko' driver.
>  Can't exec "/usr/bin/lsb_release": No such file or directory at /opt/app/oracle/product/11.2.0/grid_4/lib/osds_acfslib.pm line 564.
>  ACFS-9154: Loading 'oracleacfs.ko' driver.
>  Can't exec "/usr/bin/lsb_release": No such file or directory at /opt/app/oracle/product/11.2.0/grid_4/lib/osds_acfslib.pm line 564.
>  ACFS-9327: Verifying ADVM/ACFS devices.
>  ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
>  ACFS-9156: Detecting control device '/dev/ofsctl'.
>  ACFS-9309: ADVM/ACFS installation correctness verified.
>End Command output

The environment is OEL 6 (2.6.32-358.18.1.el6.x86_64).
Reason for this error is bug 17359415. Install redhat-lsb-core-4.0*.el6.x86_64 rpm to fix this problem.

# yum install redhat-lsb-core-4.0*.el6.x86_64
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package redhat-lsb-core.x86_64 0:4.0-7.0.1.el6 will be installed

ORA-15477 cannot communicate with the volume driver

ORA-15477: cannot communicate with the volume driver could be seen when setting the compatible.admv value on a ASM disk group. The system was running 11.2.0.3 (11.2.0.3.4) RAC on RHEL 6 (2.6.32-220.el6.x86_64) and it's a brand new installation of RAC (1488004.1).

SQL> alter diskgroup flash set attribute 'compatible.advm'='11.2';
alter diskgroup flash set attribute 'compatible.advm'='11.2'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15242: could not set attribute compatible.advm
ORA-15238: 11.2 is not a valid value for attribute compatible.advm
ORA-15477: cannot communicate with the volume driver

It seems the ACFS/ADMV modules are not loaded with system reboot as

[root@rhel6m1 ~]# lsmod | grep oracle

didn't return anything.
Reinstall acfs/admv modules manually (1371067.1)

[root@rhel6m1 bin]# ./acfsroot install
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9309: ADVM/ACFS installation correctness verified.

After this acfs modules will be loaded

lsmod  | grep oracle
oracleacfs           1857627  2
oracleadvm            238976  6
oracleoks             330757  2 oracleacfs,oracleadvm

and change of compatible.admv would proceed without an error and acfs volumes could be created

SQL> alter diskgroup flash set attribute 'compatible.advm'='11.2';
Diskgroup altered.

SQL> alter diskgroup flash add volume volume1 size 1g;
Diskgroup altered.

SQL> select volume_name,volume_device from v$asm_volume;

VOLUME_NAME                    VOLUME_DEVICE
------------------------------ --------------------
VOLUME1                        /dev/asm/volume1-149

# mkfs -t acfs /dev/asm/volume1-149
mkfs.acfs: version                   = 11.2.0.3.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/volume1-149
mkfs.acfs: volume size               = 1073741824
mkfs.acfs: Format complete.

acfsutil registry -a /dev/asm/volume1-149 /opt/acfsvol

#mount -t acfs  /dev/asm/volume1-149 /opt/acfsvol

However after the reboot of the servers the acfs modules do not get auto loaded and following could be seen on ASM alert log

SUCCESS: ALTER DISKGROUP ALL MOUNT /* asm agent call crs *//* {0:0:2} */
SQL> ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent *//* {0:0:2} */
kfvxVolOnOff: Cannot open device file
ORA-15032: not all alterations performed
ORA-15477: cannot communicate with the volume driver
ERROR: ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent *//* {0:0:2} */

To fix this run acfsroot enable. This command requires acfs to be started before running it

[root@rhel6m1 grid]# $GI_HOME/bin/acfsroot enable
ACFS-9167: ADVM/ACFS is not installed or loaded. Run 'acfsroot install'.

[root@rhel6m1 grid]# $GI_HOME/bin/acfsload start -s
[root@rhel6m1 grid]# $GI_HOME/bin/acfsroot enable
ACFS-9376: Adding ADVM/ACFS drivers resource succeeded.
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rhel6m1'
CRS-2676: Start of 'ora.drivers.acfs' on 'rhel6m1' succeeded
ACFS-9380: Starting ADVM/ACFS drivers resource succeeded.
ACFS-9368: Adding ACFS registry resource succeeded.
CRS-2672: Attempting to start 'ora.registry.acfs' on 'rhel6m1'
CRS-2676: Start of 'ora.registry.acfs' on 'rhel6m1' succeeded
ACFS-9372: Starting ACFS registry resource succeeded.

This issue is mentioned in (1488004.1 and 1486208.1)
Afterwards acfs modules were auto loading on reboot and acfs volumes were getting auto mounted without a problem.

Useful metalink notes
ACFS Supported On OS Platforms. [ID 1369107.1]
ACFS Filesystem Not Mounted After Host Reboot [ID 1473875.1]
How To Install/Reinstall Or Deinstall ACFS Modules/Installation Manually? [ID 1371067.1]
11.2.0.3.3 Grid Infrastructure Patchset: 13919095 Installation On RH 6.3 Fails With ACFS-9459 . [ID 1493491.1]
ORA.REGISTRY.ACFS Resource Appears As OFFLINE On A Brand New 11.2 RAC Installation/Configuration. [ID 1488004.1]
ACFS/ADVM is NOT started automatically after node reboot or after CRS is restarted in non-RAC environment [ID 886407.1]
“ora.drivers.acfs” Resource Was Not Configured Therefore RAC ACFS Filesystem Is Not Mounting During The Reboot. [ID 1486208.1]

Related Post
Creating and Removing ACFS

Creating and Removing ACFS

Oracle Automatic Storage Management Cluster File System (Oracle ACFS) is a multi-platform, scalable file system, and storage management technology that extends Oracle Automatic Storage Management (Oracle ASM) functionality to support customer files maintained outside of Oracle Database.

Creating ACFS

1. Set the diskgroup attribute parameters. Compatible.asm and compatible.advm should be >= 11.2

$ sqlplus  / as sysasm
SQL> alter diskgroup flash set attribute 'compatible.asm'='11.2';
SQL> alter diskgroup flash set attribute 'compatible.advm'='11.2';

2. Create a volume in the appropriate diskgroup

SQL> alter diskgroup flash add volume volume1 size 1g;

This could also be done using asmcmd. Volume device could be identified with

select volume_name,volume_device from v$asm_volume;
VOLUME_NAME    VOLUME_DEVICE
-------------- ---------------------
VOLUMNE1       /dev/asm/volumne1-398

This could also confirmed with ls /dev/asm/*

3. Create a file system on the created volume

# mkfs -t acfs /dev/asm/volumne1-398
mkfs.acfs: version                   = 11.2.0.2.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/volumne1-398
mkfs.acfs: volume size               = 1073741824
mkfs.acfs: Format complete.

(Update 2013/10/30)
mkfs could also be run as grid user provided that user has the permission on the device

[grid@rhel6m1 ~]$ ls -l /dev/asm/volume1-149
brwxrwx---. 1 root asmadmin 252, 76289 Oct 30 15:31 /dev/asm/volume1-149
[grid@rhel6m1 ~]$  mkfs -t acfs /dev/asm/volume1-149
mkfs.acfs: version                   = 11.2.0.3.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/volume1-149
mkfs.acfs: volume size               = 1073741824
mkfs.acfs: Format complete.

4. Register the file system with

acfsutil registry -a /dev/asm/volumne1-398 /opt/acfsvol

/opt/acfsvol will be the mountpoint for this file system.

5. Mount the file system

# mount -t acfs /dev/asm/volumne1-398 /opt/acfsvol

ASMCMD> volinfo -G flash VOLUMNE1
Diskgroup Name: FLASH

         Volume Name: VOLUMNE1
         Volume Device: /dev/asm/volumne1-398
         State: ENABLED
         Size (MB): 1024
         Resize Unit (MB): 256
         Redundancy: UNPROT
         Stripe Columns: 4
         Stripe Width (K): 128
         Usage: ACFS
         Mountpath: /opt/acfsvol

6. Change ownership to Oracle user to allow oracle user processes to use the file system

# chown oracle:oinstall /opt/acfsvol

6. As oracle user create a file in the mount point

cd /opt/acfsvol
touch x

In a RAC system the volume will be mounted on all nodes after server reboots and requires selinux to be permissive.




Removing ACFS

1. Unmount the file system

# umount /opt/acfsvol

2. De-register the file system

# acfsutil registry -d /opt/acfsvol
acfsutil registry: successfully removed ACFS mount point /opt/acfsvol from Oracle Registry

3. Drop the volumne from the diskgroup

SQL> alter diskgroup flash drop volume volume1;

Diskgroup altered.

SQL> select volume_name,volume_device from v$asm_volume;

no rows selected