WARNING::lib=/opt/oracle/extapi/64/asm err:9 rc:Directory does not exist

We were receiving this warning in both of our 11g RAC node's alert logs , after research i found that it is an ORACLE bug which is trying to warn you about ASMLIB not found in /opt !

error snap:

WARNING::lib=/opt/oracle/extapi/64/asm err:9 rc:Directory does not exist
location:skgdllOpenDi
errbuf=2
msgbuf=No such file or directory

Solution:

it is a warning not an error, safely ignore it!

Reference:

[1] Metalink Note. 727204.1

cheers

error, Switch to short timeout for ipc polling

if you have hard luck as me you are hit by another BUG !

symptoms:

you are receiving in /u01/app/diag/asm/+asm/+ASMSID/trace files

e.g.

Switch to short timeout for ipc polling
a session (kjzhi) is registered
session (kjzhi) is about to end
Registered session (kjzhi)[11][4][0][1] is cleaned up
Switch to long timeout for ipc polling

cause:

it is not your fault :) you require to run Patch id: 6678289

in case if you are using SPARC you may have to install SUN Patch id 123908-01 (or later) before doing Patch 6678289.

Reference:
[1] Metalink Note: 750773.1
[2] Metalink Note: 353150.1

Cheers

CRS-0184: Cannot communicate with the CRS daemon, Hunted!

after reboot of my rac nodes, i found that one of the nodes is having problem in starting up CRS i.e.

Error:

CRS-0184: Cannot communicate with the CRS daemon

Logs:

$ less /u01/app/crs/log/nodename/crsd/crsd.log

output:

2009-09-09 09:52:49.154: [ COMMCRS][2]clsc_connect: (10076b610) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_nodename_))

2009-09-09 09:52:49.154: [ CSSCLNT][1]clsssInitNative: failed to connect to (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_nodename_)), rc 9

2009-09-09 09:52:49.156: [ CRSRTI][1] CSS is not ready. Received status 3 from CSS. Waiting for good status ..

Reason:

sometimes when CRS server reboot it try to create sockets under /tmp/.oracle or /var/tmp/.oracle and there are already previous socket files...which are protecting to create new sockets.

Solution:

using root user remove all files under /tmp/.oracle or /var/tmp/.oracle

restart CRS on faulty node or even reboot that machine!

cheers

IPMP Error, All Interfaces in group ipmpsync have failed

we were having problem with our IPMP configured virtual interface, it was frequently fluctuating...and even once it was failover to other node in RAC, after investigation we found that there are following kind of errors reported in IPMP logs i.e.

error snap:

Cannot meet requested failure detection time of 10000 ms on (inet ce0) new failure detection time for group "ipmp0" is 188510 ms
Improved failure detection time 47127 ms on (inet ce1) for group "ipmp0"
All Interfaces in group ipmpsync have failed

after small googling i found this post, where this kinda behavior is explained, in fact, it is expected due to network overhead, as IPMP regularly after a small interval test its configured interfaces, and due to any reason if it is unable to test the availability it will report subject interface as down, and will increase the time interval with predefined amount of time...in case if this behavior continues it will declare interface down and possibly will failover that specific interface to any other available node.

We can manually increase or decrease the IPMP testing period/interval by modifying "FAILURE_DETECTION_TIME" to any value in milliseconds, inside /etc/default/mpathd file.

Then you need to run

pkill -HUP in.mpathd

You can continue to increase this value and once you define a rational amount of time where you stop getting this error, you can continue to work on network/communication tuning and find out the network overheads!

Cheers


Error in Voting/OCR disk during CRS installation

My CRS installation story is still continuing, as soon as i get some error, a new blog post is ready :)

now during CRS installation in the screen where you have to supply Voting/OCR disk you might get following error i.e.

The specified shared raw partition /dev/rdsk/ora_ocr_raw_280m may not have correct permission. Verify that the partition is owned by Oracle User.

and when you check your disk's permission, it is telling you something else !

$ls -ltr /dev/rdsk/ora_ocr_raw_280m
crw-rw---- 1 oracle oinstall 85, 8194 Sep 6 08:51 /dev/rdsk/ora_ocr_raw_280m

does it mean the permission is correctly set ? NO NOT YET ;)

you need to check the permission of actual device ! i.e.

# ls -ltr ../../devices/scsi_vhci/ssd@g50060e800000000000005ba500000020:a,raw
crw-r----- 1 root sys 118, 504 Aug 28 16:35 ../../devices/scsi_vhci/ssd@g50060e800000000000005ba500000020:a,raw

so from here you can see that the actual device is owned by root:sys which is causing this permission error...you need to change the permission of actual device to oracle:oinstall. i.e.

#chown -R -h oracle:oinstall ../../devices/scsi_vhci/ssd@g50060e800000000000005ba500000020:a,raw

# ls -ltr ../../devices/scsi_vhci/ssd@g50060e800000000000005ba500000020:a,raw

crw-r----- 1 oracle oinstall 118, 504 Sep 4 14:55 ../../devices/scsi_vhci/ssd@g50060e800000000000005ba500000020:a,raw


NOTE: in both voting & ocr you should use slice 0 of the disk.

cheers