IPMP Error, All Interfaces in group ipmpsync have failed

we were having problem with our IPMP configured virtual interface, it was frequently fluctuating...and even once it was failover to other node in RAC, after investigation we found that there are following kind of errors reported in IPMP logs i.e.

error snap:

Cannot meet requested failure detection time of 10000 ms on (inet ce0) new failure detection time for group "ipmp0" is 188510 ms
Improved failure detection time 47127 ms on (inet ce1) for group "ipmp0"
All Interfaces in group ipmpsync have failed

after small googling i found this post, where this kinda behavior is explained, in fact, it is expected due to network overhead, as IPMP regularly after a small interval test its configured interfaces, and due to any reason if it is unable to test the availability it will report subject interface as down, and will increase the time interval with predefined amount of time...in case if this behavior continues it will declare interface down and possibly will failover that specific interface to any other available node.

We can manually increase or decrease the IPMP testing period/interval by modifying "FAILURE_DETECTION_TIME" to any value in milliseconds, inside /etc/default/mpathd file.

Then you need to run

pkill -HUP in.mpathd

You can continue to increase this value and once you define a rational amount of time where you stop getting this error, you can continue to work on network/communication tuning and find out the network overheads!

Cheers


Concept behind IP Network Addressing in crs installation, rac

As yesterday i was installation CRS and i got stuck on Network Configuration Screen, where i have to fill the Public, Virtual & Private network IPs... i was getting following error i.e.

You must enter unique values for the public node name, the private node name and the virtual hostname for all nodes in the cluster. The name, YOUR-HOSTNAME , that you entered is being used by more than once for the same node.

AND

The virtual hostname(s), YOUR-HOSTNAME, you have specified appears to be already assigned to another system on the network. Please ensure that the virtual hostname(s) that you use for each of the nodes in the cluster are not in use currently.

after reading the ORACLE official documentation and testing the provided machine's configuration i found that, my installation server had name resolution problem, firstly our DNS was resolving hostname on wrong IP and later when they changed it to resolve from /etc/hosts it was unable to resolve from this file...anyhow later with investigation i was able to give a brief idea that how CRS IP Network Configuration should look like, its simple ! here it is:

Concept:

You should have three IPs/Hostname i.e.

1. Public Hostname/IP (Physical Interface):
Public Hostname should register in DNS or /etc/hosts file and should be accessible i.e. one can ping it.

2. VIP Hostname/IP(Logical Interface):
VIP Hostname/IP should register in DNS or /etc/hosts file and should NOT be accessible i.e. one CANNOT ping it.

NOTE: in case of IPMP VIP should be LOGICAL interface.

3. Private Hostname/IP(Physical Interface):
Private Hostname should register in DNS or /etc/hosts file and should be accessible i.e. one can ping it.

cheers

Concept behind IPMP confiugration for RAC's Public & Interconnect, Solaris

For our coming "going to born" baby Oracle 11g RAC on Solaris 10, i wrote a step by step pre installation task guide, that i am going to release pretty soon, though i mentioned on that guide about configuration of IPMP, i was expecting that our system admins will be able to do by themselves but unfortunately it was just new for them, so i have to make more research to discuss the exact IPMP concept and configuration with them. Here are my finding

Consider you have 4 Network cards in your machine i.e.

1. c0 ---> Public IP Interface
2. c1 ---> Public IP Interface (standby)
3. c2 ---> Private Interconnect IP Interface
4. c3 ---> Private Interconnect IP Interface (standby)

so consider if you have subnet (172.168.1.0) for Public Interface and (10.2.1.0) as private interconnect subnet, i am not considering netmask and other IP/Network configuration in my example, but you should keep in mind that all Public IP's for RAC network configuration should be in single/same Subnet, while all Interconnect IP's should also be in single/same subnet.

So now 1st to configure Public Interface you need three/3 IPs e.g.

172.168.1.1, 172.168.1.2 & 172.168.1.3

so you will assign them like:

1. c0 ---> will be assign 172.168.1.1 --- called Physical IP
2. c0:1 ---> will be assign 172.168.1.2 --- called Test IP
3. c1 ---> will be assign 172.168.1.3 --- called Test IP

all three should be in same/single IPMP group.

for private interconnect, you need also three IPs e.g.

10.2.1.1, 10.2.1.2 & 10.2.1.3

1. c2 ---> will be assign 10.2.1.1 --- called Physical IP
2. c2:1 ---> will be assign 10.2.1.2 --- called Test IP
3. c3 ---> will be assign 10.2.1.3 --- called Test IP

all three should be in same/single IPMP group other then Public IPMP group. You should NOT keep Public & Interconnect IPMP group names similar, otherwise it can cause serious network problems and even ORACLE discourage doing this.

References:

1. Metalink Note: 283107.1
2. Metalink Note: 368464.1

Cheers