11g RAC

Getting following error during CRS start. we had a server reboot post maintenance.
CRS-0804, PROC-44, CRS-4535, PROC-44: Error in network address

CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager

[crsd(12451930)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-44: Error in network address and interface operations Network address
and interface operations error [7]]. Details at (:CRSD00111:) in /grid/app/11.2.0.3/grid/log/sam-rac01/crsd/crsd.log.
2017-03-20 13:46:39.533:
[CLWAL][1]clsw_Initialize: OLR initlevel [70000]
2017-03-20 13:46:42.946: [ OCRAPI][1]clsu_get_private_ip_addresses: no ip addresses found.
[ OCRAPI][1]a_init_clsss: failed to call clsu_get_private_ip_addr (7)
2017-03-20 13:46:43.149: [ OCRAPI][1]a_init:13!: Clusterware init unsuccessful : [44]
2017-03-20 13:46:43.149: [ CRSOCR][1] OCR context init failure. Error: PROC-44: Error in network address and interface operations Network address and interface operat
ions error [7]
2017-03-20 13:46:43.150: [ CRSMAIN][1] Created alert : (:CRSD00111:) : Could not init OCR, error: PROC-44: Error in network address and interface operations Network ad
dress and interface operations error [7]
2017-03-20 13:46:43.150: [ CRSD][1][PANIC] CRSD exiting: Could not init OCR, code: 44
2017-03-20 13:46:43.150: [ CRSD][1] Done.</pre>

Looking at the logs above this seems to be issue with network.
Verify the same by comparing output of following two commands.

$GRID_HOME/bin/oifcfg iflist -p -n
$GRID_HOME/bin/gpnptool get

On output 1 the interface name for PRIVATE network is bond0 where as from
output 2 the interface name for cluster_interconnect in bond1.
Output 1:

grid@sam-rac01::/u01/app/grid/product/11.2.0/grid_2/bin: oifcfg iflist -p -n
bond0 10.55.9.0 PRIVATE 255.255.255.0
bond0 172.16.250.0 PRIVATE 255.255.255.0
bond0 169.254.156.0 UNKNOWN 255.255.255.0

Output 2:

grid@sam-rac01::/u01/app/grid/product/11.2.0/grid_2/bin: gpnptool get
Warning: some command line parameters were defaulted. Resulting command line:
./gpnptool.bin get -o-

<?xml version="1.0" encoding="UTF-8"?>
<gpnp:GPnP-Profile Version="1.0" xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.xsd" ProfileSequence="6" ClusterUId="53cd88e0368eeffdff7b1aa8bfeb99a3" ClusterName="its-racd" PALocation="">
<gpnp:Network-Profile>
<gpnp:HostNetwork id="gen" HostName="*">
<gpnp:Network id="net1" IP="10.55.9.0" Adapter="bond0" Use="public"/>
<gpnp:Network id="net2" IP="172.16.250.0" Adapter="bond1" Use="cluster_interconnect"/>
</gpnp:HostNetwork>
</gpnp:Network-Profile>
<orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/>
<orcl:ASM-Profile id="asm" DiscoveryString="" SPFile="+DATA/its-racd/asmparameterfile/registry.253.743595285"/>
<ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
<ds:SignedInfo>
<ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
<ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
<ds:Reference URI="">
<ds:Transforms>
<ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
<ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">
<InclusiveNamespaces xmlns="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="gpnp orcl xsi"/>
</ds:Transform></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
<ds:DigestValue>8HwbUAQu9nZIzAFBq43nRqpaNeM=</ds:DigestValue>
</ds:Reference>
</ds:SignedInfo>
<ds:SignatureValue>RjumSmZayRFBKzZTaD85jBNFCw9/CHJCXI0WRsAMv7UU4BwIRDUDL/A4YBpqWdhD99ka4JDB4hbLlAseG0SfNg/lnSh5cuVL9dWOpcmUrmRH6kZohVy6xeNvdwQ3hPPto9p2A3PlfE+iW5cl7WfBYCUwCXlVCXPOB3m6G5hpbI8=</ds:SignatureValue>
</ds:Signature>
</gpnp:GPnP-Profile>
Success.

Above XML output is formatted for better readanbility.

As per MOS note:

There are 2 possible solutions:
A. Restore the old network configuration and execute the necessary steps in the correct order.
Check with your network administrator if the old network configuration can be restored, this should allow the clusterware to start.
Once the clusterware is running again, follow the instructions in
Note 283684.1 “How to Modify Private Network Information in Oracle Clusterware” to make the subnet change.

B. De-configure and re-configure the clusterware
If for some reason it is not possible to restore the old network configuration, a rebuild of the OCR is necessary – as documented in
Note 1377349.1 “How to Deconfigure/Reconfigure(Rebuild OCR) or De-install Grid Infrastructure”

 

Excuse Typos. Thanks for reaching my blog.