[Pacemaker] Could not connect to the CIB: Remote node did not respond

[Liang.Ma at asc-csa.gc.ca]

Hi There,

After a network and power shutdown, my LAMP cluster servers were totally screwed up.

Now crm status gives me

crm status
============
Last updated: Wed Feb  9 09:44:17 2011
Stack: Heartbeat
Current DC: arsvr2 (bc6bf61d-6b5f-4307-85f3-bf7bb11531bb) - partition with quorum
Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd
2 Nodes configured, 1 expected votes
4 Resources configured.
============

Online: [ arsvr1 arsvr2 ]

None of the resources comes up.

First I found a brain split in drbd disks. I fixed that and the drbd disks are health. I can mount them manually without problem.

However if I try anything to bring up a resource or edit cib or even a query, it gives me errors as following

crm resource start fs_mysql
Call cib_replace failed (-41): Remote node did not respond <null>

crm configure edit
Could not connect to the CIB: Remote node did not respond
ERROR: creating tmp shadow __crmshell.2540 failed


cibadmin -Q
Call cib_query failed (-41): Remote node did not respond <null>

Any idea what I can do to bring the cluster back?

Thank you,

Liang Ma
Contractuel | Consultant | SED Systems Inc. 
Ground Systems Analyst
Agence spatiale canadienne | Canadian Space Agency
6767, Route de l'Aéroport, Longueuil (St-Hubert), QC, Canada, J3Y 8Y9
Tél/Tel : (450) 926-5099 | Téléc/Fax: (450) 926-5083
Courriel/E-mail : [liang.ma at space.gc.ca]
Site web/Web site : [www.space.gc.ca ] 




er