<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Trebuchet MS";
panose-1:2 11 6 3 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal>Had a failover of my active/passive cluster and now the passive node will not rejoin the cluster.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>2 nodes running Ubuntu 12.04<o:p></o:p></p><p class=MsoNormal>coro 1.4.2-2, openais 1.1.4-4, pcmk 1.1.6-2ubuntu3<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Corosync ring membership is fine on both rings.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Tried stopping coro/pace and clearing /var/lib/heartbeat/crm/ and then restarting on passive node without success.<o:p></o:p></p><p class=MsoNormal>Tried rebooting passive node (again – it was successfully fenced)<o:p></o:p></p><p class=MsoNormal>Tried updating pacemaker to latest in distro (1.1.6-2ubuntu3.3) then went back on passive node<o:p></o:p></p><p class=MsoNormal>Tried putting active node in maintenance mode and stopping pacemaker and corosync on both nodes. Then restarting on both nodes. Corosync came back fine as before but now I have the same problem on both nodes with pacemaker not starting successfully. Both show exactly same now - attrd: [24883]: ERROR: main: HA Signon failed.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Log:<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor crmd: [24885]: info: crmd_init: Starting crmd<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor cib: [24881]: info: validate_with_relaxng: Creating RNG parser context<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor lrmd: [24882]: info: enabling coredumps<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor lrmd: [24882]: info: Started.<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor corosync[24778]: [IPC ] Invalid IPC credentials.<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor attrd: [24883]: ERROR: main: HA Signon failed<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor attrd: [24883]: ERROR: main: Aborting startup<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor pacemakerd: [24877]: ERROR: pcmk_child_exit: Child process attrd exited (pid=24883, rc=100)<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor pacemakerd: [24877]: notice: pcmk_child_exit: Child process attrd no longer wishes to be respawned<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor pacemakerd: [24877]: info: update_node_processes: Node Condor now has process list: 00000000000000000000000000110312 (was 00000000000000000000000000111312)<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor stonith-ng: [24880]: info: init_ais_connection_classic: AIS connection established<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor stonith-ng: [24880]: info: get_ais_nodeid: Server details: id=167837962 uname=Condor cname=pcmk<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor stonith-ng: [24880]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor stonith-ng: [24880]: info: crm_new_peer: Node Condor now has id: 167837962<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor stonith-ng: [24880]: info: crm_new_peer: Node 167837962 is now known as Condor<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor stonith-ng: [24880]: info: main: Starting stonith-ng mainloop<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor stonith-ng: [24880]: info: crm_update_peer: Node Condor: id=167837962 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000110312 (new)<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor cib: [24881]: info: startCib: CIB Initialization completed successfully<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor cib: [24881]: info: get_cluster_type: Cluster type is: 'openais'<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor cib: [24881]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor cib: [24881]: info: init_ais_connection_classic: Creating connection to our Corosync plugin<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor corosync[24778]: [IPC ] Invalid IPC credentials.<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor cib: [24881]: info: init_ais_connection_classic: Connection to our AIS plugin (9) failed: unknown (100)<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor cib: [24881]: CRIT: cib_init: Cannot sign in to the cluster... terminating<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor pacemakerd: [24877]: ERROR: pcmk_child_exit: Child process cib exited (pid=24881, rc=100)<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor pacemakerd: [24877]: notice: pcmk_child_exit: Child process cib no longer wishes to be respawned<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor pacemakerd: [24877]: info: update_node_processes: Node Condor now has process list: 00000000000000000000000000110212 (was 00000000000000000000000000110312)<o:p></o:p></p><p class=MsoNormal>Jan 27 01:09:59 Condor stonith-ng: [24880]: info: crm_update_peer: Node Condor: id=167837962 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000110212 (new)<o:p></o:p></p><p class=MsoNormal>Jan 27 01:10:00 Condor crmd: [24885]: info: do_cib_control: Could not connect to the CIB service: connection failed<o:p></o:p></p><p class=MsoNormal>Jan 27 01:10:00 Condor crmd: [24885]: WARN: do_cib_control: Couldn't complete CIB registration 1 times... pause and retry<o:p></o:p></p><p class=MsoNormal>Jan 27 01:10:00 Condor crmd: [24885]: info: crmd_init: Starting crmd's mainloop<o:p></o:p></p><p class=MsoNormal>Jan 27 01:10:01 Condor CRON[24888]: (root) CMD (/etc/init.d/watchdog -e >/dev/null 2>&1)<o:p></o:p></p><p class=MsoNormal>Jan 27 01:10:02 Condor crmd: [24885]: info: crm_timer_popped: Wait Timer (I_NULL) just popped (2000ms)<o:p></o:p></p><p class=MsoNormal>Jan 27 01:10:03 Condor crmd: [24885]: info: do_cib_control: Could not connect to the CIB service: connection failed<o:p></o:p></p><p class=MsoNormal>Jan 27 01:10:03 Condor crmd: [24885]: WARN: do_cib_control: Couldn't complete CIB registration 2 times... pause and retry<o:p></o:p></p><p class=MsoNormal>Jan 27 01:10:05 Condor crmd: [24885]: info: crm_timer_popped: Wait Timer (I_NULL) just popped (2000ms)<o:p></o:p></p><p class=MsoNormal>Jan 27 01:10:06 Condor crmd: [24885]: info: do_cib_control: Could not connect to the CIB service: connection failed<o:p></o:p></p><p class=MsoNormal>Jan 27 01:10:06 Condor crmd: [24885]: WARN: do_cib_control: Couldn't complete CIB registration 3 times... pause and retry<o:p></o:p></p><p class=MsoNormal>Jan 27 01:10:08 Condor crmd: [24885]: info: crm_timer_popped: Wait Timer (I_NULL) just popped (2000ms)<o:p></o:p></p><p class=MsoNormal>Jan 27 01:10:09 Condor crmd: [24885]: info: do_cib_control: Could not connect to the CIB service: connection failed<o:p></o:p></p><p class=MsoNormal>Jan 27 01:10:09 Condor crmd: [24885]: WARN: do_cib_control: Couldn't complete CIB registration 4 times... pause and retry<o:p></o:p></p><p class=MsoNormal>Jan 27 01:10:11 Condor crmd: [24885]: info: crm_timer_popped: Wait Timer (I_NULL) just popped (2000ms)<o:p></o:p></p><p class=MsoNormal>Jan 27 01:10:12 Condor crmd: [24885]: info: do_cib_control: Could not connect to the CIB service: connection failed<o:p></o:p></p><p class=MsoNormal>Jan 27 01:10:12 Condor crmd: [24885]: WARN: do_cib_control: Couldn't complete CIB registration 5 times... pause and retry<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-family:"Trebuchet MS",sans-serif;color:black'>Jacob A. Smith<br>IT Manager<br>Argotec, LLC<br><br></span><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>