<div dir="ltr">Hi guys, <div><br></div><div style>Got a <font color="#000000" face="arial, sans-serif"><span style="white-space:nowrap">tricky issue with Corosync and Pacemaker over DHCP IP address using unicast. Corosync craches periodically. </span></font></div>
<div style><font color="#000000" face="arial, sans-serif"><span style="white-space:nowrap"><br></span></font></div><div style><font color="#000000" face="arial, sans-serif"><span style="white-space:nowrap">Packages are from centos 6 repos:</span></font></div>
<div style><div><font color="#000000" face="arial, sans-serif"><span style="white-space:nowrap">corosync-1.4.1-7.el6_3.1.x86_64</span></font></div><div><font color="#000000" face="arial, sans-serif"><span style="white-space:nowrap">corosynclib-1.4.1-7.el6_3.1.x86_64</span></font></div>
<div><font color="#000000" face="arial, sans-serif"><div style="white-space:nowrap">pacemaker-cluster-libs-1.1.7-6.el6.x86_64</div><div style="white-space:nowrap">pacemaker-libs-1.1.7-6.el6.x86_64</div><div style="white-space:nowrap">
pacemaker-cli-1.1.7-6.el6.x86_64</div><div style="white-space:nowrap">pacemaker-1.1.7-6.el6.x86_64</div><div style="white-space:nowrap"><br></div><div style="white-space:nowrap"><br></div><div style="white-space:nowrap"><div style="color:rgb(34,34,34);font-family:arial;white-space:normal">
<b>Logs</b></div><div style="color:rgb(34,34,34);font-family:arial;white-space:normal"><br></div><div style="color:rgb(34,34,34);font-family:arial;white-space:normal"><div>Feb 09 23:24:33 host1 lrmd: [5248]: info: rsc:P_SESSION_IP:25: monitor</div>
<div>Feb 10 00:24:39 host1 lrmd: [5248]: info: rsc:P_SESSION_IP:25: monitor</div><div>Feb 10 01:24:44 host1 lrmd: [5248]: info: rsc:P_SESSION_IP:25: monitor</div><div>Feb 10 02:24:48 host1 lrmd: [5248]: info: rsc:P_SESSION_IP:25: monitor</div>
<div>Feb 10 03:24:51 host1 lrmd: [5248]: info: rsc:P_SESSION_IP:25: monitor</div><div>Feb 10 04:24:52 host1 lrmd: [5248]: info: rsc:P_SESSION_IP:25: monitor</div><div>Feb 10 05:24:54 host1 lrmd: [5248]: info: rsc:P_SESSION_IP:25: monitor</div>
<div>Feb 10 06:25:00 host1 lrmd: [5248]: info: rsc:P_SESSION_IP:25: monitor</div><div>Feb 10 07:25:06 host1 lrmd: [5248]: info: rsc:P_SESSION_IP:25: monitor</div><div>Feb 10 07:56:22 corosync [TOTEM ] A processor failed, forming new configuration.</div>
<div>Feb 10 07:56:22 corosync [TOTEM ] The network interface is down.</div><div>Feb 10 07:56:24 corosync [TOTEM ] The network interface [172.17.0.104] is now up.</div><div>Feb 10 07:56:25 [5242] host1 pacemakerd: error: cfg_connection_destroy: Connection destroyed</div>
<div>Feb 10 07:56:25 [5251] host1 crmd: error: ais_dispatch: Receiving message body failed: (2) Library error: Resource temporarily unavailable (11)</div><div>Feb 10 07:56:25 [5246] host1 cib: error: ais_dispatch: Receiving message body failed: (2) Library error: Resource temporarily unavailable (11)</div>
<div>Feb 10 07:56:25 [5249] host1 attrd: error: ais_dispatch: Receiving message body failed: (2) Library error: Resource temporarily unavailable (11)</div><div>Feb 10 07:56:25 [5251] host1 crmd: error: ais_dispatch: AIS connection failed</div>
<div>Feb 10 07:56:25 [5242] host1 pacemakerd: error: cpg_connection_destroy: Connection destroyed</div><div>Feb 10 07:56:25 [5246] host1 cib: error: ais_dispatch: AIS connection failed</div><div>Feb 10 07:56:25 [5251] host1 crmd: info: crmd_ais_destroy: connection closed</div>
<div>Feb 10 07:56:25 [5249] host1 attrd: error: ais_dispatch: AIS connection failed</div><div>Feb 10 07:56:25 [5247] host1 stonith-ng: error: ais_dispatch: Receiving message body failed: (2) Library error: Resource temporarily unavailable (11)</div>
<div>Feb 10 07:56:25 [5246] host1 cib: error: cib_ais_destroy: AIS connection terminated</div><div>Feb 10 07:56:25 [5249] host1 attrd: crit: attrd_ais_destroy: Lost connection to OpenAIS service!</div>
<div>Feb 10 07:56:25 [5242] host1 pacemakerd: notice: pcmk_shutdown_worker: Shuting down Pacemaker</div><div>Feb 10 07:56:25 [5247] host1 stonith-ng: error: ais_dispatch: AIS connection failed</div><div>Feb 10 07:56:25 [5249] host1 attrd: notice: main: Exiting...</div>
<div>Feb 10 07:56:25 [5247] host1 stonith-ng: error: stonith_peer_ais_destroy: AIS connection terminated</div><div>Feb 10 07:56:25 [5242] host1 pacemakerd: notice: stop_child: Stopping crmd: Sent -15 to process 5251</div>
<div>Feb 10 07:56:25 [5249] host1 attrd: error: attrd_cib_connection_destroy: Connection to the CIB terminated...</div><div>Feb 10 07:56:25 [5251] host1 crmd: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated</div>
<div>Feb 10 07:56:25 [5251] host1 crmd: notice: crm_shutdown: Requesting shutdown, upper limit is 1200000ms</div><div>Feb 10 07:56:25 [5251] host1 crmd: info: do_shutdown_req: Sending shutdown request to host2</div>
<div>Feb 10 07:56:25 [5242] host1 pacemakerd: error: pcmk_child_exit: Child process stonith-ng exited (pid=5247, rc=1)</div><div>Feb 10 07:56:25 [5242] host1 pacemakerd: warning: send_ipc_message: IPC Channel to 5249 is not connected</div>
<div>Feb 10 07:56:25 [5242] host1 pacemakerd: warning: send_ipc_message: IPC Channel to 5246 is not connected</div><div>Feb 10 07:56:25 [5242] host1 pacemakerd: warning: send_ipc_message: IPC Channel to 5247 is not connected</div>
<div>Feb 10 07:56:25 [5242] host1 pacemakerd: error: send_cpg_message: Sending message via cpg FAILED: (rc=9) Bad handle</div><div>Feb 10 07:56:25 [5242] host1 pacemakerd: error: pcmk_child_exit: Child process cib exited (pid=5246, rc=1)</div>
<div>Feb 10 07:56:25 [5242] host1 pacemakerd: error: send_cpg_message: Sending message via cpg FAILED: (rc=9) Bad handle</div><div>Feb 10 07:56:25 [5242] host1 pacemakerd: error: pcmk_child_exit: Child process attrd exited (pid=5249, rc=1)</div>
<div>Feb 10 07:56:25 [5242] host1 pacemakerd: error: send_cpg_message: Sending message via cpg FAILED: (rc=9) Bad handle</div><div>Feb 10 07:56:27 [5251] host1 crmd: error: send_ais_text: Sending message 68 via pcmk: FAILED (rc=2): Library error: Connection timed out (110)</div>
<div>Feb 10 07:56:27 [5251] host1 crmd: error: do_log: FSA: Input I_ERROR from do_shutdown_req() received in state S_NOT_DC</div><div>Feb 10 07:56:27 [5251] host1 crmd: notice: do_state_transition: State transition S_NOT_DC -> S_RECOVERY [ input=I_ERROR cause=C_FSA_INTERNAL origin=do_shutdown_req ]</div>
<div>Feb 10 07:56:27 [5251] host1 crmd: error: do_recover: Action A_RECOVER (0000000001000000) not supported</div><div>Feb 10 07:56:27 [5251] host1 crmd: error: do_log: FSA: Input I_TERMINATE from do_recover() received in state S_RECOVERY</div>
<div>Feb 10 07:56:27 [5251] host1 crmd: notice: do_state_transition: State transition S_RECOVERY -> S_TERMINATE [ input=I_TERMINATE cause=C_FSA_INTERNAL origin=do_recover ]</div><div>Feb 10 07:56:27 [5251] host1 crmd: info: do_shutdown: Disconnecting STONITH...</div>
<div>Feb 10 07:56:27 [5251] host1 crmd: info: tengine_stonith_connection_destroy: Fencing daemon disconnected</div><div>Feb 10 07:56:27 host1 lrmd: [5248]: info: cancel_op: operation monitor[25] on ocf::OpenStackFloatingIP::P_SESSION_IP for client 5251, its parameters: CRM_meta_name=[monitor] crm_feature_set=[3.0.6] CRM_meta_timeout=[20000] CRM_meta_interval=[5000] ip=[172.24.0.104] cancelled</div>
<div>Feb 10 07:56:27 [5251] host1 crmd: error: verify_stopped: Resource P_SESSION_IP was active at shutdown. You may ignore this error if it is unmanaged.</div><div>Feb 10 07:56:27 [5251] host1 crmd: info: do_lrm_control: Disconnected from the LRM</div>
<div>Feb 10 07:56:27 [5251] host1 crmd: notice: terminate_ais_connection: Disconnecting from AIS</div><div>Feb 10 07:56:27 [5251] host1 crmd: info: do_ha_control: Disconnected from OpenAIS</div><div>
Feb 10 07:56:27 [5251] host1 crmd: info: do_cib_control: Disconnecting CIB</div><div>Feb 10 07:56:27 [5251] host1 crmd: error: send_ipc_message: IPC Channel to 5246 is not connected</div><div>Feb 10 07:56:27 [5251] host1 crmd: error: send_ipc_message: IPC Channel to 5246 is not connected</div>
<div>Feb 10 07:56:27 [5251] host1 crmd: error: cib_native_perform_op_delegate: Sending message to CIB service FAILED</div><div>Feb 10 07:56:27 [5251] host1 crmd: info: crmd_cib_connection_destroy: Connection to the CIB terminated...</div>
<div>Feb 10 07:56:27 [5251] host1 crmd: error: verify_stopped: Resource P_SESSION_IP was active at shutdown. You may ignore this error if it is unmanaged.</div><div>Feb 10 07:56:27 [5251] host1 crmd: info: do_exit: Performing A_EXIT_0 - gracefully exiting the CRMd</div>
<div>Feb 10 07:56:27 [5251] host1 crmd: error: do_exit: Could not recover from internal error</div><div>Feb 10 07:56:27 [5251] host1 crmd: info: free_mem: Dropping I_TERMINATE: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=do_stop ]</div>
<div>Feb 10 07:56:27 [5251] host1 crmd: info: crm_xml_cleanup: Cleaning up memory from libxml2</div><div>Feb 10 07:56:27 [5251] host1 crmd: info: do_exit: [crmd] stopped (2)</div><div>Feb 10 07:56:27 [5242] host1 pacemakerd: error: pcmk_child_exit: Child process crmd exited (pid=5251, rc=2)</div>
<div>Feb 10 07:56:27 [5242] host1 pacemakerd: warning: send_ipc_message: IPC Channel to 5251 is not connected</div><div>Feb 10 07:56:27 [5242] host1 pacemakerd: error: send_cpg_message: Sending message via cpg FAILED: (rc=9) Bad handle</div>
<div>Feb 10 07:56:27 [5242] host1 pacemakerd: notice: stop_child: Stopping pengine: Sent -15 to process 5250</div><div>Feb 10 07:56:27 [5242] host1 pacemakerd: info: pcmk_child_exit: Child process pengine exited (pid=5250, rc=0)</div>
<div>Feb 10 07:56:27 [5242] host1 pacemakerd: error: send_cpg_message: Sending message via cpg FAILED: (rc=9) Bad handle</div><div>Feb 10 07:56:27 [5242] host1 pacemakerd: notice: stop_child: Stopping lrmd: Sent -15 to process 5248</div>
<div>Feb 10 07:56:27 host1 lrmd: [5248]: info: lrmd is shutting down</div><div>Feb 10 07:56:27 [5242] host1 pacemakerd: info: pcmk_child_exit: Child process lrmd exited (pid=5248, rc=0)</div><div>Feb 10 07:56:27 [5242] host1 pacemakerd: error: send_cpg_message: Sending message via cpg FAILED: (rc=9) Bad handle</div>
<div>Feb 10 07:56:27 [5242] host1 pacemakerd: notice: pcmk_shutdown_worker: Shutdown complete</div><div>Feb 10 07:56:27 [5242] host1 pacemakerd: info: main: Exiting pacemakerd</div></div></div><div style="white-space:nowrap">
<br></div><div style="white-space:nowrap"><br></div><div style="white-space:nowrap"><b>corosync.conf:</b></div><div style="white-space:nowrap"><br></div><div><div>compatibility: whitetank</div><div><br></div><div>totem {</div>
<div> version: 2</div><div> secauth: off</div><div> nodeid: 104</div><div> interface {</div><div> member {</div><div> memberaddr: 172.17.0.104</div><div> }</div>
<div> member {</div><div> memberaddr: 172.17.0.105</div><div> }</div><div> ringnumber: 0</div><div> bindnetaddr: 172.17.0.0</div><div> mcastport: 5426</div>
<div> ttl: 1</div><div> }</div><div> transport: udpu</div><div>}</div><div><br></div><div>logging {</div><div> fileline: off</div><div> to_logfile: yes</div><div> to_syslog: yes</div>
<div> debug: on</div><div> logfile: /var/log/cluster/corosync.log</div><div> debug: off</div><div> timestamp: on</div><div> logger_subsys {</div><div> subsys: AMF</div><div>
debug: off</div><div> }</div><div>}</div><div>service {</div><div> # Load the Pacemaker Cluster Resource Manager</div><div> ver: 1</div><div> name: pacemaker</div><div>}</div>
<div><br></div><div>aisexec {</div><div> user: root</div><div> group: root</div><div>}</div></div><div><br></div><div><br></div></font></div></div><div><br></div><div>Thank you!<br clear="all"><div><br></div>
-- <br><div dir="ltr">Viacheslav Biriukov<br>BR<br><div><a href="http://biriukov.me" target="_blank">http://biriukov.me</a></div></div>
</div></div>