<div dir="ltr">Hi,<div><br></div><div style>I am having an issue with Pacemaker on the cman stack where I can reliably cause pacemaker to crash and coredump when I put the node in standby or unstandby. Here is my messages log from when I do a unstandby like so "pcs cluster unstandby `uname -n`":</div>
<div style><br></div><div style><div><div>Oct 18 18:33:11 localhost crmd[19258]: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] [7/3857]</div>
<div>Oct 18 18:33:11 localhost cib[19253]: notice: cib:diff: Diff: --- 0.45.4</div><div>Oct 18 18:33:11 localhost cib[19253]: notice: cib:diff: Diff: +++ 0.46.1 01d8e730566358dcb36a7bc5b9ac587c</div><div>Oct 18 18:33:11 localhost cib[19253]: notice: cib:diff: -- <nvpair id="nodes-node01-standby" name="standby" value="on" /></div>
<div>Oct 18 18:33:11 localhost cib[19253]: notice: cib:diff: ++ <cib epoch="46" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.7" update-origin="node01" update-client="crm_attribute" cib-last-</div>
<div>written="Fri Oct 18 18:32:57 2013" have-quorum="0" dc-uuid="node01" /></div><div>Oct 18 18:33:11 localhost pengine[19257]: notice: unpack_config: On loss of CCM Quorum: Ignore</div>
<div>Oct 18 18:33:11 localhost pengine[19257]: notice: LogActions: Start reader_vip_2#011(node01)</div><div>Oct 18 18:33:11 localhost pengine[19257]: notice: LogActions: Start writer_vip#011(node01)</div><div>Oct 18 18:33:11 localhost pengine[19257]: notice: LogActions: Start reader_vip_1#011(node01)</div>
<div>Oct 18 18:33:11 localhost cib[19253]: error: cib_ipc_closed: Client not found in the hashtable</div><div>Oct 18 18:33:11 localhost cib[19253]: error: qb_ipcs_connection_unref: ref:0 state:3 (19253-20281-21)</div>
<div>Oct 18 18:33:11 localhost pengine[19257]: notice: process_pe_message: Calculated Transition 4: /var/lib/pacemaker/pengine/pe-input-77.bz2</div><div>Oct 18 18:33:12 localhost lrmd[19255]: notice: operation_finished: reader_vip_2_start_0:20284 [ 2013/10/18_18:33:12 INFO: ip -f inet addr add <a href="http://10.10.10.9/32">10.10.10.9/32</a> brd 10.10.10.9 dev eth1 ]</div>
<div>Oct 18 18:33:12 localhost lrmd[19255]: notice: operation_finished: reader_vip_2_start_0:20284 [ 2013/10/18_18:33:12 INFO: ip link set eth1 up ]</div><div>Oct 18 18:33:12 localhost lrmd[19255]: notice: operation_finished: reader_vip_2_start_0:20284 [ 2013/10/18_18:33:12 INFO: /usr/lib64/heartbeat/send_arp -i 200 -r 5 -p /var/run/heartbeat/rsctmp/send_arp-10.10.10.9 eth1 10.10.10.9</div>
<div> auto not_used not_used ]</div><div>Oct 18 18:33:13 localhost abrtd: Directory 'ccpp-2013-10-18-18:33:11-19253' creation detected</div><div>Oct 18 18:33:13 localhost lrmd[19255]: notice: operation_finished: reader_vip_1_start_0:20286 [ 2013/10/18_18:33:12 INFO: ip -f inet addr add <a href="http://10.10.10.8/32">10.10.10.8/32</a> brd 10.10.10.8 dev eth1 ]</div>
<div>Oct 18 18:33:13 localhost lrmd[19255]: notice: operation_finished: reader_vip_1_start_0:20286 [ 2013/10/18_18:33:12 INFO: ip link set eth1 up ]</div><div>Oct 18 18:33:13 localhost lrmd[19255]: notice: operation_finished: reader_vip_1_start_0:20286 [ 2013/10/18_18:33:12 INFO: /usr/lib64/heartbeat/send_arp -i 200 -r 5 -p /var/run/heartbeat/rsctmp/send_arp-10.10.10.8 eth1 10.10.10.8</div>
<div> auto not_used not_used ]</div><div>Oct 18 18:33:13 localhost abrt[20283]: Saved core dump of pid 19253 (/usr/libexec/pacemaker/cib) to /var/spool/abrt/ccpp-2013-10-18-18:33:11-19253 (10153984 bytes)</div><div>Oct 18 18:33:13 localhost rsyslogd-2177: imuxsock lost 75 messages from pid 18983 due to rate-limiting</div>
<div>Oct 18 18:33:13 localhost pacemakerd[19247]: notice: pcmk_child_exit: Child process cib terminated with signal 6 (pid=19253, core=128)</div><div>Oct 18 18:33:13 localhost pacemakerd[19247]: notice: pcmk_process_exit: Respawning failed child process: cib</div>
<div>Oct 18 18:33:13 localhost cib[20433]: notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log</div><div>Oct 18 18:33:13 localhost lrmd[19255]: notice: operation_finished: writer_vip_start_0:20285 [ 2013/10/18_18:33:12 INFO: ip -f inet addr add <a href="http://10.10.10.10/32">10.10.10.10/32</a> brd 10.10.10.10 dev eth1 ]</div>
<div>Oct 18 18:33:13 localhost lrmd[19255]: notice: operation_finished: writer_vip_start_0:20285 [ 2013/10/18_18:33:12 INFO: ip link set eth1 up ]</div><div>Oct 18 18:33:13 localhost lrmd[19255]: notice: operation_finished: writer_vip_start_0:20285 [ 2013/10/18_18:33:13 INFO: /usr/lib64/heartbeat/send_arp -i 200 -r 5 -p /var/run/heartbeat/rsctmp/send_arp-10.10.10.10 eth1 10.10.10.10</div>
<div> auto not_used not_used ]</div><div>Oct 18 18:33:13 localhost cib[20433]: notice: crm_cluster_connect: Connecting to cluster infrastructure: cman</div><div>Oct 18 18:33:13 localhost crmd[19258]: notice: crm_ipc_send: Connection to cib_shm closed</div>
<div>Oct 18 18:33:13 localhost crmd[19258]: error: cib_native_perform_op_delegate: Couldn't perform cib_modify operation (timeout=120s): -107: Transport endpoint is not connected (107)</div><div>Oct 18 18:33:13 localhost crmd[19258]: error: cib_native_perform_op_delegate: CIB disconnected</div>
<div>Oct 18 18:33:13 localhost crmd[19258]: warning: cib_rsc_callback: Resource update -70 failed: (rc=-70) Communication error on send</div><div>Oct 18 18:33:13 localhost crmd[19258]: notice: process_lrm_event: LRM operation reader_vip_2_start_0 (call=60, rc=0, cib-update=-70, confirmed=true) ok</div>
<div>Oct 18 18:33:13 localhost crmd[19258]: warning: cib_rsc_callback: Resource update -107 failed: (rc=-107) Transport endpoint is not connected</div><div>Oct 18 18:33:13 localhost crmd[19258]: notice: process_lrm_event: LRM operation reader_vip_1_start_0 (call=64, rc=0, cib-update=-107, confirmed=true) ok</div>
<div>Oct 18 18:33:13 localhost crmd[19258]: warning: cib_rsc_callback: Resource update -107 failed: (rc=-107) Transport endpoint is not connected</div><div>Oct 18 18:33:13 localhost crmd[19258]: notice: process_lrm_event: LRM operation writer_vip_start_0 (call=62, rc=0, cib-update=-107, confirmed=true) ok</div>
<div>Oct 18 18:33:13 localhost crmd[19258]: error: crm_ipc_read: Connection to cib_shm failed</div><div>Oct 18 18:33:13 localhost crmd[19258]: error: mainloop_gio_callback: Connection to cib_shm[0x26c7080] closed (I/O condition=17)</div>
<div>Oct 18 18:33:13 localhost crmd[19258]: error: crmd_cib_connection_destroy: Connection to the CIB terminated...</div><div>Oct 18 18:33:13 localhost crmd[19258]: error: do_log: FSA: Input I_ERROR from crmd_cib_connection_destroy() received in state S_TRANSITION_ENGINE</div>
<div>Oct 18 18:33:13 localhost crmd[19258]: warning: do_state_transition: State transition S_TRANSITION_ENGINE -> S_RECOVERY [ input=I_ERROR cause=C_FSA_INTERNAL origin=crmd_cib_connection_destroy ]</div><div>Oct 18 18:33:13 localhost crmd[19258]: error: do_recover: Action A_RECOVER (0000000001000000) not supported</div>
<div>Oct 18 18:33:13 localhost crmd[19258]: warning: do_election_vote: Not voting in election, we're in state S_RECOVERY</div><div>Oct 18 18:33:13 localhost crmd[19258]: warning: destroy_action: Cancelling timer for action 4 (src=71)</div>
<div>Oct 18 18:33:13 localhost crmd[19258]: warning: destroy_action: Cancelling timer for action 6 (src=72)</div><div>Oct 18 18:33:13 localhost crmd[19258]: warning: destroy_action: Cancelling timer for action 8 (src=73)</div>
<div>Oct 18 18:33:13 localhost crmd[19258]: error: do_log: FSA: Input I_TERMINATE from do_recover() received in state S_RECOVERY</div><div>Oct 18 18:33:13 localhost crmd[19258]: error: verify_stopped: Resource reader_vip_1 was active at shutdown. You may ignore this error if it is unmanaged.</div>
<div>Oct 18 18:33:13 localhost crmd[19258]: error: verify_stopped: Resource reader_vip_2 was active at shutdown. You may ignore this error if it is unmanaged.</div><div>Oct 18 18:33:13 localhost crmd[19258]: error: verify_stopped: Resource writer_vip was active at shutdown. You may ignore this error if it is unmanaged.</div>
<div>Oct 18 18:33:13 localhost crmd[19258]: notice: terminate_cs_connection: Disconnecting from Corosync</div><div>Oct 18 18:33:13 localhost crmd[19258]: error: do_exit: Could not recover from internal error</div></div>
<div><div>Oct 18 18:33:13 localhost pacemakerd[19247]: error: qb_ipcs_connection_unref: ref:0 state:3 (19247-19258-19)</div><div>Oct 18 18:33:14 localhost abrtd: Directory 'ccpp-2013-10-18-18:33:14-19247' creation detected</div>
<div>Oct 18 18:33:14 localhost abrt[20454]: Saved core dump of pid 19247 (/usr/sbin/pacemakerd) to /var/spool/abrt/ccpp-2013-10-18-18:33:14-19247 (9646080 bytes)</div><div>Oct 18 18:33:24 localhost abrtd: Sending an email...</div>
<div>Oct 18 18:33:24 localhost abrtd: Email was sent to: root@localhost</div><div>Oct 18 18:33:25 localhost abrtd: New problem directory /var/spool/abrt/ccpp-2013-10-18-18:33:11-19253, processing</div></div><div><br></div>
<div style>I get "[abrt] full crash report" emails as well. </div><div style><br></div><div style>some basic info from the emails</div><div style><br></div><div style><div>Subject: [abrt] full crash report</div>
<div>User-Agent: Heirloom mailx 12.4 7/29/08</div><div>Content-Type: text/plain; charset=us-ascii</div><div>Status: RO</div><div><br></div><div>abrt_version: 2.0.8</div><div>cgroup:</div><div>cmdline: /usr/libexec/pacemaker/cib</div>
<div>executable: /usr/libexec/pacemaker/cib</div><div>kernel: 2.6.32-358.18.1.el6.x86_64</div><div>pid: 19253</div><div>pwd: /var/lib/pacemaker/cores</div><div>time: Fri 18 Oct 2013 06:33:11 PM PDT</div>
<div>uid: 495</div><div>username: hacluster</div><div><br></div><div>sosreport.tar.xz: Binary file, 401160 bytes</div><div><br></div><div style>and</div></div><div style><br></div><div style><div><div>Subject: [abrt] full crash report</div>
<div>User-Agent: Heirloom mailx 12.4 7/29/08</div><div>Content-Type: text/plain; charset=us-ascii</div><div>Status: R</div><div><br></div><div>abrt_version: 2.0.8</div><div>cgroup:</div><div>cmdline: pacemakerd</div>
<div>executable: /usr/sbin/pacemakerd</div><div>kernel: 2.6.32-358.18.1.el6.x86_64</div><div>pid: 19247</div><div>pwd: /var/lib/pacemaker/cores/root</div><div>time: Fri 18 Oct 2013 06:33:14 PM PDT</div>
<div>uid: 0</div><div>username: root</div><div><br></div><div>sosreport.tar.xz: Binary file, 401964 bytes</div></div><div><br></div><div><br></div><div style>This is Centos 6.4 running the following versions:</div>
<div style><br></div><div style><div>corosync-1.4.1-15.el6_4.1.x86_64</div><div>pacemaker-libs-1.1.8-7.el6.x86_64</div><div>corosynclib-1.4.1-15.el6_4.1.x86_64</div><div>pacemaker-cli-1.1.8-7.el6.x86_64</div><div>pacemaker-cluster-libs-1.1.8-7.el6.x86_64</div>
<div>pacemaker-1.1.8-7.el6.x86_64</div><div><br></div><div style>Here is my corosync.conf file:</div><div style><br></div><div style><div>compatibility: whitetank</div><div><br></div><div>totem {</div><div> version: 2</div>
<div><br></div><div> rrp_mode: none</div><div> netmtu: 1500</div><div> secuath: on</div><div> threads: 0</div><div> transport: udp</div><div> token: 1000</div><div> token_retransmit: 238</div><div> hold: 100</div>
<div> join: 50</div><div> consensus: 1200</div><div> merge: 200</div><div> downcheck: 1000</div><div> fail_recv_const: 2500</div><div> seqno_unchanged_const: 30</div><div> heartbeat_failures_allowed: 0</div><div> max_network_delay: 50</div>
<div> window_size: 50</div><div> max_messages: 17</div><div> miss_count_const: 5</div><div> rrp_problem_count_timeout: 2000</div><div> rrp_problem_count_threshold: 10</div><div> rrp_problem_count_mcast_threshold:</div>
<div> rrp_token_expired_timeout: 47</div><div> rrp_autorecovery_check_timeout: 1000</div><div> interface: {</div><div> ringnumber: 0</div><div> bindnetaddr: 10.10.10.0</div><div> broadcast: no</div><div> mcastaddr: 239.0.0.111</div>
<div> mcastport: 4000</div><div> ttl: 1</div><div><div><br></div><div> }</div><div>}</div><div><br></div><div>logging {</div><div> timestamp: off</div><div> fileline: off</div><div> to_stderr: no</div><div> to_logfile: yes</div>
<div> to_syslog: yes</div><div><br></div><div> logfile: /var/log/corosync.log</div><div><br></div><div> logfile_priority: info</div><div> syslog_facility: daemon</div><div> syslog_priority: info</div><div> debug: off</div>
<div><br></div><div>}</div><div><br></div><div>amf {</div><div> mode: disabled</div><div>}</div></div><div><br></div><div style>and my cluster.conf</div><div style><br></div><div style><div><?xml version="1.0"?></div>
<div><cluster config_version="1" name="nodes"></div><div> <logging debug="on"/></div><div> <clusternodes></div><div> <clusternode name="node01" nodeid="1"/></div>
<div> <clusternode name="node02" nodeid="2"/></div><div> </clusternodes></div><div></cluster></div></div></div><div><br></div><div style>If I'm doing something obviously wrong please let me know. If you would like the full contents of the coredump and the abrt emails, I can also provide those.</div>
<div style><br></div><div style>Thanks for your time,</div><div style><br></div><div style>Justin</div></div></div></div></div>