<div><div>node node1</div><div>primitive FreeSWITCH ocf:fssolutions:FreeSWITCH \</div><div> params ips="bond2/<a href="http://212.163.22.155/26:bond2/212.163.22.156/26" target="_blank">212.163.22.155/26:bond2/212.163.22.156/26</a>" user="freeswitch" group="freeswitch" \</div>
<div> op monitor interval="3s" role="Master" depth="0" \</div><div> op monitor interval="10s" role="Slave" depth="0" \</div><div> op start interval="0" timeout="65" \</div>
<div> op stop interval="0" timeout="60"</div><div>ms FreeSWITCH-MS FreeSWITCH \</div><div> meta master-max="1" master-node-max="1" clone-max="1" clone-node-max="1" notify="false" target-role="Master"</div>
<div>location FreeSWITCH-MS-on-node1 FreeSWITCH-MS 50: node1</div><div>property $id="cib-bootstrap-options" \</div><div> dc-version="1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14" \</div>
<div> cluster-infrastructure="corosync" \</div><div> expected-quorum-votes="2" \</div><div> stonith-enabled="false" \</div><div> no-quorum-policy="ignore"</div>
<div>rsc_defaults $id="rsc-options" \</div><div> resource-stickiness="100"</div><div><br></div><div><br></div><div>============</div><div>Last updated: Sun Aug 26 16:13:05 2012</div><div>Last change: Sun Aug 26 16:11:18 2012 via crm_resource on node1</div>
<div>Stack: corosync</div><div>Current DC: node1 - partition WITHOUT quorum</div><div>Version: 1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14</div><div>1 Nodes configured, 2 expected votes</div><div>1 Resources configured.</div>
<div>============</div><div><br></div><div>Online: [ node1 ]</div><div><br></div><div> Master/Slave Set: FreeSWITCH-MS [FreeSWITCH]</div><div> Slaves: [ node1 ]</div><div><br></div><div><br></div><div>I run this: </div>
<div><br></div><div>crm resource promote FreeSWITCH-MS</div><div><br></div><div>Resulting log:</div><div><br></div><div>Aug 26 16:19:14 node1 cib[18166]: info: cib_process_request: Operation complete: op cib_modify for section resources (origin=local/crm_resource/4, version=0.7.2): ok (rc=0)</div>
<div><br></div><div><br></div><div>That's all it does. So I try:</div><div><br></div><div>crm resource demote FreeSWITCH-MS</div><div><br></div><div>And I get this:</div><div><br></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <cib admin_epoch="0" epoch="7" num_updates="2" ></div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <configuration ></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <resources ></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <master id="FreeSWITCH-MS" ></div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <meta_attributes id="FreeSWITCH-MS-meta_attributes" ></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <nvpair value="Master" id="FreeSWITCH-MS-meta_attributes-target-role" /></div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </meta_attributes></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </master></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </resources></div>
<div>Aug 26 16:20:24 node1 crmd[18171]: info: abort_transition_graph: te_update_diff:126 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.8.1) : Non-status change</div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </configuration></div>
<div>Aug 26 16:20:24 node1 crmd[18171]: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]</div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </cib></div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <cib epoch="8" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.6" update-origin="node1" update-client="crm_resource" cib-last-written="Sun Aug 26 16:11:18 2012" have-quorum="0" dc-uuid="node1" ></div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <configuration ></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <resources ></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <master id="FreeSWITCH-MS" ></div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <meta_attributes id="FreeSWITCH-MS-meta_attributes" ></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <nvpair id="FreeSWITCH-MS-meta_attributes-target-role" name="target-role" value="Slave" /></div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </meta_attributes></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </master></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </resources></div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </configuration></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </cib></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib_process_request: Operation complete: op cib_modify for section resources (origin=local/crm_resource/4, version=0.8.1): ok (rc=0)</div>
<div>Aug 26 16:20:24 node1 pengine[18170]: notice: unpack_config: On loss of CCM Quorum: Ignore</div><div>Aug 26 16:20:24 node1 crmd[18171]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]</div>
<div>Aug 26 16:20:24 node1 crmd[18171]: info: do_te_invoke: Processing graph 5 (ref=pe_calc-dc-1345994424-17) derived from /var/lib/pengine/pe-input-5.bz2</div><div>Aug 26 16:20:24 node1 crmd[18171]: notice: run_graph: ==== Transition 5 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-5.bz2): Complete</div>
<div>Aug 26 16:20:24 node1 crmd[18171]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]</div><div>Aug 26 16:20:24 node1 pengine[18170]: notice: process_pe_message: Transition 5: PEngine Input stored in: /var/lib/pengine/pe-input-5.bz2</div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <cib admin_epoch="0" epoch="7" num_updates="2" ></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <configuration ></div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <resources ></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <master id="FreeSWITCH-MS" ></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <meta_attributes id="FreeSWITCH-MS-meta_attributes" ></div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - <nvpair value="Master" id="FreeSWITCH-MS-meta_attributes-target-role" /></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </meta_attributes></div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </master></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </resources></div><div>Aug 26 16:20:24 node1 crmd[18171]: info: abort_transition_graph: te_update_diff:126 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.8.1) : Non-status change</div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </configuration></div><div>Aug 26 16:20:24 node1 crmd[18171]: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]</div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: - </cib></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <cib epoch="8" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.6" update-origin="node1" update-client="crm_resource" cib-last-written="Sun Aug 26 16:11:18 2012" have-quorum="0" dc-uuid="node1" ></div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <configuration ></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <resources ></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <master id="FreeSWITCH-MS" ></div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <meta_attributes id="FreeSWITCH-MS-meta_attributes" ></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + <nvpair id="FreeSWITCH-MS-meta_attributes-target-role" name="target-role" value="Slave" /></div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </meta_attributes></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </master></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </resources></div>
<div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </configuration></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib:diff: + </cib></div><div>Aug 26 16:20:24 node1 cib[18166]: info: cib_process_request: Operation complete: op cib_modify for section resources (origin=local/crm_resource/4, version=0.8.1): ok (rc=0)</div>
<div>Aug 26 16:20:24 node1 pengine[18170]: notice: unpack_config: On loss of CCM Quorum: Ignore</div><div>Aug 26 16:20:24 node1 crmd[18171]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]</div>
<div>Aug 26 16:20:24 node1 crmd[18171]: info: do_te_invoke: Processing graph 5 (ref=pe_calc-dc-1345994424-17) derived from /var/lib/pengine/pe-input-5.bz2</div><div>Aug 26 16:20:24 node1 crmd[18171]: notice: run_graph: ==== Transition 5 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-5.bz2): Complete</div>
<div>Aug 26 16:20:24 node1 crmd[18171]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]</div><div>Aug 26 16:20:24 node1 pengine[18170]: notice: process_pe_message: Transition 5: PEngine Input stored in: /var/lib/pengine/pe-input-5.bz2</div>
<div><br></div><div><br></div><div>Then I try:</div><div><br></div><div>crm resource promote FreeSWITCH-MS</div><div><br></div><div><br></div><div>This time I get:</div><div><br></div><div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - <cib admin_epoch="0" epoch="8" num_updates="1" ></div>
<div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - <configuration ></div><div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - <resources ></div><div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - <master id="FreeSWITCH-MS" ></div>
<div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - <meta_attributes id="FreeSWITCH-MS-meta_attributes" ></div><div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - <nvpair value="Slave" id="FreeSWITCH-MS-meta_attributes-target-role" /></div>
<div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - </meta_attributes></div><div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - </master></div><div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - </resources></div>
<div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - </configuration></div><div>Aug 26 16:21:39 node1 crmd[18171]: info: abort_transition_graph: te_update_diff:126 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.9.1) : Non-status change</div>
<div>Aug 26 16:21:39 node1 crmd[18171]: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]</div><div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: - </cib></div>
<div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + <cib epoch="9" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.6" update-origin="node1" update-client="crm_resource" cib-last-written="Sun Aug 26 16:20:24 2012" have-quorum="0" dc-uuid="node1" ></div>
<div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + <configuration ></div><div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + <resources ></div><div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + <master id="FreeSWITCH-MS" ></div>
<div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + <meta_attributes id="FreeSWITCH-MS-meta_attributes" ></div><div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + <nvpair id="FreeSWITCH-MS-meta_attributes-target-role" name="target-role" value="Master" /></div>
<div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + </meta_attributes></div><div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + </master></div><div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + </resources></div>
<div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + </configuration></div><div>Aug 26 16:21:39 node1 cib[18166]: info: cib:diff: + </cib></div><div>Aug 26 16:21:39 node1 cib[18166]: info: cib_process_request: Operation complete: op cib_modify for section resources (origin=local/crm_resource/4, version=0.9.1): ok (rc=0)</div>
<div>Aug 26 16:21:39 node1 pengine[18170]: notice: unpack_config: On loss of CCM Quorum: Ignore</div><div>Aug 26 16:21:39 node1 crmd[18171]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]</div>
<div>Aug 26 16:21:39 node1 crmd[18171]: info: do_te_invoke: Processing graph 6 (ref=pe_calc-dc-1345994499-18) derived from /var/lib/pengine/pe-input-6.bz2</div><div>Aug 26 16:21:39 node1 crmd[18171]: notice: run_graph: ==== Transition 6 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-6.bz2): Complete</div>
<div>Aug 26 16:21:39 node1 crmd[18171]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]</div><div>Aug 26 16:21:39 node1 pengine[18170]: notice: process_pe_message: Transition 6: PEngine Input stored in: /var/lib/pengine/pe-input-6.bz2</div>
<div><br></div><div>However, crm status shows:</div><div><br></div><div>============</div><div>Last updated: Sun Aug 26 16:22:04 2012</div><div>Last change: Sun Aug 26 16:21:39 2012 via crm_resource on node1</div><div>Stack: corosync</div>
<div>Current DC: node1 - partition WITHOUT quorum</div><div>Version: 1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14</div><div>1 Nodes configured, 2 expected votes</div><div>1 Resources configured.</div><div>============</div>
<div><br></div><div>Online: [ node1 ]</div><div><br></div><div> Master/Slave Set: FreeSWITCH-MS [FreeSWITCH]</div><div> Slaves: [ node1 ]</div><div><br></div><div><br></div><div>So, I thought maybe my resource agent (which I wrote myself) might be broken. So I ran ocf-tester on it again, but it passes all tests. Further, I don't actually get any errors in the log file. </div>
<div><br></div><div>I have also tried doing a resource cleanup on FreeSWITCH-MS, restarting pacemaker and corosync, putting the node in standby and bringing it back out, upgrading pacemaker and corosync (to the version you seen in the output in this E-mail from 1.1.6 previously where it first started happening) and also completely wiping out everything in the /var/lib/corosync, /var/lib/heartbeat, etc directories and recreating the entire config from scratch (which, incidentally, DID clear up a warning I was getting about a bad UUID), but nothing I have tried has resolved it. The odd thing is, this single-node setup has been working for nearly a year with this exact configuration. The box was rebooted last week, and now it just seems to sit there playing dead.</div>
<div><br></div><div>Thanks in advance for any suggestions on how I might solve this!</div><div>i</div></div><div><br></div>