[Pacemaker] Nodes not rejoining cluster
Gregg Stock
gregg at damagecontrolusa.com
Fri Mar 30 11:38:02 EDT 2012
I took the last 200 lines of each.
On 3/29/2012 10:15 PM, Andrew Beekhof wrote:
> Gotta have logs. From all 3 nodes mentioned.
> Only then can we determine if the problem is at the corosync or
> pacemaker layer - which is the pre-requisit for figuring out what to
> do next :)
>
> On Fri, Mar 30, 2012 at 1:30 PM, Gregg Stock<gregg at damagecontrolusa.com> wrote:
>> I had a circuit breaker go out and take two of the 5 nodes in my cluster
>> down. Now that their back up and running, they are not rejoining the
>> cluster.
>>
>> Here is what I get from crm_mon -1
>>
>> node 1,2 and 3 itchy, scratchy and walter show the following:
>> ============
>> Last updated: Thu Mar 29 19:04:05 2012
>> Last change: Thu Mar 29 19:04:03 2012 via cibadmin on walter
>> Stack: openais
>> Current DC: walter - partition with quorum
>> Version: 1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558
>> 5 Nodes configured, 5 expected votes
>> 9 Resources configured.
>> ============
>>
>> Online: [ itchy scratchy walter butthead timmy ]
>>
>>
>> On butthead I get
>>
>> ============
>> Last updated: Thu Mar 29 19:04:24 2012
>> Last change: Thu Mar 29 18:42:09 2012 via cibadmin on itchy
>> Stack: openais
>> Current DC: NONE
>> 5 Nodes configured, 5 expected votes
>> 9 Resources configured.
>> ============
>>
>> OFFLINE: [ itchy scratchy walter butthead timmy ]
>>
>>
>> On Timmy, I get
>>
>> ============
>> Last updated: Thu Mar 29 19:04:20 2012
>> Last change:
>> Current DC: NONE
>> 0 Nodes configured, unknown expected votes
>> 0 Resources configured.
>> ============
>>
>>
>> I don't have anything important running yet. so I can do a full clean up of
>> everything if needed.
>>
>> I also get some weird behavior with timmy. I brought this node up with the
>> host name as timmy.example.com and I changed the host name to timmy but when
>> the cluster is offline timmy.example.com shows up as offline. I enter crm
>> node delete timmy.example.com and it goes away until timmy goes offline
>> again.
>>
>> Thanks,
>> Gregg Stock
>>
>>
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
-------------- next part --------------
Mar 29 19:55:19 butthead crmd: [6658]: info: send_direct_ack: ACK'ing resource op p_lu_iscsivg01_lun2_delete_60000 from 0:0:crm-resource-28447: lrm_invoke-lrmd-1333076119-11
Mar 29 19:55:19 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1853) from walter: not in our membership
Mar 29 19:55:19 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_sync_one message (797) from itchy: not in our membership
Mar 29 19:57:19 butthead crmd: [6658]: WARN: xmlfromIPC: No message received in the required interval (120s)
Mar 29 19:57:19 butthead crmd: [6658]: WARN: cib_native_perform_op_delegate: Call failed: Remote node did not respond
Mar 29 19:57:19 butthead crmd: [6658]: ERROR: do_lrm_invoke: Attempt of deleting resource status 'p_ip_itchy-scratchy01' from CIB for 28447_crm_resource (user=unknown) on itchy failed: (rc=-41) Remote node did not respond
Mar 29 19:57:19 butthead crmd: [6658]: WARN: decode_transition_key: Bad UUID (crm-resource-28447) in sscanf result (3) for 0:0:crm-resource-28447
Mar 29 19:57:19 butthead crmd: [6658]: info: send_direct_ack: ACK'ing resource op p_ip_itchy-scratchy01_delete_60000 from 0:0:crm-resource-28447: lrm_invoke-lrmd-1333076239-12
Mar 29 19:57:19 butthead crmd: [6658]: info: ais_dispatch_message: Membership 434456: quorum retained
Mar 29 19:57:19 butthead crmd: [6658]: info: ais_status_callback: status: timmy is now lost (was member)
Mar 29 19:57:19 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=lost (new) addr=r(0) ip(10.24.88.5) votes=1 born=434440 seen=434452 proc=00000000000000000000000000111312
Mar 29 20:03:23 butthead cib: [6654]: info: cib_stats: Processed 2 operations (0.00us average, 0% utilization) in the last 10min
Mar 29 20:03:30 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 29 20:03:36 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 29 21:03:32 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 29 21:03:45 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 29 22:03:33 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 29 22:03:54 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 29 23:03:35 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 29 23:04:03 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 00:03:37 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 00:04:12 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 01:03:38 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 01:04:21 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 02:03:40 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 02:04:30 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 03:03:42 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 03:04:39 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 04:03:43 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 04:04:48 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 05:03:45 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 05:04:57 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 06:03:47 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 06:05:05 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 07:03:48 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 07:05:14 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 08:03:50 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 08:05:23 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 08:13:23 butthead cib: [6654]: info: cib_stats: Processed 1 operations (0.00us average, 0% utilization) in the last 10min
Mar 30 08:17:23 corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 434460: memb=4, new=0, lost=0
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: memb: itchy 22550538
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: memb: scratchy 39327754
Mar 30 08:17:23 butthead crmd: [6658]: info: ais_dispatch_message: Membership 434460: quorum retained
Mar 30 08:17:23 butthead cib: [6654]: info: ais_dispatch_message: Membership 434460: quorum retained
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: memb: walter 56104970
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: memb: butthead 72882186
Mar 30 08:17:23 corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 434460: memb=5, new=1, lost=0
Mar 30 08:17:23 corosync [pcmk ] info: update_member: Node 89659402/timmy is now: member
Mar 30 08:17:23 butthead crmd: [6658]: info: ais_status_callback: status: timmy is now member (was lost)
Mar 30 08:17:23 butthead cib: [6654]: info: crm_update_peer: Node timmy: id=89659402 state=member (new) addr=r(0) ip(10.24.88.5) votes=1 born=434440 seen=434460 proc=00000000000000000000000000111312
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: NEW: timmy 89659402
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: MEMB: itchy 22550538
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: MEMB: scratchy 39327754
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: MEMB: walter 56104970
Mar 30 08:17:23 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=member (new) addr=r(0) ip(10.24.88.5) votes=1 born=434440 seen=434460 proc=00000000000000000000000000111312
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: MEMB: butthead 72882186
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: MEMB: timmy 89659402
Mar 30 08:17:23 corosync [pcmk ] info: send_member_notification: Sending membership update 434460 to 2 children
Mar 30 08:17:23 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 30 08:17:23 corosync [pcmk ] info: update_member: 0x21bbcf0 Node 89659402 (timmy) born on: 434460
Mar 30 08:17:23 corosync [pcmk ] info: send_member_notification: Sending membership update 434460 to 2 children
Mar 30 08:17:23 butthead cib: [6654]: info: ais_dispatch_message: Membership 434460: quorum retained
Mar 30 08:17:23 butthead crmd: [6658]: info: ais_dispatch_message: Membership 434460: quorum retained
Mar 30 08:17:23 corosync [CPG ] chosen downlist: sender r(0) ip(10.24.88.1) ; members(old:4 left:0)
Mar 30 08:17:23 corosync [MAIN ] Completed service synchronization, ready to provide service.
Mar 30 08:17:23 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1858) from walter: not in our membership
Mar 30 08:17:23 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1859) from walter: not in our membership
Mar 30 08:17:23 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1860) from walter: not in our membership
Mar 30 08:17:23 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1861) from walter: not in our membership
Mar 30 08:17:23 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1862) from walter: not in our membership
Mar 30 08:17:23 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1863) from walter: not in our membership
Mar 30 08:17:23 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1864) from walter: not in our membership
Mar 30 08:17:23 butthead crmd: [6658]: info: do_state_transition: State transition S_NOT_DC -> S_PENDING [ input=I_JOIN_OFFER cause=C_HA_MESSAGE origin=route_message ]
Mar 30 08:17:23 butthead crmd: [6658]: info: update_dc: Unset DC walter
Mar 30 08:17:23 butthead crmd: [6658]: info: update_dc: Set DC to walter (3.0.5)
Mar 30 08:17:23 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1865) from walter: not in our membership
Mar 30 08:17:53 butthead pacemakerd: [6649]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000000002 (was 00000000000000000000000000111312)
Mar 30 08:17:53 butthead stonith-ng: [6653]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000000002 (new)
Mar 30 08:17:53 butthead cib: [6654]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000000002 (new)
Mar 30 08:17:53 butthead pacemakerd: [6649]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000100002 (was 00000000000000000000000000000002)
Mar 30 08:17:53 butthead crmd: [6658]: notice: crmd_peer_update: Status update: Client timmy/crmd now has status [offline] (DC=walter)
Mar 30 08:17:53 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000000002 (new)
Mar 30 08:17:53 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100002 (new)
Mar 30 08:17:53 butthead stonith-ng: [6653]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000100002 (new)
Mar 30 08:17:53 butthead cib: [6654]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100002 (new)
Mar 30 08:17:53 butthead pacemakerd: [6649]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000100102 (was 00000000000000000000000000100002)
Mar 30 08:17:53 butthead stonith-ng: [6653]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000100102 (new)
Mar 30 08:17:53 butthead cib: [6654]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100102 (new)
Mar 30 08:17:53 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100102 (new)
Mar 30 08:17:53 butthead pacemakerd: [6649]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000100112 (was 00000000000000000000000000100102)
Mar 30 08:17:53 butthead stonith-ng: [6653]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000100112 (new)
Mar 30 08:17:53 butthead cib: [6654]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100112 (new)
Mar 30 08:17:53 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100112 (new)
Mar 30 08:17:53 butthead pacemakerd: [6649]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000101112 (was 00000000000000000000000000100112)
Mar 30 08:17:53 butthead stonith-ng: [6653]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000101112 (new)
Mar 30 08:17:53 butthead cib: [6654]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000101112 (new)
Mar 30 08:17:53 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000101112 (new)
Mar 30 08:17:53 butthead pacemakerd: [6649]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000111112 (was 00000000000000000000000000101112)
Mar 30 08:17:53 butthead stonith-ng: [6653]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000111112 (new)
Mar 30 08:17:53 butthead cib: [6654]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111112 (new)
Mar 30 08:17:53 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111112 (new)
Mar 30 08:17:53 butthead pacemakerd: [6649]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000111312 (was 00000000000000000000000000111112)
Mar 30 08:17:53 butthead stonith-ng: [6653]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:53 butthead cib: [6654]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:53 butthead crmd: [6658]: notice: crmd_peer_update: Status update: Client timmy/crmd now has status [online] (DC=walter)
Mar 30 08:17:53 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:53 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1866) from walter: not in our membership
Mar 30 08:17:53 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_replace message (1867) from walter: not in our membership
Mar 30 08:17:53 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1868) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_replace message (1869) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1870) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1871) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1872) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1873) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1874) from walter: not in our membership
Mar 30 08:17:56 butthead attrd: [6656]: notice: attrd_local_callback: Sending full refresh (origin=crmd)
Mar 30 08:17:56 butthead crmd: [6658]: info: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Mar 30 08:17:56 butthead attrd: [6656]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 30 08:17:56 butthead attrd: [6656]: notice: attrd_perform_update: Sent update 108: probe_complete=true
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1875) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1876) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1877) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1878) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1879) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1880) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1881) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1882) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1883) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1884) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1885) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1886) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1887) from walter: not in our membership
Mar 30 08:17:56 butthead attrd: [6656]: notice: attrd_perform_update: Sent update 111: probe_complete=true
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1888) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1889) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1890) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1891) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1892) from walter: not in our membership
Mar 30 08:17:56 butthead attrd: [6656]: notice: attrd_perform_update: Sent update 116: probe_complete=true
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1893) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1894) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1895) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1896) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1897) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1898) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1899) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (798) from itchy: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (799) from itchy: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1900) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1901) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1902) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (800) from itchy: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (801) from itchy: not in our membership
Mar 30 08:17:56 butthead attrd: [6656]: notice: attrd_perform_update: Sent update 122: probe_complete=true
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (802) from itchy: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (803) from itchy: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (804) from itchy: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1903) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1904) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1905) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1906) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1907) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1908) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1909) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1910) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1911) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1912) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1913) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1914) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1915) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1916) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1917) from walter: not in our membership
Mar 30 08:17:57 butthead attrd: [6656]: notice: attrd_perform_update: Sent update 125: probe_complete=true
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (805) from itchy: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1918) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1919) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1920) from walter: not in our membership
Mar 30 08:17:57 butthead attrd: [6656]: notice: attrd_perform_update: Sent update 128: probe_complete=true
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1921) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1922) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1923) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (806) from itchy: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1924) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1925) from walter: not in our membership
Mar 30 08:17:58 butthead attrd: [6656]: notice: attrd_perform_update: Sent update 131: probe_complete=true
Mar 30 08:17:58 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (807) from itchy: not in our membership
Mar 30 08:17:58 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1926) from walter: not in our membership
Mar 30 08:17:58 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1927) from walter: not in our membership
Mar 30 08:17:58 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1928) from walter: not in our membership
Mar 30 08:17:58 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1929) from walter: not in our membership
Mar 30 08:17:58 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1930) from walter: not in our membership
Mar 30 08:19:56 butthead attrd: [6656]: WARN: attrd_cib_callback: Update 108 for probe_complete=true failed: Remote node did not respond
Mar 30 08:19:56 butthead attrd: [6656]: WARN: attrd_cib_callback: Update 111 for probe_complete=true failed: Remote node did not respond
Mar 30 08:19:56 butthead attrd: [6656]: WARN: attrd_cib_callback: Update 116 for probe_complete=true failed: Remote node did not respond
Mar 30 08:19:56 butthead attrd: [6656]: WARN: attrd_cib_callback: Update 122 for probe_complete=true failed: Remote node did not respond
Mar 30 08:19:57 butthead attrd: [6656]: WARN: attrd_cib_callback: Update 125 for probe_complete=true failed: Remote node did not respond
Mar 30 08:19:57 butthead attrd: [6656]: WARN: attrd_cib_callback: Update 128 for probe_complete=true failed: Remote node did not respond
Mar 30 08:19:58 butthead attrd: [6656]: WARN: attrd_cib_callback: Update 131 for probe_complete=true failed: Remote node did not respond
Mar 30 08:23:23 butthead cib: [6654]: info: cib_stats: Processed 48 operations (208.00us average, 0% utilization) in the last 10min
Mar 30 08:33:23 butthead cib: [6654]: info: cib_stats: Processed 1 operations (0.00us average, 0% utilization) in the last 10min
-------------- next part --------------
Mar 29 19:55:19 butthead crmd: [6658]: WARN: decode_transition_key: Bad UUID (crm-resource-28447) in sscanf result (3) for 0:0:crm-resource-28447
Mar 29 19:55:19 butthead crmd: [6658]: info: send_direct_ack: ACK'ing resource op p_lu_iscsivg01_lun2_delete_60000 from 0:0:crm-resource-28447: lrm_invoke-lrmd-1333076119-11
Mar 29 19:55:19 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1853) from walter: not in our membership
Mar 29 19:55:19 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_sync_one message (797) from itchy: not in our membership
Mar 29 19:57:19 butthead crmd: [6658]: WARN: xmlfromIPC: No message received in the required interval (120s)
Mar 29 19:57:19 butthead crmd: [6658]: WARN: cib_native_perform_op_delegate: Call failed: Remote node did not respond
Mar 29 19:57:19 butthead crmd: [6658]: ERROR: do_lrm_invoke: Attempt of deleting resource status 'p_ip_itchy-scratchy01' from CIB for 28447_crm_resource (user=unknown) on itchy failed: (rc=-41) Remote node did not respond
Mar 29 19:57:19 butthead crmd: [6658]: WARN: decode_transition_key: Bad UUID (crm-resource-28447) in sscanf result (3) for 0:0:crm-resource-28447
Mar 29 19:57:19 butthead crmd: [6658]: info: send_direct_ack: ACK'ing resource op p_ip_itchy-scratchy01_delete_60000 from 0:0:crm-resource-28447: lrm_invoke-lrmd-1333076239-12
Mar 29 19:57:19 butthead crmd: [6658]: info: ais_dispatch_message: Membership 434456: quorum retained
Mar 29 19:57:19 butthead crmd: [6658]: info: ais_status_callback: status: timmy is now lost (was member)
Mar 29 19:57:19 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=lost (new) addr=r(0) ip(10.24.88.5) votes=1 born=434440 seen=434452 proc=00000000000000000000000000111312
Mar 29 20:03:23 butthead cib: [6654]: info: cib_stats: Processed 2 operations (0.00us average, 0% utilization) in the last 10min
Mar 29 20:03:30 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 29 20:03:36 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 29 21:03:32 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 29 21:03:45 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 29 22:03:33 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 29 22:03:54 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 29 23:03:35 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 29 23:04:03 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 00:03:37 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 00:04:12 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 01:03:38 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 01:04:21 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 02:03:40 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 02:04:30 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 03:03:42 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 03:04:39 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 04:03:43 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 04:04:48 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 05:03:45 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 05:04:57 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 06:03:47 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 06:05:05 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 07:03:48 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 07:05:14 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 08:03:50 butthead lrmd: [6655]: info: rsc:p_dnsmasq:13: monitor
Mar 30 08:05:23 butthead lrmd: [6655]: info: rsc:p_ip_dnsmasq:11: monitor
Mar 30 08:13:23 butthead cib: [6654]: info: cib_stats: Processed 1 operations (0.00us average, 0% utilization) in the last 10min
Mar 30 08:17:23 butthead corosync[2279]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 434460: memb=4, new=0, lost=0
Mar 30 08:17:23 butthead cib: [6654]: info: ais_dispatch_message: Membership 434460: quorum retained
Mar 30 08:17:23 butthead crmd: [6658]: info: ais_dispatch_message: Membership 434460: quorum retained
Mar 30 08:17:23 butthead corosync[2279]: [pcmk ] info: pcmk_peer_update: memb: itchy 22550538
Mar 30 08:17:23 butthead corosync[2279]: [pcmk ] info: pcmk_peer_update: memb: scratchy 39327754
Mar 30 08:17:23 butthead corosync[2279]: [pcmk ] info: pcmk_peer_update: memb: walter 56104970
Mar 30 08:17:23 butthead corosync[2279]: [pcmk ] info: pcmk_peer_update: memb: butthead 72882186
Mar 30 08:17:23 butthead crmd: [6658]: info: ais_status_callback: status: timmy is now member (was lost)
Mar 30 08:17:23 butthead corosync[2279]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 434460: memb=5, new=1, lost=0
Mar 30 08:17:23 butthead cib: [6654]: info: crm_update_peer: Node timmy: id=89659402 state=member (new) addr=r(0) ip(10.24.88.5) votes=1 born=434440 seen=434460 proc=00000000000000000000000000111312
Mar 30 08:17:23 butthead corosync[2279]: [pcmk ] info: update_member: Node 89659402/timmy is now: member
Mar 30 08:17:23 butthead corosync[2279]: [pcmk ] info: pcmk_peer_update: NEW: timmy 89659402
Mar 30 08:17:23 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=member (new) addr=r(0) ip(10.24.88.5) votes=1 born=434440 seen=434460 proc=00000000000000000000000000111312
Mar 30 08:17:23 butthead corosync[2279]: [pcmk ] info: pcmk_peer_update: MEMB: itchy 22550538
Mar 30 08:17:23 butthead corosync[2279]: [pcmk ] info: pcmk_peer_update: MEMB: scratchy 39327754
Mar 30 08:17:23 butthead corosync[2279]: [pcmk ] info: pcmk_peer_update: MEMB: walter 56104970
Mar 30 08:17:23 butthead corosync[2279]: [pcmk ] info: pcmk_peer_update: MEMB: butthead 72882186
Mar 30 08:17:23 butthead corosync[2279]: [pcmk ] info: pcmk_peer_update: MEMB: timmy 89659402
Mar 30 08:17:23 butthead corosync[2279]: [pcmk ] info: send_member_notification: Sending membership update 434460 to 2 children
Mar 30 08:17:23 butthead corosync[2279]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 30 08:17:23 butthead corosync[2279]: [pcmk ] info: update_member: 0x21bbcf0 Node 89659402 (timmy) born on: 434460
Mar 30 08:17:23 butthead corosync[2279]: [pcmk ] info: send_member_notification: Sending membership update 434460 to 2 children
Mar 30 08:17:23 butthead cib: [6654]: info: ais_dispatch_message: Membership 434460: quorum retained
Mar 30 08:17:23 butthead crmd: [6658]: info: ais_dispatch_message: Membership 434460: quorum retained
Mar 30 08:17:23 butthead corosync[2279]: [CPG ] chosen downlist: sender r(0) ip(10.24.88.1) ; members(old:4 left:0)
Mar 30 08:17:23 butthead corosync[2279]: [MAIN ] Completed service synchronization, ready to provide service.
Mar 30 08:17:23 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1858) from walter: not in our membership
Mar 30 08:17:23 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1859) from walter: not in our membership
Mar 30 08:17:23 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1860) from walter: not in our membership
Mar 30 08:17:23 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1861) from walter: not in our membership
Mar 30 08:17:23 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1862) from walter: not in our membership
Mar 30 08:17:23 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1863) from walter: not in our membership
Mar 30 08:17:23 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1864) from walter: not in our membership
Mar 30 08:17:23 butthead crmd: [6658]: info: do_state_transition: State transition S_NOT_DC -> S_PENDING [ input=I_JOIN_OFFER cause=C_HA_MESSAGE origin=route_message ]
Mar 30 08:17:23 butthead crmd: [6658]: info: update_dc: Unset DC walter
Mar 30 08:17:23 butthead crmd: [6658]: info: update_dc: Set DC to walter (3.0.5)
Mar 30 08:17:23 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1865) from walter: not in our membership
Mar 30 08:17:53 butthead pacemakerd: [6649]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000000002 (was 00000000000000000000000000111312)
Mar 30 08:17:53 butthead stonith-ng: [6653]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000000002 (new)
Mar 30 08:17:53 butthead cib: [6654]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000000002 (new)
Mar 30 08:17:53 butthead pacemakerd: [6649]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000100002 (was 00000000000000000000000000000002)
Mar 30 08:17:53 butthead crmd: [6658]: notice: crmd_peer_update: Status update: Client timmy/crmd now has status [offline] (DC=walter)
Mar 30 08:17:53 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000000002 (new)
Mar 30 08:17:53 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100002 (new)
Mar 30 08:17:53 butthead stonith-ng: [6653]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000100002 (new)
Mar 30 08:17:53 butthead cib: [6654]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100002 (new)
Mar 30 08:17:53 butthead pacemakerd: [6649]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000100102 (was 00000000000000000000000000100002)
Mar 30 08:17:53 butthead stonith-ng: [6653]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000100102 (new)
Mar 30 08:17:53 butthead cib: [6654]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100102 (new)
Mar 30 08:17:53 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100102 (new)
Mar 30 08:17:53 butthead pacemakerd: [6649]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000100112 (was 00000000000000000000000000100102)
Mar 30 08:17:53 butthead stonith-ng: [6653]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000100112 (new)
Mar 30 08:17:53 butthead cib: [6654]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100112 (new)
Mar 30 08:17:53 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100112 (new)
Mar 30 08:17:53 butthead pacemakerd: [6649]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000101112 (was 00000000000000000000000000100112)
Mar 30 08:17:53 butthead stonith-ng: [6653]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000101112 (new)
Mar 30 08:17:53 butthead cib: [6654]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000101112 (new)
Mar 30 08:17:53 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000101112 (new)
Mar 30 08:17:53 butthead pacemakerd: [6649]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000111112 (was 00000000000000000000000000101112)
Mar 30 08:17:53 butthead stonith-ng: [6653]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000111112 (new)
Mar 30 08:17:53 butthead cib: [6654]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111112 (new)
Mar 30 08:17:53 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111112 (new)
Mar 30 08:17:53 butthead pacemakerd: [6649]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000111312 (was 00000000000000000000000000111112)
Mar 30 08:17:53 butthead stonith-ng: [6653]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:53 butthead cib: [6654]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:53 butthead crmd: [6658]: notice: crmd_peer_update: Status update: Client timmy/crmd now has status [online] (DC=walter)
Mar 30 08:17:53 butthead crmd: [6658]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:53 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1866) from walter: not in our membership
Mar 30 08:17:53 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_replace message (1867) from walter: not in our membership
Mar 30 08:17:53 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1868) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_replace message (1869) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1870) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1871) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1872) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1873) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1874) from walter: not in our membership
Mar 30 08:17:56 butthead attrd: [6656]: notice: attrd_local_callback: Sending full refresh (origin=crmd)
Mar 30 08:17:56 butthead crmd: [6658]: info: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Mar 30 08:17:56 butthead attrd: [6656]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 30 08:17:56 butthead attrd: [6656]: notice: attrd_perform_update: Sent update 108: probe_complete=true
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1875) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1876) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1877) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1878) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1879) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1880) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1881) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1882) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1883) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1884) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1885) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1886) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1887) from walter: not in our membership
Mar 30 08:17:56 butthead attrd: [6656]: notice: attrd_perform_update: Sent update 111: probe_complete=true
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1888) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1889) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1890) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1891) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1892) from walter: not in our membership
Mar 30 08:17:56 butthead attrd: [6656]: notice: attrd_perform_update: Sent update 116: probe_complete=true
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1893) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1894) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1895) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1896) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1897) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1898) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1899) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (798) from itchy: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (799) from itchy: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1900) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1901) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1902) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (800) from itchy: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (801) from itchy: not in our membership
Mar 30 08:17:56 butthead attrd: [6656]: notice: attrd_perform_update: Sent update 122: probe_complete=true
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (802) from itchy: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (803) from itchy: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (804) from itchy: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1903) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1904) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1905) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1906) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1907) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1908) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1909) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1910) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1911) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1912) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1913) from walter: not in our membership
Mar 30 08:17:56 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1914) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1915) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1916) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1917) from walter: not in our membership
Mar 30 08:17:57 butthead attrd: [6656]: notice: attrd_perform_update: Sent update 125: probe_complete=true
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (805) from itchy: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1918) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1919) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1920) from walter: not in our membership
Mar 30 08:17:57 butthead attrd: [6656]: notice: attrd_perform_update: Sent update 128: probe_complete=true
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1921) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1922) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1923) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (806) from itchy: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1924) from walter: not in our membership
Mar 30 08:17:57 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1925) from walter: not in our membership
Mar 30 08:17:58 butthead attrd: [6656]: notice: attrd_perform_update: Sent update 131: probe_complete=true
Mar 30 08:17:58 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_modify message (807) from itchy: not in our membership
Mar 30 08:17:58 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1926) from walter: not in our membership
Mar 30 08:17:58 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1927) from walter: not in our membership
Mar 30 08:17:58 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1928) from walter: not in our membership
Mar 30 08:17:58 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1929) from walter: not in our membership
Mar 30 08:17:58 butthead cib: [6654]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1930) from walter: not in our membership
Mar 30 08:19:56 butthead attrd: [6656]: WARN: attrd_cib_callback: Update 108 for probe_complete=true failed: Remote node did not respond
Mar 30 08:19:56 butthead attrd: [6656]: WARN: attrd_cib_callback: Update 111 for probe_complete=true failed: Remote node did not respond
Mar 30 08:19:56 butthead attrd: [6656]: WARN: attrd_cib_callback: Update 116 for probe_complete=true failed: Remote node did not respond
Mar 30 08:19:56 butthead attrd: [6656]: WARN: attrd_cib_callback: Update 122 for probe_complete=true failed: Remote node did not respond
Mar 30 08:19:57 butthead attrd: [6656]: WARN: attrd_cib_callback: Update 125 for probe_complete=true failed: Remote node did not respond
Mar 30 08:19:57 butthead attrd: [6656]: WARN: attrd_cib_callback: Update 128 for probe_complete=true failed: Remote node did not respond
Mar 30 08:19:58 butthead attrd: [6656]: WARN: attrd_cib_callback: Update 131 for probe_complete=true failed: Remote node did not respond
Mar 30 08:23:23 butthead cib: [6654]: info: cib_stats: Processed 48 operations (208.00us average, 0% utilization) in the last 10min
-------------- next part --------------
Mar 29 18:58:28 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 29 18:58:28 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 29 18:58:55 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 29 18:58:55 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 29 18:58:55 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 29 18:58:55 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 29 18:59:57 corosync [TOTEM ] A processor failed, forming new configuration.
Mar 29 18:59:59 corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 434444: memb=4, new=0, lost=1
Mar 29 18:59:59 corosync [pcmk ] info: pcmk_peer_update: memb: itchy 22550538
Mar 29 18:59:59 corosync [pcmk ] info: pcmk_peer_update: memb: scratchy 39327754
Mar 29 18:59:59 corosync [pcmk ] info: pcmk_peer_update: memb: walter 56104970
Mar 29 18:59:59 corosync [pcmk ] info: pcmk_peer_update: memb: timmy 89659402
Mar 29 18:59:59 corosync [pcmk ] info: pcmk_peer_update: lost: butthead 72882186
Mar 29 18:59:59 corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 434444: memb=4, new=0, lost=0
Mar 29 18:59:59 corosync [pcmk ] info: pcmk_peer_update: MEMB: itchy 22550538
Mar 29 18:59:59 corosync [pcmk ] info: pcmk_peer_update: MEMB: scratchy 39327754
Mar 29 18:59:59 corosync [pcmk ] info: pcmk_peer_update: MEMB: walter 56104970
Mar 29 18:59:59 corosync [pcmk ] info: pcmk_peer_update: MEMB: timmy 89659402
Mar 29 18:59:59 corosync [pcmk ] info: ais_mark_unseen_peer_dead: Node butthead was not seen in the previous transition
Mar 29 18:59:59 corosync [pcmk ] info: update_member: Node 72882186/butthead is now: lost
Mar 29 18:59:59 corosync [pcmk ] info: send_member_notification: Sending membership update 434444 to 0 children
Mar 29 18:59:59 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 29 18:59:59 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 29 18:59:59 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 29 18:59:59 corosync [CPG ] chosen downlist: sender r(0) ip(10.24.88.1) ; members(old:5 left:1)
Mar 29 18:59:59 corosync [MAIN ] Completed service synchronization, ready to provide service.
Mar 29 18:59:59 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 29 18:59:59 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 29 18:59:59 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 29 18:59:59 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 29 19:00:34 corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 434452: memb=4, new=0, lost=0
Mar 29 19:00:34 corosync [pcmk ] info: pcmk_peer_update: memb: itchy 22550538
Mar 29 19:00:34 corosync [pcmk ] info: pcmk_peer_update: memb: scratchy 39327754
Mar 29 19:00:34 corosync [pcmk ] info: pcmk_peer_update: memb: walter 56104970
Mar 29 19:00:34 corosync [pcmk ] info: pcmk_peer_update: memb: timmy 89659402
Mar 29 19:00:34 corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 434452: memb=5, new=1, lost=0
Mar 29 19:00:34 corosync [pcmk ] info: update_member: Node 72882186/butthead is now: member
Mar 29 19:00:34 corosync [pcmk ] info: pcmk_peer_update: NEW: butthead 72882186
Mar 29 19:00:34 corosync [pcmk ] info: pcmk_peer_update: MEMB: itchy 22550538
Mar 29 19:00:34 corosync [pcmk ] info: pcmk_peer_update: MEMB: scratchy 39327754
Mar 29 19:00:34 corosync [pcmk ] info: pcmk_peer_update: MEMB: walter 56104970
Mar 29 19:00:34 corosync [pcmk ] info: pcmk_peer_update: MEMB: butthead 72882186
Mar 29 19:00:34 corosync [pcmk ] info: pcmk_peer_update: MEMB: timmy 89659402
Mar 29 19:00:34 corosync [pcmk ] info: send_member_notification: Sending membership update 434452 to 0 children
Mar 29 19:00:34 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 29 19:00:34 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 29 19:00:34 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 29 19:00:34 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 29 19:00:34 corosync [CPG ] chosen downlist: sender r(0) ip(10.24.88.1) ; members(old:4 left:0)
Mar 29 19:00:34 corosync [MAIN ] Completed service synchronization, ready to provide service.
Mar 29 19:00:34 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 29 19:00:34 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 29 19:00:34 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 29 19:00:34 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 29 19:00:34 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 29 19:02:58 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 29 19:02:58 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 29 19:02:58 corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x1e0c770 for stonith-ng/0
Mar 29 19:02:58 corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x1e10ad0 for attrd/0
Mar 29 19:02:58 corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x1e15160 for cib/0
Mar 29 19:02:58 corosync [pcmk ] info: pcmk_ipc: Sending membership update 434452 to cib
Mar 29 19:02:59 corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x1e19dc0 for crmd/0
Mar 29 19:02:59 corosync [pcmk ] info: pcmk_ipc: Sending membership update 434452 to crmd
Mar 29 19:02:59 corosync [pcmk ] info: update_expected_votes: Cannot update expected quorum votes 5 -> 2: value cannot be less that the current number of votes
Mar 29 19:49:52 corosync [pcmk ] info: pcmk_ipc_exit: Client attrd (conn=0x1e10ad0, async-conn=0x1e10ad0) left
Mar 29 19:49:52 corosync [pcmk ] info: pcmk_ipc_exit: Client cib (conn=0x1e15160, async-conn=0x1e15160) left
Mar 29 19:49:52 corosync [pcmk ] info: pcmk_ipc_exit: Client stonith-ng (conn=0x1e0c770, async-conn=0x1e0c770) left
Mar 29 19:49:52 corosync [pcmk ] info: pcmk_ipc_exit: Client crmd (conn=0x1e19dc0, async-conn=0x1e19dc0) left
Mar 29 19:50:04 corosync [SERV ] Unloading all Corosync service engines.
Mar 29 19:50:04 corosync [pcmk ] notice: pcmk_shutdown: Unloading Pacemaker plugin
Mar 29 19:50:04 corosync [SERV ] Service engine unloaded: Pacemaker Cluster Manager 1.1.6
Mar 29 19:50:04 corosync [SERV ] Service engine unloaded: corosync extended virtual synchrony service
Mar 29 19:50:04 corosync [SERV ] Service engine unloaded: corosync configuration service
Mar 29 19:50:04 corosync [SERV ] Service engine unloaded: corosync cluster closed process group service v1.01
Mar 29 19:50:04 corosync [SERV ] Service engine unloaded: corosync cluster config database access v1.01
Mar 29 19:50:04 corosync [SERV ] Service engine unloaded: corosync profile loading service
Mar 29 19:50:04 corosync [SERV ] Service engine unloaded: corosync cluster quorum service v0.1
Mar 29 19:50:04 corosync [MAIN ] Corosync Cluster Engine exiting with status 0 at main.c:1858.
Mar 30 08:17:23 corosync [MAIN ] Corosync Cluster Engine ('1.4.1'): started and ready to provide service.
Mar 30 08:17:23 corosync [MAIN ] Corosync built-in features: nss dbus rdma snmp
Mar 30 08:17:23 corosync [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
Mar 30 08:17:23 corosync [TOTEM ] Initializing transport (UDP/IP Multicast).
Mar 30 08:17:23 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Mar 30 08:17:23 corosync [TOTEM ] The network interface [10.24.88.5] is now up.
Mar 30 08:17:23 corosync [pcmk ] info: process_ais_conf: Reading configure
Mar 30 08:17:23 corosync [pcmk ] info: config_find_init: Local handle: 2013064636357672963 for logging
Mar 30 08:17:23 corosync [pcmk ] info: config_find_next: Processing additional logging options...
Mar 30 08:17:23 corosync [pcmk ] info: get_config_opt: Found 'off' for option: debug
Mar 30 08:17:23 corosync [pcmk ] info: get_config_opt: Found 'yes' for option: to_logfile
Set r/w permissions for uid=0, gid=0 on /var/log/cluster/corosync.log
Mar 30 08:17:23 corosync [pcmk ] info: get_config_opt: Found '/var/log/cluster/corosync.log' for option: logfile
Mar 30 08:17:23 corosync [pcmk ] info: get_config_opt: Found 'yes' for option: to_syslog
Mar 30 08:17:23 corosync [pcmk ] info: get_config_opt: Defaulting to 'daemon' for option: syslog_facility
Mar 30 08:17:23 corosync [pcmk ] info: config_find_init: Local handle: 4730966301143465988 for quorum
Mar 30 08:17:23 corosync [pcmk ] info: config_find_next: No additional configuration supplied for: quorum
Mar 30 08:17:23 corosync [pcmk ] info: get_config_opt: No default for option: provider
Mar 30 08:17:23 corosync [pcmk ] info: config_find_init: Local handle: 7739444317642555397 for service
Mar 30 08:17:23 corosync [pcmk ] info: config_find_next: Processing additional service options...
Mar 30 08:17:23 corosync [pcmk ] info: get_config_opt: Found '1' for option: ver
Mar 30 08:17:23 corosync [pcmk ] info: process_ais_conf: Enabling MCP mode: Use the Pacemaker init script to complete Pacemaker startup
Mar 30 08:17:23 corosync [pcmk ] info: get_config_opt: Defaulting to 'pcmk' for option: clustername
Mar 30 08:17:23 corosync [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_logd
Mar 30 08:17:23 corosync [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_startup: CRM: Initialized
Mar 30 08:17:23 corosync [pcmk ] Logging: Initialized pcmk_startup
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_startup: Maximum core file size is: 18446744073709551615
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_startup: Service: 10
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_startup: Local hostname: timmy
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_update_nodeid: Local node id: 89659402
Mar 30 08:17:23 corosync [pcmk ] info: update_member: Creating entry for node 89659402 born on 0
Mar 30 08:17:23 corosync [pcmk ] info: update_member: 0x11dc560 Node 89659402 now known as timmy (was: (null))
Mar 30 08:17:23 corosync [pcmk ] info: update_member: Node timmy now has 1 quorum votes (was 0)
Mar 30 08:17:23 corosync [pcmk ] info: update_member: Node 89659402/timmy is now: member
Mar 30 08:17:23 corosync [SERV ] Service engine loaded: Pacemaker Cluster Manager 1.1.6
Mar 30 08:17:23 corosync [SERV ] Service engine loaded: corosync extended virtual synchrony service
Mar 30 08:17:23 corosync [SERV ] Service engine loaded: corosync configuration service
Mar 30 08:17:23 corosync [SERV ] Service engine loaded: corosync cluster closed process group service v1.01
Mar 30 08:17:23 corosync [SERV ] Service engine loaded: corosync cluster config database access v1.01
Mar 30 08:17:23 corosync [SERV ] Service engine loaded: corosync profile loading service
Mar 30 08:17:23 corosync [SERV ] Service engine loaded: corosync cluster quorum service v0.1
Mar 30 08:17:23 corosync [MAIN ] Compatibility mode set to none. Using V2 of the synchronization engine.
Mar 30 08:17:23 corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 434456: memb=0, new=0, lost=0
Mar 30 08:17:23 corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 434456: memb=1, new=1, lost=0
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: NEW: timmy 89659402
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: MEMB: timmy 89659402
Mar 30 08:17:23 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 30 08:17:23 corosync [CPG ] chosen downlist: sender r(0) ip(10.24.88.5) ; members(old:0 left:0)
Mar 30 08:17:23 corosync [MAIN ] Completed service synchronization, ready to provide service.
Mar 30 08:17:23 corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 434460: memb=1, new=0, lost=0
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: memb: timmy 89659402
Mar 30 08:17:23 corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 434460: memb=5, new=4, lost=0
Mar 30 08:17:23 corosync [pcmk ] info: update_member: Creating entry for node 22550538 born on 434460
Mar 30 08:17:23 corosync [pcmk ] info: update_member: Node 22550538/unknown is now: member
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: NEW: .pending. 22550538
Mar 30 08:17:23 corosync [pcmk ] info: update_member: Creating entry for node 39327754 born on 434460
Mar 30 08:17:23 corosync [pcmk ] info: update_member: Node 39327754/unknown is now: member
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: NEW: .pending. 39327754
Mar 30 08:17:23 corosync [pcmk ] info: update_member: Creating entry for node 56104970 born on 434460
Mar 30 08:17:23 corosync [pcmk ] info: update_member: Node 56104970/unknown is now: member
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: NEW: .pending. 56104970
Mar 30 08:17:23 corosync [pcmk ] info: update_member: Creating entry for node 72882186 born on 434460
Mar 30 08:17:23 corosync [pcmk ] info: update_member: Node 72882186/unknown is now: member
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: NEW: .pending. 72882186
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: MEMB: .pending. 22550538
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: MEMB: .pending. 39327754
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: MEMB: .pending. 56104970
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: MEMB: .pending. 72882186
Mar 30 08:17:23 corosync [pcmk ] info: pcmk_peer_update: MEMB: timmy 89659402
Mar 30 08:17:23 corosync [pcmk ] info: send_member_notification: Sending membership update 434460 to 0 children
Mar 30 08:17:23 corosync [pcmk ] info: update_member: 0x11dc560 Node 89659402 ((null)) born on: 434460
Mar 30 08:17:23 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 30 08:17:23 corosync [pcmk ] info: update_member: 0x11e5da0 Node 39327754 (scratchy) born on: 433696
Mar 30 08:17:23 corosync [pcmk ] info: update_member: 0x11e5da0 Node 39327754 now known as scratchy (was: (null))
Mar 30 08:17:23 corosync [pcmk ] info: update_member: Node scratchy now has 1 quorum votes (was 0)
Mar 30 08:17:23 corosync [pcmk ] info: send_member_notification: Sending membership update 434460 to 0 children
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 30 08:17:23 corosync [pcmk ] info: update_member: 0x11e5fd0 Node 56104970 (walter) born on: 433692
Mar 30 08:17:23 corosync [pcmk ] info: update_member: 0x11e5fd0 Node 56104970 now known as walter (was: (null))
Mar 30 08:17:23 corosync [pcmk ] info: update_member: Node walter now has 1 quorum votes (was 0)
Mar 30 08:17:23 corosync [pcmk ] info: update_expected_votes: Expected quorum votes 2 -> 3
Mar 30 08:17:23 corosync [pcmk ] info: send_member_notification: Sending membership update 434460 to 0 children
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 30 08:17:23 corosync [pcmk ] info: update_member: 0x11e70f0 Node 72882186 (butthead) born on: 434392
Mar 30 08:17:23 corosync [pcmk ] info: update_member: 0x11e70f0 Node 72882186 now known as butthead (was: (null))
Mar 30 08:17:23 corosync [pcmk ] info: update_member: Node butthead now has 1 quorum votes (was 0)
Mar 30 08:17:23 corosync [pcmk ] info: update_expected_votes: Expected quorum votes 3 -> 4
Mar 30 08:17:23 corosync [pcmk ] info: send_member_notification: Sending membership update 434460 to 0 children
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 30 08:17:23 corosync [pcmk ] info: update_member: 0x11e6420 Node 22550538 (itchy) born on: 433684
Mar 30 08:17:23 corosync [pcmk ] info: update_member: 0x11e6420 Node 22550538 now known as itchy (was: (null))
Mar 30 08:17:23 corosync [pcmk ] info: update_member: Node itchy now has 1 quorum votes (was 0)
Mar 30 08:17:23 corosync [pcmk ] info: update_expected_votes: Expected quorum votes 4 -> 5
Mar 30 08:17:23 corosync [pcmk ] info: send_member_notification: Sending membership update 434460 to 0 children
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 30 08:17:23 corosync [CPG ] chosen downlist: sender r(0) ip(10.24.88.1) ; members(old:4 left:0)
Mar 30 08:17:23 corosync [MAIN ] Completed service synchronization, ready to provide service.
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 30 08:17:23 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 30 08:17:53 corosync [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
Mar 30 08:17:53 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 30 08:17:53 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 30 08:17:53 corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x11f2f00 for stonith-ng/0
Mar 30 08:17:53 corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x11f7260 for attrd/0
Mar 30 08:17:53 corosync [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
Mar 30 08:17:53 corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x11fb5c0 for cib/0
Mar 30 08:17:53 corosync [pcmk ] info: pcmk_ipc: Sending membership update 434460 to cib
Mar 30 08:17:54 corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x1202980 for crmd/0
Mar 30 08:17:54 corosync [pcmk ] info: pcmk_ipc: Sending membership update 434460 to crmd
Mar 30 08:17:54 corosync [pcmk ] info: update_expected_votes: Cannot update expected quorum votes 5 -> 2: value cannot be less that the current number of votes
-------------- next part --------------
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1886) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1887) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1888) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1889) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1890) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1891) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1892) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1893) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1894) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1895) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1896) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1897) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1898) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1899) from walter: not in our membership
Mar 30 08:17:56 timmy crmd: [11732]: info: do_lrm_rsc_op: Performing key=11:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761 op=p_drbd_iscsivg01:0_monitor_0 )
Mar 30 08:17:56 timmy lrmd: [11729]: info: rsc:p_drbd_iscsivg01:0:2: probe
Mar 30 08:17:56 timmy lrmd: [11738]: ERROR: (raexecocf.c:execra:178) execl failed for /usr/lib/ocf/resource.d//linbit/drbd: No such file or directory
Mar 30 08:17:56 timmy crmd: [11732]: info: do_lrm_rsc_op: Performing key=12:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761 op=p_lvm_iscsivg01_monitor_0 )
Mar 30 08:17:56 timmy lrmd: [11729]: info: rsc:p_lvm_iscsivg01:3: probe
Mar 30 08:17:56 timmy crmd: [11732]: info: do_lrm_rsc_op: Performing key=13:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761 op=p_target_iscsivg01_monitor_0 )
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (798) from itchy: not in our membership
Mar 30 08:17:56 timmy lrmd: [11729]: info: rsc:p_target_iscsivg01:4: probe
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (799) from itchy: not in our membership
Mar 30 08:17:56 timmy crmd: [11732]: info: do_lrm_rsc_op: Performing key=14:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761 op=p_lu_iscsivg01_lun1_monitor_0 )
Mar 30 08:17:56 timmy lrmd: [11729]: info: rsc:p_lu_iscsivg01_lun1:5: probe
Mar 30 08:17:56 timmy crmd: [11732]: info: do_lrm_rsc_op: Performing key=15:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761 op=p_lu_iscsivg01_lun2_monitor_0 )
Mar 30 08:17:56 timmy lrmd: [11729]: info: rsc:p_lu_iscsivg01_lun2:6: probe
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1900) from walter: not in our membership
Mar 30 08:17:56 timmy crmd: [11732]: info: do_lrm_rsc_op: Performing key=16:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761 op=p_ip_itchy-scratchy01_monitor_0 )
Mar 30 08:17:56 timmy crmd: [11732]: info: do_lrm_rsc_op: Performing key=17:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761 op=p_ip_dnsmasq_monitor_0 )
Mar 30 08:17:56 timmy lrmd: [11729]: notice: lrmd_rsc_new(): No lrm_rprovider field in message
Mar 30 08:17:56 timmy crmd: [11732]: info: do_lrm_rsc_op: Performing key=18:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761 op=p_dnsmasq_monitor_0 )
Mar 30 08:17:56 timmy crmd: [11732]: info: process_lrm_event: LRM operation p_drbd_iscsivg01:0_monitor_0 (call=2, rc=5, cib-update=7, confirmed=true) not installed
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1901) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1902) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (800) from itchy: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (801) from itchy: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (802) from itchy: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (803) from itchy: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (804) from itchy: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1903) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1904) from walter: not in our membership
Mar 30 08:17:56 timmy iSCSITarget[11740]: WARNING: Configuration parameter "portals" is not supported by the iSCSI implementation and will be ignored.
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1905) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1906) from walter: not in our membership
Mar 30 08:17:56 timmy lrmd: [11729]: info: RA output: (p_target_iscsivg01:probe:stderr) tgtadm: can't send the request to the tgt daemon, Transport endpoint is not connected
Mar 30 08:17:56 timmy lrmd: [11729]: info: RA output: (p_lu_iscsivg01_lun1:probe:stderr) tgtadm: can't send the request to the tgt daemon, Transport endpoint is not connected
Mar 30 08:17:56 timmy crmd: [11732]: info: process_lrm_event: LRM operation p_target_iscsivg01_monitor_0 (call=4, rc=7, cib-update=8, confirmed=true) not running
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1907) from walter: not in our membership
Mar 30 08:17:56 timmy crmd: [11732]: info: process_lrm_event: LRM operation p_lu_iscsivg01_lun1_monitor_0 (call=5, rc=7, cib-update=9, confirmed=true) not running
Mar 30 08:17:56 timmy lrmd: [11729]: info: RA output: (p_lu_iscsivg01_lun2:probe:stderr) tgtadm: can't send the request to the tgt daemon, Transport endpoint is not connected
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1908) from walter: not in our membership
Mar 30 08:17:56 timmy crmd: [11732]: info: process_lrm_event: LRM operation p_lu_iscsivg01_lun2_monitor_0 (call=6, rc=7, cib-update=10, confirmed=true) not running
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1909) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1910) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1911) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1912) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1913) from walter: not in our membership
Mar 30 08:17:56 timmy LVM[11739]: INFO: LVM Volume vg_iscsi0 is offline
Mar 30 08:17:56 timmy crmd: [11732]: info: process_lrm_event: LRM operation p_lvm_iscsivg01_monitor_0 (call=3, rc=7, cib-update=11, confirmed=true) not running
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1914) from walter: not in our membership
Mar 30 08:17:57 timmy lrmd: [11729]: info: rsc:p_ip_itchy-scratchy01:7: probe
Mar 30 08:17:57 timmy lrmd: [11729]: info: rsc:p_ip_dnsmasq:8: probe
Mar 30 08:17:57 timmy lrmd: [11729]: info: rsc:p_dnsmasq:9: probe
Mar 30 08:17:57 timmy crmd: [11732]: info: process_lrm_event: LRM operation p_ip_itchy-scratchy01_monitor_0 (call=7, rc=7, cib-update=12, confirmed=true) not running
Mar 30 08:17:57 timmy crmd: [11732]: info: process_lrm_event: LRM operation p_ip_dnsmasq_monitor_0 (call=8, rc=7, cib-update=13, confirmed=true) not running
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1915) from walter: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1916) from walter: not in our membership
Mar 30 08:17:57 timmy crmd: [11732]: info: process_lrm_event: LRM operation p_dnsmasq_monitor_0 (call=9, rc=7, cib-update=14, confirmed=true) not running
Mar 30 08:17:57 timmy attrd: [11730]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1917) from walter: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (805) from itchy: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1918) from walter: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1919) from walter: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1920) from walter: not in our membership
Mar 30 08:17:57 timmy attrd: [11730]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1921) from walter: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1922) from walter: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1923) from walter: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (806) from itchy: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1924) from walter: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1925) from walter: not in our membership
Mar 30 08:17:58 timmy attrd: [11730]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 30 08:17:58 timmy attrd: [11730]: notice: attrd_perform_update: Sent update 4: probe_complete=true
Mar 30 08:17:58 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (807) from itchy: not in our membership
Mar 30 08:17:58 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1926) from walter: not in our membership
Mar 30 08:17:58 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1927) from walter: not in our membership
Mar 30 08:17:58 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1928) from walter: not in our membership
Mar 30 08:17:58 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1929) from walter: not in our membership
Mar 30 08:17:58 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1930) from walter: not in our membership
Mar 30 08:18:56 timmy crmd: [11732]: WARN: cib_rsc_callback: Resource update 7 failed: (rc=-41) Remote node did not respond
Mar 30 08:18:56 timmy crmd: [11732]: WARN: cib_rsc_callback: Resource update 8 failed: (rc=-41) Remote node did not respond
Mar 30 08:18:56 timmy crmd: [11732]: WARN: cib_rsc_callback: Resource update 9 failed: (rc=-41) Remote node did not respond
Mar 30 08:18:56 timmy crmd: [11732]: WARN: cib_rsc_callback: Resource update 10 failed: (rc=-41) Remote node did not respond
Mar 30 08:18:56 timmy crmd: [11732]: WARN: cib_rsc_callback: Resource update 11 failed: (rc=-41) Remote node did not respond
Mar 30 08:18:57 timmy crmd: [11732]: WARN: cib_rsc_callback: Resource update 12 failed: (rc=-41) Remote node did not respond
Mar 30 08:18:57 timmy crmd: [11732]: WARN: cib_rsc_callback: Resource update 13 failed: (rc=-41) Remote node did not respond
Mar 30 08:18:57 timmy crmd: [11732]: WARN: cib_rsc_callback: Resource update 14 failed: (rc=-41) Remote node did not respond
Mar 30 08:19:56 timmy crmd: [11732]: notice: erase_xpath_callback: Deletion of "//node_state[@uname='timmy']/transient_attributes": Remote node did not respond (rc=-41)
Mar 30 08:19:58 timmy attrd: [11730]: WARN: attrd_cib_callback: Update 4 for probe_complete=true failed: Remote node did not respond
[root at timmy crm]# tail /var/log/messages -n 200
Mar 30 08:17:53 timmy cib: [11728]: info: crm_new_peer: Node timmy now has id: 89659402
Mar 30 08:17:53 timmy cib: [11728]: info: crm_new_peer: Node 89659402 is now known as timmy
Mar 30 08:17:53 timmy cib: [11728]: info: cib_init: Starting cib mainloop
Mar 30 08:17:53 timmy cib: [11728]: notice: ais_dispatch_message: Membership 434460: quorum acquired
Mar 30 08:17:53 timmy cib: [11728]: info: crm_update_peer: Node timmy: id=89659402 state=member (new) addr=r(0) ip(10.24.88.5) (new) votes=1 (new) born=434460 seen=434460 proc=00000000000000000000000000000000
Mar 30 08:17:53 timmy cib: [11728]: info: crm_new_peer: Node itchy now has id: 22550538
Mar 30 08:17:53 timmy cib: [11728]: info: crm_new_peer: Node 22550538 is now known as itchy
Mar 30 08:17:53 timmy cib: [11728]: info: crm_update_peer: Node itchy: id=22550538 state=member (new) addr=r(0) ip(10.24.88.1) votes=1 born=433684 seen=434460 proc=00000000000000000000000000000000
Mar 30 08:17:53 timmy cib: [11728]: info: crm_new_peer: Node scratchy now has id: 39327754
Mar 30 08:17:53 timmy cib: [11728]: info: crm_new_peer: Node 39327754 is now known as scratchy
Mar 30 08:17:53 timmy cib: [11728]: info: crm_update_peer: Node scratchy: id=39327754 state=member (new) addr=r(0) ip(10.24.88.2) votes=1 born=433696 seen=434460 proc=00000000000000000000000000000000
Mar 30 08:17:53 timmy cib: [11728]: info: crm_new_peer: Node walter now has id: 56104970
Mar 30 08:17:53 timmy cib: [11728]: info: crm_new_peer: Node 56104970 is now known as walter
Mar 30 08:17:53 timmy cib: [11728]: info: crm_update_peer: Node walter: id=56104970 state=member (new) addr=r(0) ip(10.24.88.3) votes=1 born=433692 seen=434460 proc=00000000000000000000000000000000
Mar 30 08:17:53 timmy cib: [11728]: info: crm_new_peer: Node butthead now has id: 72882186
Mar 30 08:17:53 timmy cib: [11728]: info: crm_new_peer: Node 72882186 is now known as butthead
Mar 30 08:17:53 timmy cib: [11728]: info: crm_update_peer: Node butthead: id=72882186 state=member (new) addr=r(0) ip(10.24.88.4) votes=1 born=434392 seen=434460 proc=00000000000000000000000000000000
Mar 30 08:17:53 timmy cib: [11728]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:53 timmy cib: [11728]: info: crm_update_peer: Node scratchy: id=39327754 state=member addr=r(0) ip(10.24.88.2) votes=1 born=433696 seen=434460 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:53 timmy cib: [11728]: info: crm_update_peer: Node butthead: id=72882186 state=member addr=r(0) ip(10.24.88.4) votes=1 born=434392 seen=434460 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:54 timmy lrmd: [11729]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Mar 30 08:17:54 timmy lrmd: [11729]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Mar 30 08:17:54 timmy lrmd: [11729]: info: enabling coredumps
Mar 30 08:17:54 timmy lrmd: [11729]: info: G_main_add_SignalHandler: Added signal handler for signal 10
Mar 30 08:17:54 timmy lrmd: [11729]: info: G_main_add_SignalHandler: Added signal handler for signal 12
Mar 30 08:17:54 timmy lrmd: [11729]: info: Started.
Mar 30 08:17:54 timmy crmd: [11732]: info: do_cib_control: CIB connection established
Mar 30 08:17:54 timmy crmd: [11732]: info: get_cluster_type: Cluster type is: 'openais'
Mar 30 08:17:54 timmy crmd: [11732]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
Mar 30 08:17:54 timmy crmd: [11732]: info: init_ais_connection_classic: Creating connection to our Corosync plugin
Mar 30 08:17:54 timmy crmd: [11732]: info: init_ais_connection_classic: AIS connection established
Mar 30 08:17:54 timmy corosync[11693]: [pcmk ] info: pcmk_ipc: Recorded connection 0x1202980 for crmd/0
Mar 30 08:17:54 timmy corosync[11693]: [pcmk ] info: pcmk_ipc: Sending membership update 434460 to crmd
Mar 30 08:17:54 timmy crmd: [11732]: info: get_ais_nodeid: Server details: id=89659402 uname=timmy cname=pcmk
Mar 30 08:17:54 timmy crmd: [11732]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_new_peer: Node timmy now has id: 89659402
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_new_peer: Node 89659402 is now known as timmy
Mar 30 08:17:54 timmy crmd: [11732]: info: ais_status_callback: status: timmy is now unknown
Mar 30 08:17:54 timmy crmd: [11732]: info: do_ha_control: Connected to the cluster
Mar 30 08:17:54 timmy crmd: [11732]: info: do_started: Delaying start, no membership data (0000000000100000)
Mar 30 08:17:54 timmy crmd: [11732]: info: crmd_init: Starting crmd's mainloop
Mar 30 08:17:54 timmy crmd: [11732]: info: config_query_callback: Shutdown escalation occurs after: 1200000ms
Mar 30 08:17:54 timmy crmd: [11732]: info: config_query_callback: Checking for expired actions every 900000ms
Mar 30 08:17:54 timmy crmd: [11732]: info: config_query_callback: Sending expected-votes=2 to corosync
Mar 30 08:17:54 timmy corosync[11693]: [pcmk ] info: update_expected_votes: Cannot update expected quorum votes 5 -> 2: value cannot be less that the current number of votes
Mar 30 08:17:54 timmy crmd: [11732]: notice: ais_dispatch_message: Membership 434460: quorum acquired
Mar 30 08:17:54 timmy crmd: [11732]: info: ais_status_callback: status: timmy is now member (was unknown)
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_update_peer: Node timmy: id=89659402 state=member (new) addr=r(0) ip(10.24.88.5) (new) votes=1 (new) born=434460 seen=434460 proc=00000000000000000000000000000000
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_new_peer: Node itchy now has id: 22550538
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_new_peer: Node 22550538 is now known as itchy
Mar 30 08:17:54 timmy crmd: [11732]: info: ais_status_callback: status: itchy is now unknown
Mar 30 08:17:54 timmy crmd: [11732]: info: ais_status_callback: status: itchy is now member (was unknown)
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_update_peer: Node itchy: id=22550538 state=member (new) addr=r(0) ip(10.24.88.1) votes=1 born=433684 seen=434460 proc=00000000000000000000000000000000
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_new_peer: Node scratchy now has id: 39327754
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_new_peer: Node 39327754 is now known as scratchy
Mar 30 08:17:54 timmy crmd: [11732]: info: ais_status_callback: status: scratchy is now unknown
Mar 30 08:17:54 timmy crmd: [11732]: info: ais_status_callback: status: scratchy is now member (was unknown)
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_update_peer: Node scratchy: id=39327754 state=member (new) addr=r(0) ip(10.24.88.2) votes=1 born=433696 seen=434460 proc=00000000000000000000000000000000
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_new_peer: Node walter now has id: 56104970
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_new_peer: Node 56104970 is now known as walter
Mar 30 08:17:54 timmy crmd: [11732]: info: ais_status_callback: status: walter is now unknown
Mar 30 08:17:54 timmy crmd: [11732]: info: ais_status_callback: status: walter is now member (was unknown)
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_update_peer: Node walter: id=56104970 state=member (new) addr=r(0) ip(10.24.88.3) votes=1 born=433692 seen=434460 proc=00000000000000000000000000000000
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_new_peer: Node butthead now has id: 72882186
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_new_peer: Node 72882186 is now known as butthead
Mar 30 08:17:54 timmy crmd: [11732]: info: ais_status_callback: status: butthead is now unknown
Mar 30 08:17:54 timmy crmd: [11732]: info: ais_status_callback: status: butthead is now member (was unknown)
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_update_peer: Node butthead: id=72882186 state=member (new) addr=r(0) ip(10.24.88.4) votes=1 born=434392 seen=434460 proc=00000000000000000000000000000000
Mar 30 08:17:54 timmy crmd: [11732]: info: ais_dispatch_message: Membership 434460: quorum retained
Mar 30 08:17:54 timmy crmd: [11732]: notice: crmd_peer_update: Status update: Client timmy/crmd now has status [online] (DC=<null>)
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:54 timmy crmd: [11732]: notice: crmd_peer_update: Status update: Client scratchy/crmd now has status [online] (DC=<null>)
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_update_peer: Node scratchy: id=39327754 state=member addr=r(0) ip(10.24.88.2) votes=1 born=433696 seen=434460 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:54 timmy crmd: [11732]: notice: crmd_peer_update: Status update: Client butthead/crmd now has status [online] (DC=<null>)
Mar 30 08:17:54 timmy crmd: [11732]: info: crm_update_peer: Node butthead: id=72882186 state=member addr=r(0) ip(10.24.88.4) votes=1 born=434392 seen=434460 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:54 timmy crmd: [11732]: info: do_started: The local CRM is operational
Mar 30 08:17:54 timmy crmd: [11732]: info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
Mar 30 08:17:55 timmy crmd: [11732]: info: te_connect_stonith: Attempting connection to fencing daemon...
Mar 30 08:17:56 timmy crmd: [11732]: info: te_connect_stonith: Connected
Mar 30 08:17:56 timmy crmd: [11732]: info: update_dc: Set DC to walter (3.0.5)
Mar 30 08:17:56 timmy crmd: [11732]: info: update_attrd: Connecting to attrd...
Mar 30 08:17:56 timmy crmd: [11732]: info: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Mar 30 08:17:56 timmy attrd: [11730]: notice: attrd_local_callback: Sending full refresh (origin=crmd)
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_replace message (1869) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1870) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1871) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1872) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1873) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1874) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1875) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1876) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1877) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1878) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1879) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1880) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1881) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1882) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1883) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1884) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1885) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1886) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1887) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1888) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1889) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1890) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1891) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1892) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1893) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1894) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1895) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1896) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1897) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1898) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1899) from walter: not in our membership
Mar 30 08:17:56 timmy crmd: [11732]: info: do_lrm_rsc_op: Performing key=11:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761 op=p_drbd_iscsivg01:0_monitor_0 )
Mar 30 08:17:56 timmy lrmd: [11729]: info: rsc:p_drbd_iscsivg01:0:2: probe
Mar 30 08:17:56 timmy lrmd: [11738]: ERROR: (raexecocf.c:execra:178) execl failed for /usr/lib/ocf/resource.d//linbit/drbd: No such file or directory
Mar 30 08:17:56 timmy crmd: [11732]: info: do_lrm_rsc_op: Performing key=12:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761 op=p_lvm_iscsivg01_monitor_0 )
Mar 30 08:17:56 timmy lrmd: [11729]: info: rsc:p_lvm_iscsivg01:3: probe
Mar 30 08:17:56 timmy crmd: [11732]: info: do_lrm_rsc_op: Performing key=13:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761 op=p_target_iscsivg01_monitor_0 )
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (798) from itchy: not in our membership
Mar 30 08:17:56 timmy lrmd: [11729]: info: rsc:p_target_iscsivg01:4: probe
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (799) from itchy: not in our membership
Mar 30 08:17:56 timmy crmd: [11732]: info: do_lrm_rsc_op: Performing key=14:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761 op=p_lu_iscsivg01_lun1_monitor_0 )
Mar 30 08:17:56 timmy lrmd: [11729]: info: rsc:p_lu_iscsivg01_lun1:5: probe
Mar 30 08:17:56 timmy crmd: [11732]: info: do_lrm_rsc_op: Performing key=15:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761 op=p_lu_iscsivg01_lun2_monitor_0 )
Mar 30 08:17:56 timmy lrmd: [11729]: info: rsc:p_lu_iscsivg01_lun2:6: probe
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1900) from walter: not in our membership
Mar 30 08:17:56 timmy crmd: [11732]: info: do_lrm_rsc_op: Performing key=16:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761 op=p_ip_itchy-scratchy01_monitor_0 )
Mar 30 08:17:56 timmy crmd: [11732]: info: do_lrm_rsc_op: Performing key=17:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761 op=p_ip_dnsmasq_monitor_0 )
Mar 30 08:17:56 timmy lrmd: [11729]: notice: lrmd_rsc_new(): No lrm_rprovider field in message
Mar 30 08:17:56 timmy crmd: [11732]: info: do_lrm_rsc_op: Performing key=18:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761 op=p_dnsmasq_monitor_0 )
Mar 30 08:17:56 timmy crmd: [11732]: info: process_lrm_event: LRM operation p_drbd_iscsivg01:0_monitor_0 (call=2, rc=5, cib-update=7, confirmed=true) not installed
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1901) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1902) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (800) from itchy: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (801) from itchy: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (802) from itchy: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (803) from itchy: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (804) from itchy: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1903) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1904) from walter: not in our membership
Mar 30 08:17:56 timmy iSCSITarget[11740]: WARNING: Configuration parameter "portals" is not supported by the iSCSI implementation and will be ignored.
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1905) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1906) from walter: not in our membership
Mar 30 08:17:56 timmy lrmd: [11729]: info: RA output: (p_target_iscsivg01:probe:stderr) tgtadm: can't send the request to the tgt daemon, Transport endpoint is not connected
Mar 30 08:17:56 timmy lrmd: [11729]: info: RA output: (p_lu_iscsivg01_lun1:probe:stderr) tgtadm: can't send the request to the tgt daemon, Transport endpoint is not connected
Mar 30 08:17:56 timmy crmd: [11732]: info: process_lrm_event: LRM operation p_target_iscsivg01_monitor_0 (call=4, rc=7, cib-update=8, confirmed=true) not running
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1907) from walter: not in our membership
Mar 30 08:17:56 timmy crmd: [11732]: info: process_lrm_event: LRM operation p_lu_iscsivg01_lun1_monitor_0 (call=5, rc=7, cib-update=9, confirmed=true) not running
Mar 30 08:17:56 timmy lrmd: [11729]: info: RA output: (p_lu_iscsivg01_lun2:probe:stderr) tgtadm: can't send the request to the tgt daemon, Transport endpoint is not connected
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1908) from walter: not in our membership
Mar 30 08:17:56 timmy crmd: [11732]: info: process_lrm_event: LRM operation p_lu_iscsivg01_lun2_monitor_0 (call=6, rc=7, cib-update=10, confirmed=true) not running
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1909) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1910) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1911) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1912) from walter: not in our membership
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1913) from walter: not in our membership
Mar 30 08:17:56 timmy LVM[11739]: INFO: LVM Volume vg_iscsi0 is offline
Mar 30 08:17:56 timmy crmd: [11732]: info: process_lrm_event: LRM operation p_lvm_iscsivg01_monitor_0 (call=3, rc=7, cib-update=11, confirmed=true) not running
Mar 30 08:17:56 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1914) from walter: not in our membership
Mar 30 08:17:57 timmy lrmd: [11729]: info: rsc:p_ip_itchy-scratchy01:7: probe
Mar 30 08:17:57 timmy lrmd: [11729]: info: rsc:p_ip_dnsmasq:8: probe
Mar 30 08:17:57 timmy lrmd: [11729]: info: rsc:p_dnsmasq:9: probe
Mar 30 08:17:57 timmy crmd: [11732]: info: process_lrm_event: LRM operation p_ip_itchy-scratchy01_monitor_0 (call=7, rc=7, cib-update=12, confirmed=true) not running
Mar 30 08:17:57 timmy crmd: [11732]: info: process_lrm_event: LRM operation p_ip_dnsmasq_monitor_0 (call=8, rc=7, cib-update=13, confirmed=true) not running
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1915) from walter: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1916) from walter: not in our membership
Mar 30 08:17:57 timmy crmd: [11732]: info: process_lrm_event: LRM operation p_dnsmasq_monitor_0 (call=9, rc=7, cib-update=14, confirmed=true) not running
Mar 30 08:17:57 timmy attrd: [11730]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1917) from walter: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (805) from itchy: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1918) from walter: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1919) from walter: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1920) from walter: not in our membership
Mar 30 08:17:57 timmy attrd: [11730]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1921) from walter: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1922) from walter: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1923) from walter: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (806) from itchy: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1924) from walter: not in our membership
Mar 30 08:17:57 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1925) from walter: not in our membership
Mar 30 08:17:58 timmy attrd: [11730]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 30 08:17:58 timmy attrd: [11730]: notice: attrd_perform_update: Sent update 4: probe_complete=true
Mar 30 08:17:58 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_modify message (807) from itchy: not in our membership
Mar 30 08:17:58 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1926) from walter: not in our membership
Mar 30 08:17:58 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1927) from walter: not in our membership
Mar 30 08:17:58 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1928) from walter: not in our membership
Mar 30 08:17:58 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1929) from walter: not in our membership
Mar 30 08:17:58 timmy cib: [11728]: WARN: cib_peer_callback: Discarding cib_apply_diff message (1930) from walter: not in our membership
Mar 30 08:18:56 timmy crmd: [11732]: WARN: cib_rsc_callback: Resource update 7 failed: (rc=-41) Remote node did not respond
Mar 30 08:18:56 timmy crmd: [11732]: WARN: cib_rsc_callback: Resource update 8 failed: (rc=-41) Remote node did not respond
Mar 30 08:18:56 timmy crmd: [11732]: WARN: cib_rsc_callback: Resource update 9 failed: (rc=-41) Remote node did not respond
Mar 30 08:18:56 timmy crmd: [11732]: WARN: cib_rsc_callback: Resource update 10 failed: (rc=-41) Remote node did not respond
Mar 30 08:18:56 timmy crmd: [11732]: WARN: cib_rsc_callback: Resource update 11 failed: (rc=-41) Remote node did not respond
Mar 30 08:18:57 timmy crmd: [11732]: WARN: cib_rsc_callback: Resource update 12 failed: (rc=-41) Remote node did not respond
Mar 30 08:18:57 timmy crmd: [11732]: WARN: cib_rsc_callback: Resource update 13 failed: (rc=-41) Remote node did not respond
Mar 30 08:18:57 timmy crmd: [11732]: WARN: cib_rsc_callback: Resource update 14 failed: (rc=-41) Remote node did not respond
Mar 30 08:19:56 timmy crmd: [11732]: notice: erase_xpath_callback: Deletion of "//node_state[@uname='timmy']/transient_attributes": Remote node did not respond (rc=-41)
Mar 30 08:19:58 timmy attrd: [11730]: WARN: attrd_cib_callback: Update 4 for probe_complete=true failed: Remote node did not respond
-------------- next part --------------
Mar 30 08:17:23 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:126 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.247.1) : Non-status change
Mar 30 08:17:23 walter crmd: [3503]: WARN: match_down_event: No match for shutdown action on timmy.damnet.us
Mar 30 08:17:23 walter crmd: [3503]: info: te_update_diff: Stonith/shutdown of timmy.damnet.us not matched
Mar 30 08:17:23 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/1132, version=0.247.6): ok (rc=0)
Mar 30 08:17:23 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:234 - Triggered transition abort (complete=1, tag=node_state, id=timmy.damnet.us, magic=NA, cib=0.247.2) : Node failure
Mar 30 08:17:23 walter crmd: [3503]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 30 08:17:23 walter crmd: [3503]: info: do_state_transition: Membership changed: 434452 -> 434460 - join restart
Mar 30 08:17:23 walter crmd: [3503]: info: do_pe_invoke: Query 1133: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:23 walter crmd: [3503]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=do_state_transition ]
Mar 30 08:17:23 walter crmd: [3503]: info: update_dc: Unset DC walter
Mar 30 08:17:23 walter crmd: [3503]: info: join_make_offer: Making join offers based on membership 434460
Mar 30 08:17:23 walter crmd: [3503]: info: do_dc_join_offer_all: join-28: Waiting on 5 outstanding join acks
Mar 30 08:17:23 walter crmd: [3503]: info: update_dc: Set DC to walter (3.0.5)
Mar 30 08:17:53 walter pacemakerd: [3494]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000000002 (was 00000000000000000000000000111312)
Mar 30 08:17:53 walter pacemakerd: [3494]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000100002 (was 00000000000000000000000000000002)
Mar 30 08:17:53 walter stonith-ng: [3498]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000000002 (new)
Mar 30 08:17:53 walter crmd: [3503]: notice: crmd_peer_update: Status update: Client timmy/crmd now has status [offline] (DC=true)
Mar 30 08:17:53 walter cib: [3499]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000000002 (new)
Mar 30 08:17:53 walter crmd: [3503]: info: erase_node_from_join: Removed node timmy from join calculations: welcomed=1 itegrated=0 finalized=0 confirmed=0
Mar 30 08:17:53 walter crmd: [3503]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000000002 (new)
Mar 30 08:17:53 walter stonith-ng: [3498]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000100002 (new)
Mar 30 08:17:53 walter pacemakerd: [3494]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000100102 (was 00000000000000000000000000100002)
Mar 30 08:17:53 walter crmd: [3503]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100002 (new)
Mar 30 08:17:53 walter cib: [3499]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100002 (new)
Mar 30 08:17:53 walter crmd: [3503]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 30 08:17:53 walter crmd: [3503]: info: do_state_transition: All 4 cluster nodes responded to the join offer.
Mar 30 08:17:53 walter crmd: [3503]: info: do_dc_join_finalize: join-28: Syncing the CIB from walter to the rest of the cluster
Mar 30 08:17:53 walter pacemakerd: [3494]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000100112 (was 00000000000000000000000000100102)
Mar 30 08:17:53 walter crmd: [3503]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100102 (new)
Mar 30 08:17:53 walter stonith-ng: [3498]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000100102 (new)
Mar 30 08:17:53 walter pacemakerd: [3494]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000101112 (was 00000000000000000000000000100112)
Mar 30 08:17:53 walter crmd: [3503]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100112 (new)
Mar 30 08:17:53 walter stonith-ng: [3498]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000100112 (new)
Mar 30 08:17:53 walter crmd: [3503]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000101112 (new)
Mar 30 08:17:53 walter pacemakerd: [3494]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000111112 (was 00000000000000000000000000101112)
Mar 30 08:17:53 walter stonith-ng: [3498]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000101112 (new)
Mar 30 08:17:53 walter pacemakerd: [3494]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000111312 (was 00000000000000000000000000111112)
Mar 30 08:17:53 walter crmd: [3503]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111112 (new)
Mar 30 08:17:53 walter stonith-ng: [3498]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000111112 (new)
Mar 30 08:17:53 walter stonith-ng: [3498]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:53 walter crmd: [3503]: notice: crmd_peer_update: Status update: Client timmy/crmd now has status [online] (DC=true)
Mar 30 08:17:53 walter crmd: [3503]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:53 walter crmd: [3503]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=crmd_peer_update ]
Mar 30 08:17:53 walter crmd: [3503]: info: update_dc: Unset DC walter
Mar 30 08:17:53 walter crmd: [3503]: info: do_dc_join_offer_all: join-29: Waiting on 5 outstanding join acks
Mar 30 08:17:53 walter crmd: [3503]: info: update_dc: Set DC to walter (3.0.5)
Mar 30 08:17:53 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/1136, version=0.247.7): ok (rc=0)
Mar 30 08:17:53 walter cib: [3499]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100102 (new)
Mar 30 08:17:53 walter cib: [3499]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100112 (new)
Mar 30 08:17:53 walter cib: [3499]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000101112 (new)
Mar 30 08:17:53 walter cib: [3499]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111112 (new)
Mar 30 08:17:53 walter cib: [3499]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:55 walter crmd: [3503]: info: update_dc: Unset DC walter
Mar 30 08:17:55 walter crmd: [3503]: info: do_dc_join_offer_all: A new node joined the cluster
Mar 30 08:17:55 walter crmd: [3503]: info: do_dc_join_offer_all: join-30: Waiting on 5 outstanding join acks
Mar 30 08:17:55 walter crmd: [3503]: info: update_dc: Set DC to walter (3.0.5)
Mar 30 08:17:56 walter crmd: [3503]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 30 08:17:56 walter crmd: [3503]: info: do_state_transition: All 5 cluster nodes responded to the join offer.
Mar 30 08:17:56 walter crmd: [3503]: info: do_dc_join_finalize: join-30: Syncing the CIB from walter to the rest of the cluster
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/1140, version=0.247.8): ok (rc=0)
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1141, version=0.247.9): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: do_dc_join_ack: join-30: Updating node state to member for timmy
Mar 30 08:17:56 walter crmd: [3503]: info: do_dc_join_ack: join-30: Updating node state to member for scratchy
Mar 30 08:17:56 walter crmd: [3503]: info: do_dc_join_ack: join-30: Updating node state to member for walter
Mar 30 08:17:56 walter crmd: [3503]: info: do_dc_join_ack: join-30: Updating node state to member for itchy
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1142, version=0.247.10): ok (rc=0)
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1143, version=0.247.11): ok (rc=0)
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1144, version=0.247.12): ok (rc=0)
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1145, version=0.247.13): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: do_dc_join_ack: join-30: Updating node state to member for butthead
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='timmy']/transient_attributes (origin=timmy/crmd/6, version=0.247.14): ok (rc=0)
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='timmy']/lrm (origin=local/crmd/1146, version=0.247.16): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: erase_xpath_callback: Deletion of "//node_state[@uname='timmy']/lrm": ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 30 08:17:56 walter crmd: [3503]: info: do_state_transition: All 5 cluster nodes are eligible to run resources.
Mar 30 08:17:56 walter crmd: [3503]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date
Mar 30 08:17:56 walter crmd: [3503]: info: crm_update_quorum: Updating quorum status to true (call=1158)
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1159: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='scratchy']/lrm (origin=local/crmd/1148, version=0.247.18): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:320 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=p_ip_dnsmasq_last_0, magic=0:7;13:151:7:a3d5b99d-d9b9-4760-94ed-a977b296fcee, cib=0.247.18) : Resource op removal
Mar 30 08:17:56 walter crmd: [3503]: info: erase_xpath_callback: Deletion of "//node_state[@uname='scratchy']/lrm": ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1160: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter crmd: [3503]: info: te_update_diff: Detected LRM refresh - 8 resources updated: Skipping all resource events
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:276 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.247.19) : LRM Refresh
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1161: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='walter']/lrm (origin=local/crmd/1150, version=0.247.20): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:320 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=p_ip_dnsmasq_last_0, magic=0:0;42:49:0:cc023a3a-5b96-4d1f-b376-25a6b68b3761, cib=0.247.20) : Resource op removal
Mar 30 08:17:56 walter crmd: [3503]: info: erase_xpath_callback: Deletion of "//node_state[@uname='walter']/lrm": ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1162: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter attrd: [3501]: notice: attrd_local_callback: Sending full refresh (origin=crmd)
Mar 30 08:17:56 walter attrd: [3501]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 30 08:17:56 walter crmd: [3503]: info: te_update_diff: Detected LRM refresh - 8 resources updated: Skipping all resource events
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:276 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.247.21) : LRM Refresh
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1163: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='itchy']/lrm (origin=local/crmd/1152, version=0.247.22): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:320 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=p_ip_dnsmasq_last_0, magic=0:7;10:151:7:a3d5b99d-d9b9-4760-94ed-a977b296fcee, cib=0.247.22) : Resource op removal
Mar 30 08:17:56 walter crmd: [3503]: info: erase_xpath_callback: Deletion of "//node_state[@uname='itchy']/lrm": ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1164: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter crmd: [3503]: info: te_update_diff: Detected LRM refresh - 8 resources updated: Skipping all resource events
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:276 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.247.23) : LRM Refresh
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1165: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='butthead']/lrm (origin=local/crmd/1154, version=0.247.24): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:320 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=p_drbd_iscsivg01:0_last_failure_0, magic=0:5;10:48:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761, cib=0.247.24) : Resource op removal
Mar 30 08:17:56 walter crmd: [3503]: info: erase_xpath_callback: Deletion of "//node_state[@uname='butthead']/lrm": ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1166: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter crmd: [3503]: info: te_update_diff: Detected LRM refresh - 8 resources updated: Skipping all resource events
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:276 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.247.25) : LRM Refresh
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1167: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter attrd: [3501]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-p_ip_dnsmasq (1332707473)
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1156, version=0.247.28): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: WARN: match_down_event: No match for shutdown action on timmy.damnet.us
Mar 30 08:17:56 walter crmd: [3503]: info: te_update_diff: Stonith/shutdown of timmy.damnet.us not matched
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:234 - Triggered transition abort (complete=1, tag=node_state, id=timmy.damnet.us, magic=NA, cib=0.247.29) : Node failure
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1168: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/1158, version=0.247.30): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke_callback: Invoking the PE: query=1168, ref=pe_calc-dc-1333120676-718, seq=434460, quorate=1
Mar 30 08:17:56 walter pengine: [3502]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 30 08:17:56 walter pengine: [3502]: notice: unpack_rsc_op: Hard error - p_drbd_iscsivg01:0_last_failure_0 failed with rc=5: Preventing ms_drbd_iscsivg01 from re-starting on butthead
Mar 30 08:17:56 walter pengine: [3502]: notice: unpack_rsc_op: Hard error - p_drbd_iscsivg01:0_last_failure_0 failed with rc=5: Preventing ms_drbd_iscsivg01 from re-starting on walter
Mar 30 08:17:56 walter pengine: [3502]: notice: unpack_rsc_op: Operation p_dnsmasq_last_failure_0 found resource p_dnsmasq active on scratchy
Mar 30 08:17:56 walter pengine: [3502]: notice: unpack_rsc_op: Operation p_drbd_iscsivg01:0_last_failure_0 found resource p_drbd_iscsivg01:0 active on scratchy
Mar 30 08:17:56 walter pengine: [3502]: notice: unpack_rsc_op: Operation p_dnsmasq_last_failure_0 found resource p_dnsmasq active on itchy
Mar 30 08:17:56 walter pengine: [3502]: notice: unpack_rsc_op: Hard error - p_drbd_iscsivg01:1_monitor_29000 failed with rc=5: Preventing ms_drbd_iscsivg01 from re-starting on itchy
Mar 30 08:17:56 walter pengine: [3502]: WARN: unpack_rsc_op: Processing failed op p_drbd_iscsivg01:1_monitor_29000 on itchy: not installed (5)
Mar 30 08:17:56 walter pengine: [3502]: WARN: common_apply_stickiness: Forcing ms_drbd_iscsivg01 away from itchy after 1000000 failures (max=1000000)
Mar 30 08:17:56 walter pengine: [3502]: WARN: common_apply_stickiness: Forcing ms_drbd_iscsivg01 away from itchy after 1000000 failures (max=1000000)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_drbd_iscsivg01:0 (Slave scratchy)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_drbd_iscsivg01:1 (Stopped)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_lvm_iscsivg01 (Stopped)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_target_iscsivg01 (Stopped)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_lu_iscsivg01_lun1 (Stopped)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_lu_iscsivg01_lun2 (Stopped)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_ip_itchy-scratchy01 (Stopped)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_ip_dnsmasq (Started butthead)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_dnsmasq (Started butthead)
Mar 30 08:17:56 walter crmd: [3503]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 30 08:17:56 walter crmd: [3503]: info: unpack_graph: Unpacked transition 115: 10 actions in 10 synapses
Mar 30 08:17:56 walter crmd: [3503]: info: do_te_invoke: Processing graph 115 (ref=pe_calc-dc-1333120676-718) derived from /var/lib/pengine/pe-input-56.bz2
Mar 30 08:17:56 walter crmd: [3503]: info: te_rsc_command: Initiating action 11: monitor p_drbd_iscsivg01:0_monitor_0 on timmy
Mar 30 08:17:56 walter crmd: [3503]: info: te_rsc_command: Initiating action 12: monitor p_lvm_iscsivg01_monitor_0 on timmy
Mar 30 08:17:56 walter crmd: [3503]: info: te_rsc_command: Initiating action 13: monitor p_target_iscsivg01_monitor_0 on timmy
Mar 30 08:17:56 walter crmd: [3503]: info: te_rsc_command: Initiating action 14: monitor p_lu_iscsivg01_lun1_monitor_0 on timmy
Mar 30 08:17:56 walter crmd: [3503]: info: te_rsc_command: Initiating action 15: monitor p_lu_iscsivg01_lun2_monitor_0 on timmy
Mar 30 08:17:56 walter crmd: [3503]: info: te_rsc_command: Initiating action 16: monitor p_ip_itchy-scratchy01_monitor_0 on timmy
Mar 30 08:17:56 walter crmd: [3503]: info: te_rsc_command: Initiating action 17: monitor p_ip_dnsmasq_monitor_0 on timmy
Mar 30 08:17:56 walter crmd: [3503]: info: te_rsc_command: Initiating action 18: monitor p_dnsmasq_monitor_0 on timmy
Mar 30 08:17:56 walter crmd: [3503]: WARN: status_from_rc: Action 11 (p_drbd_iscsivg01:0_monitor_0) on timmy failed (target: 7 vs. rc: 5): Error
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: match_graph_event:277 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=p_drbd_iscsivg01:0_last_failure_0, magic=0:5;11:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761, cib=0.247.41) : Event failed
Mar 30 08:17:56 walter crmd: [3503]: info: update_abort_priority: Abort priority upgraded from 0 to 1
Mar 30 08:17:56 walter crmd: [3503]: info: update_abort_priority: Abort action done superceeded by restart
Mar 30 08:17:56 walter crmd: [3503]: info: match_graph_event: Action p_drbd_iscsivg01:0_monitor_0 (11) confirmed on timmy (rc=4)
Mar 30 08:17:56 walter pengine: [3502]: notice: process_pe_message: Transition 115: PEngine Input stored in: /var/lib/pengine/pe-input-56.bz2
Mar 30 08:17:56 walter crmd: [3503]: info: match_graph_event: Action p_target_iscsivg01_monitor_0 (13) confirmed on timmy (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: match_graph_event: Action p_lu_iscsivg01_lun1_monitor_0 (14) confirmed on timmy (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: match_graph_event: Action p_lu_iscsivg01_lun2_monitor_0 (15) confirmed on timmy (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: match_graph_event: Action p_lvm_iscsivg01_monitor_0 (12) confirmed on timmy (rc=0)
Mar 30 08:17:57 walter crmd: [3503]: info: match_graph_event: Action p_ip_itchy-scratchy01_monitor_0 (16) confirmed on timmy (rc=0)
Mar 30 08:17:57 walter crmd: [3503]: info: match_graph_event: Action p_ip_dnsmasq_monitor_0 (17) confirmed on timmy (rc=0)
Mar 30 08:17:57 walter crmd: [3503]: info: match_graph_event: Action p_dnsmasq_monitor_0 (18) confirmed on timmy (rc=0)
Mar 30 08:17:57 walter crmd: [3503]: info: te_rsc_command: Initiating action 10: probe_complete probe_complete on timmy - no waiting
Mar 30 08:17:57 walter crmd: [3503]: info: run_graph: ====================================================
Mar 30 08:17:57 walter crmd: [3503]: notice: run_graph: Transition 115 (Complete=9, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-input-56.bz2): Stopped
Mar 30 08:17:57 walter crmd: [3503]: info: te_graph_trigger: Transition 115 is now complete
Mar 30 08:17:57 walter crmd: [3503]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 30 08:17:57 walter crmd: [3503]: info: do_state_transition: All 5 cluster nodes are eligible to run resources.
Mar 30 08:17:57 walter crmd: [3503]: info: do_pe_invoke: Query 1169: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:57 walter crmd: [3503]: info: do_pe_invoke_callback: Invoking the PE: query=1169, ref=pe_calc-dc-1333120677-728, seq=434460, quorate=1
Mar 30 08:17:57 walter pengine: [3502]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 30 08:17:57 walter pengine: [3502]: notice: unpack_rsc_op: Hard error - p_drbd_iscsivg01:0_last_failure_0 failed with rc=5: Preventing ms_drbd_iscsivg01 from re-starting on butthead
Mar 30 08:17:57 walter pengine: [3502]: notice: unpack_rsc_op: Hard error - p_drbd_iscsivg01:0_last_failure_0 failed with rc=5: Preventing ms_drbd_iscsivg01 from re-starting on walter
Mar 30 08:17:57 walter pengine: [3502]: notice: unpack_rsc_op: Operation p_dnsmasq_last_failure_0 found resource p_dnsmasq active on scratchy
Mar 30 08:17:57 walter pengine: [3502]: notice: unpack_rsc_op: Operation p_drbd_iscsivg01:0_last_failure_0 found resource p_drbd_iscsivg01:0 active on scratchy
Mar 30 08:17:57 walter pengine: [3502]: notice: unpack_rsc_op: Operation p_dnsmasq_last_failure_0 found resource p_dnsmasq active on itchy
Mar 30 08:17:57 walter pengine: [3502]: notice: unpack_rsc_op: Hard error - p_drbd_iscsivg01:1_monitor_29000 failed with rc=5: Preventing ms_drbd_iscsivg01 from re-starting on itchy
Mar 30 08:17:57 walter pengine: [3502]: WARN: unpack_rsc_op: Processing failed op p_drbd_iscsivg01:1_monitor_29000 on itchy: not installed (5)
Mar 30 08:17:57 walter pengine: [3502]: notice: unpack_rsc_op: Hard error - p_drbd_iscsivg01:0_last_failure_0 failed with rc=5: Preventing ms_drbd_iscsivg01 from re-starting on timmy
Mar 30 08:17:57 walter pengine: [3502]: WARN: common_apply_stickiness: Forcing ms_drbd_iscsivg01 away from itchy after 1000000 failures (max=1000000)
Mar 30 08:17:57 walter pengine: [3502]: WARN: common_apply_stickiness: Forcing ms_drbd_iscsivg01 away from itchy after 1000000 failures (max=1000000)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_drbd_iscsivg01:0 (Slave scratchy)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_drbd_iscsivg01:1 (Stopped)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_lvm_iscsivg01 (Stopped)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_target_iscsivg01 (Stopped)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_lu_iscsivg01_lun1 (Stopped)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_lu_iscsivg01_lun2 (Stopped)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_ip_itchy-scratchy01 (Stopped)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_ip_dnsmasq (Started butthead)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_dnsmasq (Started butthead)
Mar 30 08:17:57 walter crmd: [3503]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 30 08:17:57 walter crmd: [3503]: info: unpack_graph: Unpacked transition 116: 1 actions in 1 synapses
Mar 30 08:17:57 walter crmd: [3503]: info: do_te_invoke: Processing graph 116 (ref=pe_calc-dc-1333120677-728) derived from /var/lib/pengine/pe-input-57.bz2
Mar 30 08:17:57 walter crmd: [3503]: info: te_rsc_command: Initiating action 10: probe_complete probe_complete on timmy - no waiting
Mar 30 08:17:57 walter crmd: [3503]: info: run_graph: ====================================================
Mar 30 08:17:57 walter crmd: [3503]: notice: run_graph: Transition 116 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-57.bz2): Complete
Mar 30 08:17:57 walter crmd: [3503]: info: te_graph_trigger: Transition 116 is now complete
Mar 30 08:17:57 walter crmd: [3503]: info: notify_crmd: Transition 116 status: done - <null>
Mar 30 08:17:57 walter crmd: [3503]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 30 08:17:57 walter crmd: [3503]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 30 08:17:57 walter pengine: [3502]: notice: process_pe_message: Transition 116: PEngine Input stored in: /var/lib/pengine/pe-input-57.bz2
Mar 30 08:21:50 walter cib: [3499]: info: cib_stats: Processed 102 operations (2058.00us average, 0% utilization) in the last 10min
-------------- next part --------------
Mar 30 08:17:23 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:126 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.247.1) : Non-status change
Mar 30 08:17:23 walter crmd: [3503]: WARN: match_down_event: No match for shutdown action on timmy.damnet.us
Mar 30 08:17:23 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/1132, version=0.247.6): ok (rc=0)
Mar 30 08:17:23 walter crmd: [3503]: info: te_update_diff: Stonith/shutdown of timmy.damnet.us not matched
Mar 30 08:17:23 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:234 - Triggered transition abort (complete=1, tag=node_state, id=timmy.damnet.us, magic=NA, cib=0.247.2) : Node failure
Mar 30 08:17:23 walter crmd: [3503]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 30 08:17:23 walter crmd: [3503]: info: do_state_transition: Membership changed: 434452 -> 434460 - join restart
Mar 30 08:17:23 walter crmd: [3503]: info: do_pe_invoke: Query 1133: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:23 walter crmd: [3503]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=do_state_transition ]
Mar 30 08:17:23 walter crmd: [3503]: info: update_dc: Unset DC walter
Mar 30 08:17:23 walter crmd: [3503]: info: join_make_offer: Making join offers based on membership 434460
Mar 30 08:17:23 walter crmd: [3503]: info: do_dc_join_offer_all: join-28: Waiting on 5 outstanding join acks
Mar 30 08:17:23 walter crmd: [3503]: info: update_dc: Set DC to walter (3.0.5)
Mar 30 08:17:53 walter pacemakerd: [3494]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000000002 (was 00000000000000000000000000111312)
Mar 30 08:17:53 walter pacemakerd: [3494]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000100002 (was 00000000000000000000000000000002)
Mar 30 08:17:53 walter stonith-ng: [3498]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000000002 (new)
Mar 30 08:17:53 walter crmd: [3503]: notice: crmd_peer_update: Status update: Client timmy/crmd now has status [offline] (DC=true)
Mar 30 08:17:53 walter cib: [3499]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000000002 (new)
Mar 30 08:17:53 walter crmd: [3503]: info: erase_node_from_join: Removed node timmy from join calculations: welcomed=1 itegrated=0 finalized=0 confirmed=0
Mar 30 08:17:53 walter crmd: [3503]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000000002 (new)
Mar 30 08:17:53 walter pacemakerd: [3494]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000100102 (was 00000000000000000000000000100002)
Mar 30 08:17:53 walter stonith-ng: [3498]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000100002 (new)
Mar 30 08:17:53 walter crmd: [3503]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100002 (new)
Mar 30 08:17:53 walter cib: [3499]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100002 (new)
Mar 30 08:17:53 walter crmd: [3503]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 30 08:17:53 walter crmd: [3503]: info: do_state_transition: All 4 cluster nodes responded to the join offer.
Mar 30 08:17:53 walter crmd: [3503]: info: do_dc_join_finalize: join-28: Syncing the CIB from walter to the rest of the cluster
Mar 30 08:17:53 walter pacemakerd: [3494]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000100112 (was 00000000000000000000000000100102)
Mar 30 08:17:53 walter crmd: [3503]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100102 (new)
Mar 30 08:17:53 walter stonith-ng: [3498]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000100102 (new)
Mar 30 08:17:53 walter pacemakerd: [3494]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000101112 (was 00000000000000000000000000100112)
Mar 30 08:17:53 walter crmd: [3503]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100112 (new)
Mar 30 08:17:53 walter stonith-ng: [3498]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000100112 (new)
Mar 30 08:17:53 walter pacemakerd: [3494]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000111112 (was 00000000000000000000000000101112)
Mar 30 08:17:53 walter crmd: [3503]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000101112 (new)
Mar 30 08:17:53 walter stonith-ng: [3498]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000101112 (new)
Mar 30 08:17:53 walter pacemakerd: [3494]: info: update_node_processes: Node timmy now has process list: 00000000000000000000000000111312 (was 00000000000000000000000000111112)
Mar 30 08:17:53 walter stonith-ng: [3498]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000111112 (new)
Mar 30 08:17:53 walter crmd: [3503]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111112 (new)
Mar 30 08:17:53 walter stonith-ng: [3498]: info: crm_update_peer: Node timmy: id=0 state=unknown addr=(null) votes=0 born=0 seen=0 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:53 walter crmd: [3503]: notice: crmd_peer_update: Status update: Client timmy/crmd now has status [online] (DC=true)
Mar 30 08:17:53 walter crmd: [3503]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:53 walter crmd: [3503]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=crmd_peer_update ]
Mar 30 08:17:53 walter crmd: [3503]: info: update_dc: Unset DC walter
Mar 30 08:17:53 walter crmd: [3503]: info: do_dc_join_offer_all: join-29: Waiting on 5 outstanding join acks
Mar 30 08:17:53 walter crmd: [3503]: info: update_dc: Set DC to walter (3.0.5)
Mar 30 08:17:53 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/1136, version=0.247.7): ok (rc=0)
Mar 30 08:17:53 walter cib: [3499]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100102 (new)
Mar 30 08:17:53 walter cib: [3499]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000100112 (new)
Mar 30 08:17:53 walter cib: [3499]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000101112 (new)
Mar 30 08:17:53 walter cib: [3499]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111112 (new)
Mar 30 08:17:53 walter cib: [3499]: info: crm_update_peer: Node timmy: id=89659402 state=member addr=r(0) ip(10.24.88.5) votes=1 born=434460 seen=434460 proc=00000000000000000000000000111312 (new)
Mar 30 08:17:55 walter crmd: [3503]: info: update_dc: Unset DC walter
Mar 30 08:17:55 walter crmd: [3503]: info: do_dc_join_offer_all: A new node joined the cluster
Mar 30 08:17:55 walter crmd: [3503]: info: do_dc_join_offer_all: join-30: Waiting on 5 outstanding join acks
Mar 30 08:17:55 walter crmd: [3503]: info: update_dc: Set DC to walter (3.0.5)
Mar 30 08:17:56 walter crmd: [3503]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 30 08:17:56 walter crmd: [3503]: info: do_state_transition: All 5 cluster nodes responded to the join offer.
Mar 30 08:17:56 walter crmd: [3503]: info: do_dc_join_finalize: join-30: Syncing the CIB from walter to the rest of the cluster
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/1140, version=0.247.8): ok (rc=0)
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1141, version=0.247.9): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: do_dc_join_ack: join-30: Updating node state to member for timmy
Mar 30 08:17:56 walter crmd: [3503]: info: do_dc_join_ack: join-30: Updating node state to member for scratchy
Mar 30 08:17:56 walter crmd: [3503]: info: do_dc_join_ack: join-30: Updating node state to member for walter
Mar 30 08:17:56 walter crmd: [3503]: info: do_dc_join_ack: join-30: Updating node state to member for itchy
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1142, version=0.247.10): ok (rc=0)
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1143, version=0.247.11): ok (rc=0)
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1144, version=0.247.12): ok (rc=0)
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1145, version=0.247.13): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: do_dc_join_ack: join-30: Updating node state to member for butthead
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='timmy']/transient_attributes (origin=timmy/crmd/6, version=0.247.14): ok (rc=0)
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='timmy']/lrm (origin=local/crmd/1146, version=0.247.16): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: erase_xpath_callback: Deletion of "//node_state[@uname='timmy']/lrm": ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 30 08:17:56 walter crmd: [3503]: info: do_state_transition: All 5 cluster nodes are eligible to run resources.
Mar 30 08:17:56 walter crmd: [3503]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date
Mar 30 08:17:56 walter crmd: [3503]: info: crm_update_quorum: Updating quorum status to true (call=1158)
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1159: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='scratchy']/lrm (origin=local/crmd/1148, version=0.247.18): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:320 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=p_ip_dnsmasq_last_0, magic=0:7;13:151:7:a3d5b99d-d9b9-4760-94ed-a977b296fcee, cib=0.247.18) : Resource op removal
Mar 30 08:17:56 walter crmd: [3503]: info: erase_xpath_callback: Deletion of "//node_state[@uname='scratchy']/lrm": ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1160: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter crmd: [3503]: info: te_update_diff: Detected LRM refresh - 8 resources updated: Skipping all resource events
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:276 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.247.19) : LRM Refresh
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1161: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='walter']/lrm (origin=local/crmd/1150, version=0.247.20): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:320 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=p_ip_dnsmasq_last_0, magic=0:0;42:49:0:cc023a3a-5b96-4d1f-b376-25a6b68b3761, cib=0.247.20) : Resource op removal
Mar 30 08:17:56 walter crmd: [3503]: info: erase_xpath_callback: Deletion of "//node_state[@uname='walter']/lrm": ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1162: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter attrd: [3501]: notice: attrd_local_callback: Sending full refresh (origin=crmd)
Mar 30 08:17:56 walter attrd: [3501]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 30 08:17:56 walter crmd: [3503]: info: te_update_diff: Detected LRM refresh - 8 resources updated: Skipping all resource events
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:276 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.247.21) : LRM Refresh
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1163: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='itchy']/lrm (origin=local/crmd/1152, version=0.247.22): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:320 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=p_ip_dnsmasq_last_0, magic=0:7;10:151:7:a3d5b99d-d9b9-4760-94ed-a977b296fcee, cib=0.247.22) : Resource op removal
Mar 30 08:17:56 walter crmd: [3503]: info: erase_xpath_callback: Deletion of "//node_state[@uname='itchy']/lrm": ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1164: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter crmd: [3503]: info: te_update_diff: Detected LRM refresh - 8 resources updated: Skipping all resource events
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:276 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.247.23) : LRM Refresh
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1165: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='butthead']/lrm (origin=local/crmd/1154, version=0.247.24): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:320 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=p_drbd_iscsivg01:0_last_failure_0, magic=0:5;10:48:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761, cib=0.247.24) : Resource op removal
Mar 30 08:17:56 walter crmd: [3503]: info: erase_xpath_callback: Deletion of "//node_state[@uname='butthead']/lrm": ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1166: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter crmd: [3503]: info: te_update_diff: Detected LRM refresh - 8 resources updated: Skipping all resource events
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:276 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.247.25) : LRM Refresh
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1167: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter attrd: [3501]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-p_ip_dnsmasq (1332707473)
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1156, version=0.247.28): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: WARN: match_down_event: No match for shutdown action on timmy.damnet.us
Mar 30 08:17:56 walter crmd: [3503]: info: te_update_diff: Stonith/shutdown of timmy.damnet.us not matched
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: te_update_diff:234 - Triggered transition abort (complete=1, tag=node_state, id=timmy.damnet.us, magic=NA, cib=0.247.29) : Node failure
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke: Query 1168: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:56 walter cib: [3499]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/1158, version=0.247.30): ok (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: do_pe_invoke_callback: Invoking the PE: query=1168, ref=pe_calc-dc-1333120676-718, seq=434460, quorate=1
Mar 30 08:17:56 walter pengine: [3502]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 30 08:17:56 walter pengine: [3502]: notice: unpack_rsc_op: Hard error - p_drbd_iscsivg01:0_last_failure_0 failed with rc=5: Preventing ms_drbd_iscsivg01 from re-starting on butthead
Mar 30 08:17:56 walter pengine: [3502]: notice: unpack_rsc_op: Hard error - p_drbd_iscsivg01:0_last_failure_0 failed with rc=5: Preventing ms_drbd_iscsivg01 from re-starting on walter
Mar 30 08:17:56 walter pengine: [3502]: notice: unpack_rsc_op: Operation p_dnsmasq_last_failure_0 found resource p_dnsmasq active on scratchy
Mar 30 08:17:56 walter pengine: [3502]: notice: unpack_rsc_op: Operation p_drbd_iscsivg01:0_last_failure_0 found resource p_drbd_iscsivg01:0 active on scratchy
Mar 30 08:17:56 walter pengine: [3502]: notice: unpack_rsc_op: Operation p_dnsmasq_last_failure_0 found resource p_dnsmasq active on itchy
Mar 30 08:17:56 walter pengine: [3502]: notice: unpack_rsc_op: Hard error - p_drbd_iscsivg01:1_monitor_29000 failed with rc=5: Preventing ms_drbd_iscsivg01 from re-starting on itchy
Mar 30 08:17:56 walter pengine: [3502]: WARN: unpack_rsc_op: Processing failed op p_drbd_iscsivg01:1_monitor_29000 on itchy: not installed (5)
Mar 30 08:17:56 walter pengine: [3502]: WARN: common_apply_stickiness: Forcing ms_drbd_iscsivg01 away from itchy after 1000000 failures (max=1000000)
Mar 30 08:17:56 walter pengine: [3502]: WARN: common_apply_stickiness: Forcing ms_drbd_iscsivg01 away from itchy after 1000000 failures (max=1000000)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_drbd_iscsivg01:0#011(Slave scratchy)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_drbd_iscsivg01:1#011(Stopped)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_lvm_iscsivg01#011(Stopped)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_target_iscsivg01#011(Stopped)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_lu_iscsivg01_lun1#011(Stopped)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_lu_iscsivg01_lun2#011(Stopped)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_ip_itchy-scratchy01#011(Stopped)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_ip_dnsmasq#011(Started butthead)
Mar 30 08:17:56 walter pengine: [3502]: notice: LogActions: Leave p_dnsmasq#011(Started butthead)
Mar 30 08:17:56 walter crmd: [3503]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 30 08:17:56 walter crmd: [3503]: info: unpack_graph: Unpacked transition 115: 10 actions in 10 synapses
Mar 30 08:17:56 walter crmd: [3503]: info: do_te_invoke: Processing graph 115 (ref=pe_calc-dc-1333120676-718) derived from /var/lib/pengine/pe-input-56.bz2
Mar 30 08:17:56 walter crmd: [3503]: info: te_rsc_command: Initiating action 11: monitor p_drbd_iscsivg01:0_monitor_0 on timmy
Mar 30 08:17:56 walter crmd: [3503]: info: te_rsc_command: Initiating action 12: monitor p_lvm_iscsivg01_monitor_0 on timmy
Mar 30 08:17:56 walter crmd: [3503]: info: te_rsc_command: Initiating action 13: monitor p_target_iscsivg01_monitor_0 on timmy
Mar 30 08:17:56 walter crmd: [3503]: info: te_rsc_command: Initiating action 14: monitor p_lu_iscsivg01_lun1_monitor_0 on timmy
Mar 30 08:17:56 walter crmd: [3503]: info: te_rsc_command: Initiating action 15: monitor p_lu_iscsivg01_lun2_monitor_0 on timmy
Mar 30 08:17:56 walter crmd: [3503]: info: te_rsc_command: Initiating action 16: monitor p_ip_itchy-scratchy01_monitor_0 on timmy
Mar 30 08:17:56 walter crmd: [3503]: info: te_rsc_command: Initiating action 17: monitor p_ip_dnsmasq_monitor_0 on timmy
Mar 30 08:17:56 walter crmd: [3503]: info: te_rsc_command: Initiating action 18: monitor p_dnsmasq_monitor_0 on timmy
Mar 30 08:17:56 walter crmd: [3503]: WARN: status_from_rc: Action 11 (p_drbd_iscsivg01:0_monitor_0) on timmy failed (target: 7 vs. rc: 5): Error
Mar 30 08:17:56 walter crmd: [3503]: info: abort_transition_graph: match_graph_event:277 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=p_drbd_iscsivg01:0_last_failure_0, magic=0:5;11:115:7:cc023a3a-5b96-4d1f-b376-25a6b68b3761, cib=0.247.41) : Event failed
Mar 30 08:17:56 walter crmd: [3503]: info: update_abort_priority: Abort priority upgraded from 0 to 1
Mar 30 08:17:56 walter crmd: [3503]: info: update_abort_priority: Abort action done superceeded by restart
Mar 30 08:17:56 walter crmd: [3503]: info: match_graph_event: Action p_drbd_iscsivg01:0_monitor_0 (11) confirmed on timmy (rc=4)
Mar 30 08:17:56 walter pengine: [3502]: notice: process_pe_message: Transition 115: PEngine Input stored in: /var/lib/pengine/pe-input-56.bz2
Mar 30 08:17:56 walter crmd: [3503]: info: match_graph_event: Action p_target_iscsivg01_monitor_0 (13) confirmed on timmy (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: match_graph_event: Action p_lu_iscsivg01_lun1_monitor_0 (14) confirmed on timmy (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: match_graph_event: Action p_lu_iscsivg01_lun2_monitor_0 (15) confirmed on timmy (rc=0)
Mar 30 08:17:56 walter crmd: [3503]: info: match_graph_event: Action p_lvm_iscsivg01_monitor_0 (12) confirmed on timmy (rc=0)
Mar 30 08:17:57 walter crmd: [3503]: info: match_graph_event: Action p_ip_itchy-scratchy01_monitor_0 (16) confirmed on timmy (rc=0)
Mar 30 08:17:57 walter crmd: [3503]: info: match_graph_event: Action p_ip_dnsmasq_monitor_0 (17) confirmed on timmy (rc=0)
Mar 30 08:17:57 walter crmd: [3503]: info: match_graph_event: Action p_dnsmasq_monitor_0 (18) confirmed on timmy (rc=0)
Mar 30 08:17:57 walter crmd: [3503]: info: te_rsc_command: Initiating action 10: probe_complete probe_complete on timmy - no waiting
Mar 30 08:17:57 walter crmd: [3503]: info: run_graph: ====================================================
Mar 30 08:17:57 walter crmd: [3503]: notice: run_graph: Transition 115 (Complete=9, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-input-56.bz2): Stopped
Mar 30 08:17:57 walter crmd: [3503]: info: te_graph_trigger: Transition 115 is now complete
Mar 30 08:17:57 walter crmd: [3503]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 30 08:17:57 walter crmd: [3503]: info: do_state_transition: All 5 cluster nodes are eligible to run resources.
Mar 30 08:17:57 walter crmd: [3503]: info: do_pe_invoke: Query 1169: Requesting the current CIB: S_POLICY_ENGINE
Mar 30 08:17:57 walter crmd: [3503]: info: do_pe_invoke_callback: Invoking the PE: query=1169, ref=pe_calc-dc-1333120677-728, seq=434460, quorate=1
Mar 30 08:17:57 walter pengine: [3502]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 30 08:17:57 walter pengine: [3502]: notice: unpack_rsc_op: Hard error - p_drbd_iscsivg01:0_last_failure_0 failed with rc=5: Preventing ms_drbd_iscsivg01 from re-starting on butthead
Mar 30 08:17:57 walter pengine: [3502]: notice: unpack_rsc_op: Hard error - p_drbd_iscsivg01:0_last_failure_0 failed with rc=5: Preventing ms_drbd_iscsivg01 from re-starting on walter
Mar 30 08:17:57 walter pengine: [3502]: notice: unpack_rsc_op: Operation p_dnsmasq_last_failure_0 found resource p_dnsmasq active on scratchy
Mar 30 08:17:57 walter pengine: [3502]: notice: unpack_rsc_op: Operation p_drbd_iscsivg01:0_last_failure_0 found resource p_drbd_iscsivg01:0 active on scratchy
Mar 30 08:17:57 walter pengine: [3502]: notice: unpack_rsc_op: Operation p_dnsmasq_last_failure_0 found resource p_dnsmasq active on itchy
Mar 30 08:17:57 walter pengine: [3502]: notice: unpack_rsc_op: Hard error - p_drbd_iscsivg01:1_monitor_29000 failed with rc=5: Preventing ms_drbd_iscsivg01 from re-starting on itchy
Mar 30 08:17:57 walter pengine: [3502]: WARN: unpack_rsc_op: Processing failed op p_drbd_iscsivg01:1_monitor_29000 on itchy: not installed (5)
Mar 30 08:17:57 walter pengine: [3502]: notice: unpack_rsc_op: Hard error - p_drbd_iscsivg01:0_last_failure_0 failed with rc=5: Preventing ms_drbd_iscsivg01 from re-starting on timmy
Mar 30 08:17:57 walter pengine: [3502]: WARN: common_apply_stickiness: Forcing ms_drbd_iscsivg01 away from itchy after 1000000 failures (max=1000000)
Mar 30 08:17:57 walter pengine: [3502]: WARN: common_apply_stickiness: Forcing ms_drbd_iscsivg01 away from itchy after 1000000 failures (max=1000000)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_drbd_iscsivg01:0#011(Slave scratchy)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_drbd_iscsivg01:1#011(Stopped)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_lvm_iscsivg01#011(Stopped)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_target_iscsivg01#011(Stopped)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_lu_iscsivg01_lun1#011(Stopped)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_lu_iscsivg01_lun2#011(Stopped)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_ip_itchy-scratchy01#011(Stopped)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_ip_dnsmasq#011(Started butthead)
Mar 30 08:17:57 walter pengine: [3502]: notice: LogActions: Leave p_dnsmasq#011(Started butthead)
Mar 30 08:17:57 walter crmd: [3503]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 30 08:17:57 walter crmd: [3503]: info: unpack_graph: Unpacked transition 116: 1 actions in 1 synapses
Mar 30 08:17:57 walter crmd: [3503]: info: do_te_invoke: Processing graph 116 (ref=pe_calc-dc-1333120677-728) derived from /var/lib/pengine/pe-input-57.bz2
Mar 30 08:17:57 walter crmd: [3503]: info: te_rsc_command: Initiating action 10: probe_complete probe_complete on timmy - no waiting
Mar 30 08:17:57 walter crmd: [3503]: info: run_graph: ====================================================
Mar 30 08:17:57 walter crmd: [3503]: notice: run_graph: Transition 116 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-57.bz2): Complete
Mar 30 08:17:57 walter crmd: [3503]: info: te_graph_trigger: Transition 116 is now complete
Mar 30 08:17:57 walter crmd: [3503]: info: notify_crmd: Transition 116 status: done - <null>
Mar 30 08:17:57 walter crmd: [3503]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 30 08:17:57 walter crmd: [3503]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 30 08:17:57 walter pengine: [3502]: notice: process_pe_message: Transition 116: PEngine Input stored in: /var/lib/pengine/pe-input-57.bz2
Mar 30 08:21:50 walter cib: [3499]: info: cib_stats: Processed 102 operations (2058.00us average, 0% utilization) in the last 10min
More information about the Pacemaker
mailing list