Apr  1 01:29:03 nebula4 lvm[6759]: Waiting for next pre command
Apr  1 01:29:03 nebula4 lvm[6759]: read on PIPE 12: 4 bytes: status: 0
Apr  1 01:29:03 nebula4 lvm[6759]: background routine status was 0, sock_client=0x218eab0
Apr  1 01:29:03 nebula4 lvm[6759]: Send local reply
Apr  1 01:29:03 nebula4 lvm[6759]: Read on local socket 5, len = 31
Apr  1 01:29:03 nebula4 lvm[6759]: check_all_clvmds_running
Apr  1 01:29:03 nebula4 lvm[6759]: down_callback. node 1084811137, state = 3
Apr  1 01:29:03 nebula4 lvm[6759]: down_callback. node 1084811139, state = 3
Apr  1 01:29:03 nebula4 lvm[6759]: down_callback. node 1084811138, state = 3
Apr  1 01:29:03 nebula4 lvm[6759]: down_callback. node 1084811140, state = 3
Apr  1 01:29:03 nebula4 lvm[6759]: Got pre command condition...
Apr  1 01:29:03 nebula4 lvm[6759]: Writing status 0 down pipe 13
Apr  1 01:29:03 nebula4 lvm[6759]: Waiting to do post command - state = 0
Apr  1 01:29:03 nebula4 lvm[6759]: read on PIPE 12: 4 bytes: status: 0
Apr  1 01:29:03 nebula4 lvm[6759]: background routine status was 0, sock_client=0x218eab0
Apr  1 01:29:03 nebula4 lvm[6759]: distribute command: XID = 43973, flags=0x0 ()
Apr  1 01:29:03 nebula4 lvm[6759]: num_nodes = 4
Apr  1 01:29:03 nebula4 lvm[6759]: add_to_lvmqueue: cmd=0x218f100. client=0x218eab0, msg=0x218ebc0, len=31, csid=(nil), xid=43973
Apr  1 01:29:03 nebula4 lvm[6759]: Sending message to all cluster nodes
Apr  1 01:29:03 nebula4 lvm[6759]: process_work_item: local
Apr  1 01:29:03 nebula4 lvm[6759]: process_local_command: SYNC_NAMES (0x2d) msg=0x218ed00, msglen =31, client=0x218eab0
Apr  1 01:29:03 nebula4 lvm[6759]: Syncing device names
Apr  1 01:29:03 nebula4 lvm[6759]: Reply from node 40a8e784: 0 bytes
Apr  1 01:29:03 nebula4 lvm[6759]: Got 1 replies, expecting: 4
Apr  1 01:29:03 nebula4 lvm[6759]: LVM thread waiting for work
Apr  1 01:29:03 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811140 for 0. len 31
Apr  1 01:29:03 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811138 for 1084811140. len 18
Apr  1 01:29:03 nebula4 lvm[6759]: Reply from node 40a8e782: 0 bytes
Apr  1 01:29:03 nebula4 lvm[6759]: Got 2 replies, expecting: 4
Apr  1 01:29:03 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811139 for 1084811140. len 18
Apr  1 01:29:03 nebula4 lvm[6759]: Reply from node 40a8e783: 0 bytes
Apr  1 01:29:03 nebula4 lvm[6759]: Got 3 replies, expecting: 4
Apr  1 01:29:03 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811137 for 1084811140. len 18
Apr  1 01:29:03 nebula4 lvm[6759]: Reply from node 40a8e781: 0 bytes
Apr  1 01:29:03 nebula4 lvm[6759]: Got 4 replies, expecting: 4
Apr  1 01:29:03 nebula4 lvm[6759]: Got post command condition...
Apr  1 01:29:03 nebula4 lvm[6759]: Waiting for next pre command
Apr  1 01:29:03 nebula4 lvm[6759]: read on PIPE 12: 4 bytes: status: 0
Apr  1 01:29:03 nebula4 lvm[6759]: background routine status was 0, sock_client=0x218eab0
Apr  1 01:29:03 nebula4 lvm[6759]: Send local reply
Apr  1 01:29:03 nebula4 lvm[6759]: Read on local socket 5, len = 30
Apr  1 01:29:03 nebula4 lvm[6759]: Got pre command condition...
Apr  1 01:29:03 nebula4 lvm[6759]: doing PRE command LOCK_VG 'V_vg-one-2' at 6 (client=0x218eab0)
Apr  1 01:29:03 nebula4 lvm[6759]: unlock_resource: V_vg-one-2 lockid: 1
Apr  1 01:29:03 nebula4 lvm[6759]: Writing status 0 down pipe 13
Apr  1 01:29:03 nebula4 lvm[6759]: Waiting to do post command - state = 0
Apr  1 01:29:03 nebula4 lvm[6759]: read on PIPE 12: 4 bytes: status: 0
Apr  1 01:29:03 nebula4 lvm[6759]: background routine status was 0, sock_client=0x218eab0
Apr  1 01:29:03 nebula4 lvm[6759]: distribute command: XID = 43974, flags=0x1 (LOCAL)
Apr  1 01:29:03 nebula4 lvm[6759]: add_to_lvmqueue: cmd=0x218ed00. client=0x218eab0, msg=0x218ebc0, len=30, csid=(nil), xid=43974
Apr  1 01:29:03 nebula4 lvm[6759]: process_work_item: local
Apr  1 01:29:03 nebula4 lvm[6759]: process_local_command: LOCK_VG (0x33) msg=0x218ed40, msglen =30, client=0x218eab0
Apr  1 01:29:03 nebula4 lvm[6759]: do_lock_vg: resource 'V_vg-one-2', cmd = 0x6 LCK_VG (UNLOCK|VG), flags = 0x0 ( ), critical_section = 0
Apr  1 01:29:03 nebula4 lvm[6759]: Invalidating cached metadata for VG vg-one-2
Apr  1 01:29:03 nebula4 lvm[6759]: Reply from node 40a8e784: 0 bytes
Apr  1 01:29:03 nebula4 lvm[6759]: Got 1 replies, expecting: 1
Apr  1 01:29:03 nebula4 lvm[6759]: LVM thread waiting for work
Apr  1 01:29:03 nebula4 lvm[6759]: Got post command condition...
Apr  1 01:29:03 nebula4 lvm[6759]: Waiting for next pre command
Apr  1 01:29:03 nebula4 lvm[6759]: read on PIPE 12: 4 bytes: status: 0
Apr  1 01:29:03 nebula4 lvm[6759]: background routine status was 0, sock_client=0x218eab0
Apr  1 01:29:03 nebula4 lvm[6759]: Send local reply
Apr  1 01:29:03 nebula4 lvm[6759]: Read on local socket 5, len = 0
Apr  1 01:29:03 nebula4 lvm[6759]: EOF on local socket: inprogress=0
Apr  1 01:29:03 nebula4 lvm[6759]: Waiting for child thread
Apr  1 01:29:03 nebula4 lvm[6759]: Got pre command condition...
Apr  1 01:29:03 nebula4 lvm[6759]: Subthread finished
Apr  1 01:29:03 nebula4 lvm[6759]: Joined child thread
Apr  1 01:29:03 nebula4 lvm[6759]: ret == 0, errno = 0. removing client
Apr  1 01:29:03 nebula4 lvm[6759]: add_to_lvmqueue: cmd=0x218ebc0. client=0x218eab0, msg=(nil), len=0, csid=(nil), xid=43974
Apr  1 01:29:03 nebula4 lvm[6759]: process_work_item: free fd -1
Apr  1 01:29:03 nebula4 lvm[6759]: LVM thread waiting for work
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811138 for 0. len 31
Apr  1 01:29:16 nebula4 lvm[6759]: add_to_lvmqueue: cmd=0x218eea0. client=0x6a1d60, msg=0x7fff260dfcac, len=31, csid=0x7fff260de67c, xid=0
Apr  1 01:29:16 nebula4 lvm[6759]: process_work_item: remote
Apr  1 01:29:16 nebula4 lvm[6759]: process_remote_command SYNC_NAMES (0x2d) for clientid 0x5000000 XID 39602 on node 40a8e782
Apr  1 01:29:16 nebula4 lvm[6759]: Syncing device names
Apr  1 01:29:16 nebula4 lvm[6759]: LVM thread waiting for work
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811137 for 1084811138. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811139 for 1084811138. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811140 for 1084811138. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811137 for 0. len 31
Apr  1 01:29:16 nebula4 lvm[6759]: add_to_lvmqueue: cmd=0x218eab0. client=0x6a1d60, msg=0x7fff260dfcac, len=31, csid=0x7fff260de67c, xid=0
Apr  1 01:29:16 nebula4 lvm[6759]: process_work_item: remote
Apr  1 01:29:16 nebula4 lvm[6759]: process_remote_command SYNC_NAMES (0x2d) for clientid 0x5000000 XID 44354 on node 40a8e781
Apr  1 01:29:16 nebula4 lvm[6759]: Syncing device names
Apr  1 01:29:16 nebula4 lvm[6759]: LVM thread waiting for work
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811140 for 1084811137. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811138 for 1084811137. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811139 for 1084811137. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811138 for 0. len 31
Apr  1 01:29:16 nebula4 lvm[6759]: add_to_lvmqueue: cmd=0x218eea0. client=0x6a1d60, msg=0x7fff260dfcac, len=31, csid=0x7fff260de67c, xid=0
Apr  1 01:29:16 nebula4 lvm[6759]: process_work_item: remote
Apr  1 01:29:16 nebula4 lvm[6759]: process_remote_command SYNC_NAMES (0x2d) for clientid 0x5000000 XID 39605 on node 40a8e782
Apr  1 01:29:16 nebula4 lvm[6759]: Syncing device names
Apr  1 01:29:16 nebula4 lvm[6759]: LVM thread waiting for work
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811137 for 1084811138. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811139 for 1084811138. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811140 for 1084811138. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811137 for 0. len 31
Apr  1 01:29:16 nebula4 lvm[6759]: add_to_lvmqueue: cmd=0x218eab0. client=0x6a1d60, msg=0x7fff260dfcac, len=31, csid=0x7fff260de67c, xid=0
Apr  1 01:29:16 nebula4 lvm[6759]: process_work_item: remote
Apr  1 01:29:16 nebula4 lvm[6759]: process_remote_command SYNC_NAMES (0x2d) for clientid 0x5000000 XID 44357 on node 40a8e781
Apr  1 01:29:16 nebula4 lvm[6759]: Syncing device names
Apr  1 01:29:16 nebula4 lvm[6759]: LVM thread waiting for work
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811140 for 1084811137. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811138 for 1084811137. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811139 for 1084811137. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811138 for 0. len 31
Apr  1 01:29:16 nebula4 lvm[6759]: add_to_lvmqueue: cmd=0x218eea0. client=0x6a1d60, msg=0x7fff260dfcac, len=31, csid=0x7fff260de67c, xid=0
Apr  1 01:29:16 nebula4 lvm[6759]: process_work_item: remote
Apr  1 01:29:16 nebula4 lvm[6759]: process_remote_command SYNC_NAMES (0x2d) for clientid 0x5000000 XID 39608 on node 40a8e782
Apr  1 01:29:16 nebula4 lvm[6759]: Syncing device names
Apr  1 01:29:16 nebula4 lvm[6759]: LVM thread waiting for work
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811137 for 1084811138. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811137 for 0. len 31
Apr  1 01:29:16 nebula4 lvm[6759]: add_to_lvmqueue: cmd=0x218eab0. client=0x6a1d60, msg=0x7fff260dfcac, len=31, csid=0x7fff260de67c, xid=0
Apr  1 01:29:16 nebula4 lvm[6759]: process_work_item: remote
Apr  1 01:29:16 nebula4 lvm[6759]: process_remote_command SYNC_NAMES (0x2d) for clientid 0x5000000 XID 44360 on node 40a8e781
Apr  1 01:29:16 nebula4 lvm[6759]: Syncing device names
Apr  1 01:29:16 nebula4 lvm[6759]: LVM thread waiting for work
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811139 for 1084811138. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811140 for 1084811138. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811140 for 1084811137. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811138 for 1084811137. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811139 for 1084811137. len 18
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811137 for 0. len 31
Apr  1 01:29:16 nebula4 lvm[6759]: add_to_lvmqueue: cmd=0x218eea0. client=0x6a1d60, msg=0x7fff260dfcac, len=31, csid=0x7fff260de67c, xid=0
Apr  1 01:29:16 nebula4 lvm[6759]: process_work_item: remote
Apr  1 01:29:16 nebula4 lvm[6759]: process_remote_command SYNC_NAMES (0x2d) for clientid 0x5000000 XID 44363 on node 40a8e781
Apr  1 01:29:16 nebula4 lvm[6759]: Syncing device names
Apr  1 01:29:16 nebula4 lvm[6759]: LVM thread waiting for work
Apr  1 01:29:16 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811140 for 1084811137. len 18
Apr  1 01:29:23 nebula4 lvm[6759]: Got new connection on fd 5
Apr  1 01:29:23 nebula4 lvm[6759]: Read on local socket 5, len = 30
Apr  1 01:29:23 nebula4 lvm[6759]: creating pipe, [12, 13]
Apr  1 01:29:23 nebula4 lvm[6759]: Creating pre&post thread
Apr  1 01:29:23 nebula4 lvm[6759]: Created pre&post thread, state = 0
Apr  1 01:29:23 nebula4 lvm[6759]: in sub thread: client = 0x218eab0
Apr  1 01:29:23 nebula4 lvm[6759]: doing PRE command LOCK_VG 'V_vg-one-0' at 1 (client=0x218eab0)
Apr  1 01:29:23 nebula4 lvm[6759]: lock_resource 'V_vg-one-0', flags=0, mode=3
Apr  1 01:29:23 nebula4 lvm[6759]: lock_resource returning 0, lock_id=1
Apr  1 01:29:23 nebula4 lvm[6759]: Writing status 0 down pipe 13
Apr  1 01:29:23 nebula4 lvm[6759]: Waiting to do post command - state = 0
Apr  1 01:29:23 nebula4 lvm[6759]: read on PIPE 12: 4 bytes: status: 0
Apr  1 01:29:23 nebula4 lvm[6759]: background routine status was 0, sock_client=0x218eab0
Apr  1 01:29:23 nebula4 lvm[6759]: distribute command: XID = 43975, flags=0x1 (LOCAL)
Apr  1 01:29:23 nebula4 lvm[6759]: add_to_lvmqueue: cmd=0x218ed00. client=0x218eab0, msg=0x218ebc0, len=30, csid=(nil), xid=43975
Apr  1 01:29:23 nebula4 lvm[6759]: process_work_item: local
Apr  1 01:29:23 nebula4 lvm[6759]: process_local_command: LOCK_VG (0x33) msg=0x218ed40, msglen =30, client=0x218eab0
Apr  1 01:29:23 nebula4 lvm[6759]: do_lock_vg: resource 'V_vg-one-0', cmd = 0x1 LCK_VG (READ|VG), flags = 0x0 ( ), critical_section = 0
Apr  1 01:29:23 nebula4 lvm[6759]: Invalidating cached metadata for VG vg-one-0
Apr  1 01:29:23 nebula4 lvm[6759]: Reply from node 40a8e784: 0 bytes
Apr  1 01:29:23 nebula4 lvm[6759]: Got 1 replies, expecting: 1
Apr  1 01:29:23 nebula4 lvm[6759]: LVM thread waiting for work
Apr  1 01:29:23 nebula4 lvm[6759]: Got post command condition...
Apr  1 01:29:23 nebula4 lvm[6759]: Waiting for next pre command
Apr  1 01:29:23 nebula4 lvm[6759]: read on PIPE 12: 4 bytes: status: 0
Apr  1 01:29:23 nebula4 lvm[6759]: background routine status was 0, sock_client=0x218eab0
Apr  1 01:29:23 nebula4 lvm[6759]: Send local reply
Apr  1 01:29:23 nebula4 lvm[6759]: Read on local socket 5, len = 31
Apr  1 01:29:23 nebula4 lvm[6759]: check_all_clvmds_running
Apr  1 01:29:23 nebula4 lvm[6759]: down_callback. node 1084811137, state = 3
Apr  1 01:29:23 nebula4 lvm[6759]: down_callback. node 1084811139, state = 3
Apr  1 01:29:23 nebula4 lvm[6759]: down_callback. node 1084811138, state = 3
Apr  1 01:29:23 nebula4 lvm[6759]: down_callback. node 1084811140, state = 3
Apr  1 01:29:23 nebula4 lvm[6759]: Got pre command condition...
Apr  1 01:29:23 nebula4 lvm[6759]: Writing status 0 down pipe 13
Apr  1 01:29:23 nebula4 lvm[6759]: Waiting to do post command - state = 0
Apr  1 01:29:23 nebula4 lvm[6759]: read on PIPE 12: 4 bytes: status: 0
Apr  1 01:29:23 nebula4 lvm[6759]: background routine status was 0, sock_client=0x218eab0
Apr  1 01:29:23 nebula4 lvm[6759]: distribute command: XID = 43976, flags=0x0 ()
Apr  1 01:29:23 nebula4 lvm[6759]: num_nodes = 4
Apr  1 01:29:23 nebula4 lvm[6759]: add_to_lvmqueue: cmd=0x218f100. client=0x218eab0, msg=0x218ebc0, len=31, csid=(nil), xid=43976
Apr  1 01:29:23 nebula4 lvm[6759]: Sending message to all cluster nodes
Apr  1 01:29:23 nebula4 lvm[6759]: process_work_item: local
Apr  1 01:29:23 nebula4 lvm[6759]: process_local_command: SYNC_NAMES (0x2d) msg=0x218ed00, msglen =31, client=0x218eab0
Apr  1 01:29:23 nebula4 lvm[6759]: Syncing device names
Apr  1 01:29:23 nebula4 lvm[6759]: Reply from node 40a8e784: 0 bytes
Apr  1 01:29:23 nebula4 lvm[6759]: Got 1 replies, expecting: 4
Apr  1 01:29:23 nebula4 lvm[6759]: LVM thread waiting for work
Apr  1 01:29:26 nebula4 corosync[6411]:   [TOTEM ] A processor failed, forming new configuration.
Apr  1 01:29:29 nebula4 corosync[6411]:   [TOTEM ] A new membership (192.168.231.129:1204) was formed. Members left: 1084811138
Apr  1 01:29:29 nebula4 lvm[6759]: confchg callback. 0 joined, 1 left, 3 members
Apr  1 01:29:29 nebula4 crmd[6490]:  warning: match_down_event: No match for shutdown action on 1084811138
Apr  1 01:29:29 nebula4 crmd[6490]:   notice: peer_update_callback: Stonith/shutdown of nebula2 not matched
Apr  1 01:29:29 nebula4 crmd[6490]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Apr  1 01:29:29 nebula4 corosync[6411]:   [QUORUM] Members[3]: 1084811137 1084811139 1084811140
Apr  1 01:29:29 nebula4 corosync[6411]:   [MAIN  ] Completed service synchronization, ready to provide service.
Apr  1 01:29:29 nebula4 crmd[6490]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node nebula2[1084811138] - state is now lost (was member)
Apr  1 01:29:29 nebula4 crmd[6490]:  warning: match_down_event: No match for shutdown action on 1084811138
Apr  1 01:29:29 nebula4 pacemakerd[6483]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node nebula2[1084811138] - state is now lost (was member)
Apr  1 01:29:29 nebula4 crmd[6490]:   notice: peer_update_callback: Stonith/shutdown of nebula2 not matched
Apr  1 01:29:29 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811140 for 0. len 31
Apr  1 01:29:29 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811139 for 1084811137. len 18
Apr  1 01:29:29 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811139 for 0. len 31
Apr  1 01:29:29 nebula4 lvm[6759]: add_to_lvmqueue: cmd=0x218ed00. client=0x6a1d60, msg=0x7fff260dfcac, len=31, csid=0x7fff260de67c, xid=0
Apr  1 01:29:29 nebula4 lvm[6759]: process_work_item: remote
Apr  1 01:29:29 nebula4 lvm[6759]: process_remote_command SYNC_NAMES (0x2d) for clientid 0x5000000 XID 43802 on node 40a8e783
Apr  1 01:29:29 nebula4 lvm[6759]: Syncing device names
Apr  1 01:29:29 nebula4 lvm[6759]: LVM thread waiting for work
Apr  1 01:29:29 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811139 for 1084811140. len 18
Apr  1 01:29:29 nebula4 lvm[6759]: Reply from node 40a8e783: 0 bytes
Apr  1 01:29:29 nebula4 lvm[6759]: Got 2 replies, expecting: 4
Apr  1 01:29:29 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811137 for 1084811140. len 18
Apr  1 01:29:29 nebula4 lvm[6759]: Reply from node 40a8e781: 0 bytes
Apr  1 01:29:29 nebula4 lvm[6759]: Got 3 replies, expecting: 4
Apr  1 01:29:29 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811137 for 1084811139. len 18
Apr  1 01:29:29 nebula4 lvm[6759]: 1084811140 got message from nodeid 1084811140 for 1084811139. len 18
Apr  1 01:29:29 nebula4 dlm_controld[6737]: 50759 fence status 1084811138 receive 1 from 1084811137 walltime 1427844569 local 50759
Apr  1 01:29:29 nebula4 kernel: [50799.162381] dlm: closing connection to node 1084811138
Apr  1 01:29:29 nebula4 dlm_controld[6737]: 50759 fence status 1084811138 receive 1 from 1084811139 walltime 1427844569 local 50759
Apr  1 01:29:29 nebula4 dlm_controld[6737]: 50759 fence request 1084811138 pid 44527 nodedown time 1427844569 fence_all dlm_stonith
Apr  1 01:29:29 nebula4 dlm_controld[6737]: 50759 fence result 1084811138 pid 44527 result 1 exit status
Apr  1 01:29:29 nebula4 dlm_controld[6737]: 50759 fence status 1084811138 receive 1 from 1084811140 walltime 1427844569 local 50759
Apr  1 01:29:29 nebula4 dlm_controld[6737]: 50759 fence request 1084811138 no actor
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: pe_fence_node: Node nebula2 will be fenced because the node is no longer part of the cluster
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: determine_online_status: Node nebula2 is unclean
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: unpack_rsc_op: Processing failed op start for stonith-nebula4-IPMILAN on nebula3: unknown error (1)
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: unpack_rsc_op: Processing failed op start for stonith-nebula4-IPMILAN on nebula1: unknown error (1)
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: unpack_rsc_op: Processing failed op start for stonith-nebula4-IPMILAN on nebula2: unknown error (1)
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: common_apply_stickiness: Forcing stonith-nebula4-IPMILAN away from nebula1 after 1000000 failures (max=1000000)
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: common_apply_stickiness: Forcing stonith-nebula4-IPMILAN away from nebula2 after 1000000 failures (max=1000000)
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: common_apply_stickiness: Forcing stonith-nebula4-IPMILAN away from nebula3 after 1000000 failures (max=1000000)
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: custom_action: Action p_dlm:3_stop_0 on nebula2 is unrunnable (offline)
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: custom_action: Action p_dlm:3_stop_0 on nebula2 is unrunnable (offline)
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: custom_action: Action p_clvm:3_stop_0 on nebula2 is unrunnable (offline)
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: custom_action: Action p_clvm:3_stop_0 on nebula2 is unrunnable (offline)
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: custom_action: Action p_vg_one:3_stop_0 on nebula2 is unrunnable (offline)
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: custom_action: Action p_vg_one:3_stop_0 on nebula2 is unrunnable (offline)
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: custom_action: Action p_fs_one-datastores:3_stop_0 on nebula2 is unrunnable (offline)
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: custom_action: Action p_fs_one-datastores:3_stop_0 on nebula2 is unrunnable (offline)
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: custom_action: Action stonith-nebula1-IPMILAN_stop_0 on nebula2 is unrunnable (offline)
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: stage6: Scheduling Node nebula2 for STONITH
Apr  1 01:29:30 nebula4 pengine[6489]:   notice: LogActions: Stop    p_dlm:3#011(nebula2)
Apr  1 01:29:30 nebula4 pengine[6489]:   notice: LogActions: Stop    p_clvm:3#011(nebula2)
Apr  1 01:29:30 nebula4 pengine[6489]:   notice: LogActions: Stop    p_vg_one:3#011(nebula2)
Apr  1 01:29:30 nebula4 pengine[6489]:   notice: LogActions: Stop    p_fs_one-datastores:3#011(nebula2)
Apr  1 01:29:30 nebula4 pengine[6489]:   notice: LogActions: Move    stonith-nebula1-IPMILAN#011(Started nebula2 -> nebula3)
Apr  1 01:29:30 nebula4 pengine[6489]:  warning: process_pe_message: Calculated Transition 101: /var/lib/pacemaker/pengine/pe-warn-22.bz2
Apr  1 01:29:30 nebula4 crmd[6490]:   notice: te_fence_node: Executing reboot fencing operation (98) on nebula2 (timeout=30000)
Apr  1 01:29:30 nebula4 stonith-ng[6486]:   notice: handle_request: Client crmd.6490.2707e557 wants to fence (reboot) 'nebula2' with device '(any)'
Apr  1 01:29:30 nebula4 stonith-ng[6486]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for nebula2: 39eaf3a2-d7e0-417d-8a01-d2f373973d6b (0)
Apr  1 01:29:30 nebula4 stonith-ng[6486]:   notice: can_fence_host_with_device: stonith-nebula1-IPMILAN can not fence nebula2: static-list
Apr  1 01:29:30 nebula4 stonith-ng[6486]:   notice: can_fence_host_with_device: stonith-nebula2-IPMILAN can fence nebula2: static-list
Apr  1 01:29:30 nebula4 stonith-ng[6486]:   notice: can_fence_host_with_device: stonith-one-frontend can not fence nebula2: static-list
Apr  1 01:29:30 nebula4 stonith-ng[6486]:   notice: can_fence_host_with_device: stonith-nebula3-IPMILAN can not fence nebula2: static-list
Apr  1 01:29:32 nebula4 stonith-ng[6486]:   notice: remote_op_done: Operation reboot of nebula2 by nebula3 for crmd.6490@nebula4.39eaf3a2: OK
Apr  1 01:29:32 nebula4 crmd[6490]:   notice: tengine_stonith_callback: Stonith operation 2/98:101:0:28913388-04df-49cb-9927-362b21a74014: OK (0)
Apr  1 01:29:32 nebula4 crmd[6490]:   notice: tengine_stonith_notify: Peer nebula2 was terminated (reboot) by nebula3 for nebula4: OK (ref=39eaf3a2-d7e0-417d-8a01-d2f373973d6b) by client crmd.6490
Apr  1 01:29:32 nebula4 crmd[6490]:   notice: te_rsc_command: Initiating action 91: start stonith-nebula1-IPMILAN_start_0 on nebula3
Apr  1 01:29:33 nebula4 crmd[6490]:   notice: run_graph: Transition 101 (Complete=13, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-22.bz2): Stopped
Apr  1 01:29:33 nebula4 pengine[6489]:  warning: unpack_rsc_op: Processing failed op start for stonith-nebula4-IPMILAN on nebula3: unknown error (1)
Apr  1 01:29:33 nebula4 pengine[6489]:  warning: unpack_rsc_op: Processing failed op start for stonith-nebula4-IPMILAN on nebula1: unknown error (1)
Apr  1 01:29:33 nebula4 pengine[6489]:  warning: common_apply_stickiness: Forcing stonith-nebula4-IPMILAN away from nebula1 after 1000000 failures (max=1000000)
Apr  1 01:29:33 nebula4 pengine[6489]:  warning: common_apply_stickiness: Forcing stonith-nebula4-IPMILAN away from nebula3 after 1000000 failures (max=1000000)
Apr  1 01:29:33 nebula4 pengine[6489]:   notice: process_pe_message: Calculated Transition 102: /var/lib/pacemaker/pengine/pe-input-129.bz2
Apr  1 01:29:33 nebula4 crmd[6490]:   notice: te_rsc_command: Initiating action 88: monitor stonith-nebula1-IPMILAN_monitor_3600000 on nebula3
Apr  1 01:29:34 nebula4 crmd[6490]:   notice: run_graph: Transition 102 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-129.bz2): Complete
Apr  1 01:29:34 nebula4 crmd[6490]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Apr  1 01:30:01 nebula4 CRON[44640]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then munin-run apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then munin-run apt update 7200 12 >/dev/null; fi)
Apr  1 01:30:25 nebula4 dlm_controld[6737]: 50815 datastores wait for fencing
Apr  1 01:30:25 nebula4 dlm_controld[6737]: 50815 clvmd wait for fencing
Apr  1 01:30:29 nebula4 lvm[6759]: Request timed-out (send: 1427844563, now: 1427844629)
Apr  1 01:30:29 nebula4 lvm[6759]: Request timed-out. padding
Apr  1 01:30:29 nebula4 lvm[6759]: down_callback. node 1084811137, state = 3
Apr  1 01:30:29 nebula4 lvm[6759]: Checking for a reply from 40a8e781
Apr  1 01:30:29 nebula4 lvm[6759]: down_callback. node 1084811139, state = 3
Apr  1 01:30:29 nebula4 lvm[6759]: Checking for a reply from 40a8e783
Apr  1 01:30:29 nebula4 lvm[6759]: down_callback. node 1084811138, state = 1
Apr  1 01:30:29 nebula4 lvm[6759]: down_callback. node 1084811140, state = 3
Apr  1 01:30:29 nebula4 lvm[6759]: Checking for a reply from 40a8e784
Apr  1 01:30:29 nebula4 lvm[6759]: Got post command condition...
Apr  1 01:30:29 nebula4 lvm[6759]: Waiting for next pre command
Apr  1 01:30:29 nebula4 lvm[6759]: read on PIPE 12: 4 bytes: status: 0
Apr  1 01:30:29 nebula4 lvm[6759]: background routine status was 0, sock_client=0x218eab0
Apr  1 01:30:29 nebula4 lvm[6759]: Send local reply
Apr  1 01:30:29 nebula4 lvm[6759]: Read on local socket 5, len = 30
Apr  1 01:30:29 nebula4 lvm[6759]: Got pre command condition...
Apr  1 01:30:29 nebula4 lvm[6759]: doing PRE command LOCK_VG 'V_vg-one-0' at 6 (client=0x218eab0)
Apr  1 01:30:29 nebula4 lvm[6759]: unlock_resource: V_vg-one-0 lockid: 1
Apr  1 01:40:01 nebula4 CRON[47640]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then munin-run apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then munin-run apt update 7200 12 >/dev/null; fi)
Apr  1 01:44:34 nebula4 crmd[6490]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Apr  1 01:44:34 nebula4 pengine[6489]:  warning: unpack_rsc_op: Processing failed op start for stonith-nebula4-IPMILAN on nebula3: unknown error (1)
Apr  1 01:44:34 nebula4 pengine[6489]:  warning: unpack_rsc_op: Processing failed op start for stonith-nebula4-IPMILAN on nebula1: unknown error (1)
Apr  1 01:44:34 nebula4 pengine[6489]:  warning: common_apply_stickiness: Forcing stonith-nebula4-IPMILAN away from nebula1 after 1000000 failures (max=1000000)
Apr  1 01:44:34 nebula4 pengine[6489]:  warning: common_apply_stickiness: Forcing stonith-nebula4-IPMILAN away from nebula3 after 1000000 failures (max=1000000)
Apr  1 01:44:34 nebula4 pengine[6489]:   notice: process_pe_message: Calculated Transition 103: /var/lib/pacemaker/pengine/pe-input-130.bz2
Apr  1 01:44:34 nebula4 crmd[6490]:   notice: run_graph: Transition 103 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-130.bz2): Complete
Apr  1 01:44:34 nebula4 crmd[6490]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Apr  1 01:45:01 nebula4 CRON[49089]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then munin-run apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then munin-run apt update 7200 12 >/dev/null; fi)
Apr  1 01:46:01 nebula4 CRON[570]: (root) CMD (if test -x /usr/sbin/apticron; then /usr/sbin/apticron --cron; else true; fi)
Apr  1 01:49:20 nebula4 lvm[6759]: Got new connection on fd 17
Apr  1 01:49:20 nebula4 lvm[6759]: Read on local socket 17, len = 30
Apr  1 01:49:20 nebula4 lvm[6759]: creating pipe, [18, 19]
Apr  1 01:49:20 nebula4 lvm[6759]: Creating pre&post thread
Apr  1 01:49:20 nebula4 lvm[6759]: Created pre&post thread, state = 0
Apr  1 01:49:20 nebula4 lvm[6759]: in sub thread: client = 0x218f1f0
Apr  1 01:49:20 nebula4 lvm[6759]: doing PRE command LOCK_VG 'V_vg-one-0' at 1 (client=0x218f1f0)
Apr  1 01:49:20 nebula4 lvm[6759]: lock_resource 'V_vg-one-0', flags=0, mode=3
