[ClusterLabs] Resources wont start on new node unless it is the only active node
Ryan Anstey
ryan at treasuremart.net
Tue Nov 8 13:54:32 EST 2016
I've been running a ceph cluster with pacemaker for a few months now.
Everything has been working normally, but when I added a fourth node it
won't work like the others, even though their OS is the same and the
configs are all synced via salt. I also don't understand pacemaker that
well since I followed a guide for it. If anyone could steer me in the right
direction I would greatly appreciate it. Thank you!
- My resources only start if the new node is the only active node.
- Once started on new node, if they are moved back to one of the original
nodes, it won't go back to the new one.
- My resources work 100% if I start them manually (without pacemaker).
- (In the logs/configs below, my resources are named "unifi", "rbd_unifi"
being the main one that's not working.)
Log when running cleaning up the resource on the NEW node:
Nov 08 09:25:20 h4 Filesystem(fs_unifi)[18044]: WARNING: Couldn't find
device [/dev/rbd/rbd/unifi]. Expected /dev/??? to exist
Nov 08 09:25:20 h4 lrmd[3564]: notice: lxc_unifi_monitor_0:18018:stderr [
unifi doesn't exist ]
Nov 08 09:25:20 h4 crmd[3567]: notice: Operation lxc_unifi_monitor_0: not
running (node=h4, call=484, rc=7, cib-update=390, confirmed=true)
Nov 08 09:25:20 h4 crmd[3567]: notice: h4-lxc_unifi_monitor_0:484 [ unifi
doesn't exist\n ]
Nov 08 09:25:20 h4 crmd[3567]: notice: Operation fs_unifi_monitor_0: not
running (node=h4, call=480, rc=7, cib-update=391, confirmed=true)
Nov 08 09:25:20 h4 crmd[3567]: notice: Operation rbd_unifi_monitor_0: not
running (node=h4, call=476, rc=7, cib-update=392, confirmed=true)
Log when running cleaning up the resource on the OLD node:
Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
167838209
Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
167838210
Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
167838212
Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
167838209
Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
167838209
Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
167838210
Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
167838212
Nov 08 09:21:18 h3 crmd[11394]: notice: State transition S_IDLE ->
S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
origin=abort_transition_graph ]
Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
167838210
Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
167838212
Nov 08 09:21:18 h3 cib[11389]: warning: A-Sync reply to crmd failed: No
message of desired type
Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
167838211
Nov 08 09:21:22 h3 crmd[11394]: notice: Notifications disabled
Nov 08 09:21:24 h3 crmd[11394]: notice: Notifications disabled
Nov 08 09:21:24 h3 crmd[11394]: warning: No match for shutdown action on
167838211
Nov 08 09:21:25 h3 crmd[11394]: notice: Notifications disabled
Nov 08 09:21:25 h3 crmd[11394]: warning: No match for shutdown action on
167838211
Nov 08 09:21:26 h3 pengine[11393]: notice: Start rbd_unifi (h3)
Nov 08 09:21:26 h3 pengine[11393]: notice: Start fs_unifi (h3)
Nov 08 09:21:26 h3 pengine[11393]: notice: Start lxc_unifi (h3)
Nov 08 09:21:26 h3 pengine[11393]: notice: Calculated Transition 119:
/var/lib/pacemaker/pengine/pe-input-739.bz2
Nov 08 09:21:26 h3 crmd[11394]: notice: Processing graph 119
(ref=pe_calc-dc-1478625686-648) derived from
/var/lib/pacemaker/pengine/pe-input-739.bz2
Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 12: monitor
rbd_unifi_monitor_0 on h4
Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 9: monitor
rbd_unifi_monitor_0 on h3 (local)
Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 6: monitor
rbd_unifi_monitor_0 on h2
Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 3: monitor
rbd_unifi_monitor_0 on h1
Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 13: monitor
fs_unifi_monitor_0 on h4
Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 10: monitor
fs_unifi_monitor_0 on h3 (local)
Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 7: monitor
fs_unifi_monitor_0 on h2
Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 4: monitor
fs_unifi_monitor_0 on h1
Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 14: monitor
lxc_unifi_monitor_0 on h4
Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 11: monitor
lxc_unifi_monitor_0 on h3 (local)
Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 8: monitor
lxc_unifi_monitor_0 on h2
Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 5: monitor
lxc_unifi_monitor_0 on h1
Nov 08 09:21:26 h3 crmd[11394]: notice: Operation fs_unifi_monitor_0: ok
(node=h3, call=446, rc=0, cib-update=767, confirmed=true)
Nov 08 09:21:26 h3 crmd[11394]: warning: Action 10 (fs_unifi_monitor_0) on
h3 failed (target: 7 vs. rc: 0): Error
Nov 08 09:21:26 h3 crmd[11394]: notice: Transition aborted by
fs_unifi_monitor_0 'create' on h3: Event failed
(magic=0:0;10:119:7:2bc5b523-b947-4035-b296-4c099646298b, cib=0.1445.8,
source=match_graph_event:381, 0)
Nov 08 09:21:26 h3 crmd[11394]: warning: Action 10 (fs_unifi_monitor_0) on
h3 failed (target: 7 vs. rc: 0): Error
Nov 08 09:21:26 h3 crmd[11394]: notice: Operation lxc_unifi_monitor_0: ok
(node=h3, call=450, rc=0, cib-update=768, confirmed=true)
Nov 08 09:21:26 h3 crmd[11394]: warning: Action 11 (lxc_unifi_monitor_0) on
h3 failed (target: 7 vs. rc: 0): Error
Nov 08 09:21:26 h3 crmd[11394]: warning: Action 11 (lxc_unifi_monitor_0) on
h3 failed (target: 7 vs. rc: 0): Error
Nov 08 09:21:26 h3 crmd[11394]: notice: Operation rbd_unifi_monitor_0: ok
(node=h3, call=442, rc=0, cib-update=769, confirmed=true)
Nov 08 09:21:26 h3 crmd[11394]: warning: Action 9 (rbd_unifi_monitor_0) on
h3 failed (target: 7 vs. rc: 0): Error
Nov 08 09:21:26 h3 crmd[11394]: warning: Action 9 (rbd_unifi_monitor_0) on
h3 failed (target: 7 vs. rc: 0): Error
Nov 08 09:21:26 h3 crmd[11394]: notice: Transition 119 (Complete=13,
Pending=0, Fired=0, Skipped=0, Incomplete=4,
Source=/var/lib/pacemaker/pengine/pe-input-739.bz2): Complete
Nov 08 09:21:26 h3 pengine[11393]: notice: Calculated Transition 120:
/var/lib/pacemaker/pengine/pe-input-740.bz2
Nov 08 09:21:26 h3 crmd[11394]: notice: Processing graph 120
(ref=pe_calc-dc-1478625686-661) derived from
/var/lib/pacemaker/pengine/pe-input-740.bz2
Nov 08 09:21:26 h3 crmd[11394]: notice: Transition 120 (Complete=0,
Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pacemaker/pengine/pe-input-740.bz2): Complete
Nov 08 09:21:26 h3 crmd[11394]: notice: State transition
S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
origin=notify_crmd ]
Output of crm configure show:
node 167838209: h1 \
attributes standby=off
node 167838210: h2 \
attributes standby=off
node 167838211: h3 \
attributes standby=off
node 167838212: h4 \
attributes standby=off
rsc_template fs Filesystem \
op start timeout=60s interval=0 \
op stop timeout=60s interval=0
rsc_template kvm VirtualDomain \
params hypervisor="qemu:///system" \
meta allow-migrate=true \
op start timeout=120s interval=0 \
op stop timeout=120s interval=0 \
op monitor timeout=30 interval=10 depth=0
rsc_template lxc lxc \
params use_screen=true \
op stop timeout=300s interval=0
primitive fs_first @fs \
params device="/dev/rbd/rbd/first" fstype=xfs
directory="/srv/first" \
meta target-role=Stopped
primitive fs_fs1 @fs \
params device="/dev/rbd/rbd/fs1" fstype=xfs
directory="/var/lib/lxc/fs1"
primitive fs_ld1 @fs \
params device="/dev/rbd/rbd/ld1" fstype=xfs
directory="/var/lib/lxc/ld1"
primitive fs_ld2 @fs \
params device="/dev/rbd/rbd/ld2" fstype=xfs
directory="/var/lib/lxc/ld2"
primitive fs_ld3 @fs \
params device="/dev/rbd/rbd/ld3" fstype=xfs
directory="/var/lib/lxc/ld3"
primitive fs_ls1 @fs \
params device="/dev/rbd/rbd/ls1" fstype=xfs
directory="/var/lib/lxc/ls1"
primitive fs_ls2 @fs \
params device="/dev/rbd/rbd/ls2" fstype=xfs
directory="/var/lib/lxc/ls2"
primitive fs_ls3 @fs \
params device="/dev/rbd/rbd/ls3" fstype=xfs
directory="/var/lib/lxc/ls3"
primitive fs_ls4 @fs \
params device="/dev/rbd/rbd/ls4" fstype=xfs
directory="/var/lib/lxc/ls4"
primitive fs_ls5 @fs \
params device="/dev/rbd/rbd/ls5" fstype=xfs
directory="/var/lib/lxc/ls5"
primitive fs_rdosrv @fs \
params device="/dev/rbd/rbd/rdosrv" fstype=xfs
directory="/var/lib/lxc/rdosrv"
primitive fs_unifi @fs \
params device="/dev/rbd/rbd/unifi" fstype=xfs
directory="/var/lib/lxc/unifi" \
meta target-role=Stopped is-managed=false
primitive kvm_foxy @kvm \
params config="/srv/cluster/containers/foxy.xml" \
utilization cpu=8 hv_memory=4000 \
meta target-role=Stopped is-managed=true
primitive kvm_ws5 @kvm \
params config="/srv/cluster/containers/ws5.xml" \
utilization cpu=4 hv_memory=4096 \
meta target-role=Started
primitive lxc_fs1 @lxc \
params container=fs1 config="/var/lib/lxc/fs1/config"
primitive lxc_ld1 @lxc \
params container=ld1 config="/var/lib/lxc/ld1/config" \
meta target-role=Stopped
primitive lxc_ld2 @lxc \
params container=ld2 config="/var/lib/lxc/ld2/config"
primitive lxc_ld3 @lxc \
params container=ld3 config="/var/lib/lxc/ld3/config"
primitive lxc_ls1 @lxc \
params container=ls1 config="/var/lib/lxc/ls1/config"
primitive lxc_ls2 @lxc \
params container=ls2 config="/var/lib/lxc/ls2/config"
primitive lxc_ls3 @lxc \
params container=ls3 config="/var/lib/lxc/ls3/config"
primitive lxc_ls4 @lxc \
params container=ls4 config="/var/lib/lxc/ls4/config"
primitive lxc_ls5 @lxc \
params container=ls5 config="/var/lib/lxc/ls5/config" \
meta target-role=Started
primitive lxc_rdosrv @lxc \
params container=rdosrv config="/var/lib/lxc/rdosrv/config"
primitive lxc_unifi @lxc \
params container=unifi config="/var/lib/lxc/unifi/config" \
meta target-role=Stopped is-managed=false
primitive rbd_first ocf:ceph:rbd \
params name=first \
meta target-role=Stopped
primitive rbd_fs1 ocf:ceph:rbd \
params name=fs1
primitive rbd_ld1 ocf:ceph:rbd \
params name=ld1
primitive rbd_ld2 ocf:ceph:rbd \
params name=ld2
primitive rbd_ld3 ocf:ceph:rbd \
params name=ld3
primitive rbd_ls1 ocf:ceph:rbd \
params name=ls1
primitive rbd_ls2 ocf:ceph:rbd \
params name=ls2
primitive rbd_ls3 ocf:ceph:rbd \
params name=ls3
primitive rbd_ls4 ocf:ceph:rbd \
params name=ls4
primitive rbd_ls5 ocf:ceph:rbd \
params name=ls5
primitive rbd_rdosrv ocf:ceph:rbd \
params name=rdosrv \
meta target-role=Started
primitive rbd_unifi ocf:ceph:rbd \
params name=unifi \
meta target-role=Started
group grp_first rbd_first fs_first \
meta is-managed=true
group grp_fs1 rbd_fs1 fs_fs1 lxc_fs1 \
meta is-managed=true target-role=Started
group grp_ld1 rbd_ld1 fs_ld1 lxc_ld1 \
meta target-role=Stopped
group grp_ld2 rbd_ld2 fs_ld2 lxc_ld2 \
meta target-role=Stopped
group grp_ld3 rbd_ld3 fs_ld3 lxc_ld3 \
meta target-role=Started
group grp_ls1 rbd_ls1 fs_ls1 lxc_ls1 \
meta target-role=Started
group grp_ls2 rbd_ls2 fs_ls2 lxc_ls2 \
meta is-managed=true
group grp_ls3 rbd_ls3 fs_ls3 lxc_ls3 \
meta target-role=Started
group grp_ls4 rbd_ls4 fs_ls4 lxc_ls4 \
meta target-role=Stopped
group grp_ls5 rbd_ls5 fs_ls5 lxc_ls5 \
meta target-role=Started
group grp_rdosrv rbd_rdosrv fs_rdosrv lxc_rdosrv \
meta target-role=Started is-managed=true
group grp_unifi rbd_unifi fs_unifi lxc_unifi \
meta target-role=Started
location cli-prefer-fs_ls5 fs_ls5 role=Started inf: h3
location cli-prefer-grp_fs1 grp_fs1 role=Started inf: h4
location cli-prefer-grp_ld1 grp_ld1 role=Started inf: h3
location cli-prefer-grp_ld2 grp_ld2 role=Started inf: h2
location cli-prefer-grp_ld3 grp_ld3 role=Started inf: h4
location cli-prefer-grp_ls1 grp_ls1 role=Started inf: h2
location cli-prefer-grp_ls2 grp_ls2 role=Started inf: h2
location cli-prefer-grp_ls3 grp_ls3 role=Started inf: h4
location cli-prefer-grp_ls4 grp_ls4 role=Started inf: h3
location cli-prefer-grp_ls5 grp_ls5 role=Started inf: h4
location cli-prefer-grp_rdosrv grp_rdosrv role=Started inf: h4
location cli-prefer-grp_unifi grp_unifi role=Started inf: h4
location cli-prefer-kvm_foxy kvm_foxy role=Started inf: h1
location cli-prefer-kvm_ws5 kvm_ws5 role=Started inf: h1
location cli-prefer-lxc_fs1 lxc_fs1 role=Started inf: h3
location cli-prefer-lxc_ld3 lxc_ld3 role=Started inf: h3
location cli-prefer-lxc_ls1 lxc_ls1 role=Started inf: h2
location cli-prefer-lxc_ls3 lxc_ls3 role=Started inf: h2
location cli-prefer-lxc_ls4 lxc_ls4 role=Started inf: h3
location cli-prefer-lxc_ls5 lxc_ls5 role=Started inf: h3
location cli-prefer-lxc_rdosrv lxc_rdosrv role=Started inf: h3
location cli-prefer-lxc_unifi lxc_unifi role=Started inf: h3
location cli-prefer-rbd_ls5 rbd_ls5 role=Started inf: h3
location cli-prefer-rbd_rdosrv rbd_rdosrv role=Started inf: h4
property cib-bootstrap-options: \
have-watchdog=false \
dc-version=1.1.13-20.1-6f22ad7 \
cluster-infrastructure=corosync \
cluster-name=hosteria \
stonith-enabled=false \
last-lrm-refresh=1478629707
rsc_defaults rsc-options: \
resource-stickiness=100
Pacemaker logs:
Nov 08 09:35:00 [3562] h4 cib: info: cib_process_request: Forwarding
cib_delete operation for section
//node_state[@uname='h4']//lrm_resource[@id='rbd_unifi'] to master
(origin=local/crmd/393)
Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1445.14 2
Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1445.15 0baf07ee77ccf8d61824f54c68f116b7
Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='rbd_unifi']
Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=15
Nov 08 09:35:00 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h1']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
origin=h1/crmd/345, version=0.1445.14)
Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1445.14 2
Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1445.15 (null)
Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='rbd_unifi']
Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=15
Nov 08 09:35:00 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h4']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
origin=h4/crmd/393, version=0.1445.14)
Nov 08 09:35:00 [3567] h4 crmd: info: delete_resource: Removing
resource rbd_unifi for d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 (root) on h4
Nov 08 09:35:00 [3567] h4 crmd: info: notify_deleted: Notifying
d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 on h4 that rbd_unifi was deleted
Nov 08 09:35:00 [3562] h4 cib: info: cib_process_request: Forwarding
cib_delete operation for section
//node_state[@uname='h4']//lrm_resource[@id='rbd_unifi'] to master
(origin=local/crmd/394)
Nov 08 09:35:00 [3562] h4 cib: info: cib_process_request: Forwarding
cib_modify operation for section crm_config to master
(origin=local/crmd/396)
Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1445.14 2
Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1445.15 (null)
Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='rbd_unifi']
Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=15
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h4']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
origin=h4/crmd/394, version=0.1445.15)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1445.15 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1446.0 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@epoch=1446, @num_updates=0
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: +
/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']:
@value=1478626500
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section crm_config: OK (rc=0, origin=h4/crmd/396,
version=0.1446.0)
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Forwarding
cib_delete operation for section
//node_state[@uname='h4']//lrm_resource[@id='fs_unifi'] to master
(origin=local/crmd/397)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1446.0 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1446.1 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='fs_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=1
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h4']//lrm_resource[@id='fs_unifi']: OK (rc=0,
origin=h4/crmd/397, version=0.1446.0)
Nov 08 09:35:01 [3567] h4 crmd: info: delete_resource: Removing
resource fs_unifi for d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 (root) on h4
Nov 08 09:35:01 [3567] h4 crmd: info: notify_deleted: Notifying
d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 on h4 that fs_unifi was deleted
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1446.0 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1446.1 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='rbd_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=1
Nov 08 09:35:01 [3562] h4 cib: info: cib_file_backup: Archived
previous version as /var/lib/pacemaker/cib/cib-64.raw
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h1']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
origin=h1/crmd/346, version=0.1446.1)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1446.1 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1447.0 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@epoch=1447, @num_updates=0
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: +
/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']:
@value=1478626267
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section crm_config: OK (rc=0, origin=h1/crmd/348,
version=0.1447.0)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1447.0 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1447.1 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='fs_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=1
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h1']//lrm_resource[@id='fs_unifi']: OK (rc=0,
origin=h1/crmd/349, version=0.1447.0)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1447.0 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1447.1 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='fs_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=1
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h1']//lrm_resource[@id='fs_unifi']: OK (rc=0,
origin=h1/crmd/350, version=0.1447.1)
Nov 08 09:35:01 [3562] h4 cib: info:
cib_file_write_with_digest: Wrote
version 0.1446.0 of the CIB to disk (digest:
13247e2b093b5d2dee902604bc9a3208)
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section crm_config: OK (rc=0, origin=h1/crmd/352,
version=0.1447.1)
Nov 08 09:35:01 [3562] h4 cib: info:
cib_file_write_with_digest: Reading
cluster configuration file /var/lib/pacemaker/cib/cib.111AJI (digest:
/var/lib/pacemaker/cib/cib.Zv2YLg)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1447.1 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1447.2 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='lxc_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=2
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h1']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
origin=h1/crmd/353, version=0.1447.1)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1447.1 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1447.2 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='lxc_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=2
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h1']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
origin=h1/crmd/354, version=0.1447.2)
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section crm_config: OK (rc=0, origin=h1/crmd/356,
version=0.1447.2)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1447.2 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1447.3 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='rbd_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=3
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h2']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
origin=h2/crmd/346, version=0.1447.2)
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Forwarding
cib_delete operation for section
//node_state[@uname='h4']//lrm_resource[@id='fs_unifi'] to master
(origin=local/crmd/398)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1447.2 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1447.3 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='fs_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=3
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h4']//lrm_resource[@id='fs_unifi']: OK (rc=0,
origin=h4/crmd/398, version=0.1447.3)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1447.3 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1447.4 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='rbd_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=4
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h2']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
origin=h2/crmd/347, version=0.1447.4)
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section crm_config: OK (rc=0, origin=h2/crmd/349,
version=0.1447.4)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1447.4 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1447.5 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='fs_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=5
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h2']//lrm_resource[@id='fs_unifi']: OK (rc=0,
origin=h2/crmd/350, version=0.1447.4)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1447.4 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1447.5 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='fs_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=5
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h2']//lrm_resource[@id='fs_unifi']: OK (rc=0,
origin=h2/crmd/351, version=0.1447.5)
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section crm_config: OK (rc=0, origin=h2/crmd/353,
version=0.1447.5)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1447.5 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1447.6 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='lxc_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=6
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h2']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
origin=h2/crmd/354, version=0.1447.5)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1447.5 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1447.6 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='lxc_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=6
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h2']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
origin=h2/crmd/355, version=0.1447.6)
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section crm_config: OK (rc=0, origin=h2/crmd/357,
version=0.1447.6)
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Forwarding
cib_modify operation for section crm_config to master
(origin=local/crmd/400)
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Forwarding
cib_delete operation for section
//node_state[@uname='h4']//lrm_resource[@id='lxc_unifi'] to master
(origin=local/crmd/401)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1447.6 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1448.0 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@epoch=1448, @num_updates=0
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: +
/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']:
@value=1478626501
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section crm_config: OK (rc=0, origin=h4/crmd/400,
version=0.1448.0)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1448.0 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1448.1 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='lxc_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=1
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h4']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
origin=h4/crmd/401, version=0.1448.0)
Nov 08 09:35:01 [3567] h4 crmd: info: delete_resource: Removing
resource lxc_unifi for d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 (root) on h4
Nov 08 09:35:01 [3567] h4 crmd: info: notify_deleted: Notifying
d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 on h4 that lxc_unifi was deleted
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Forwarding
cib_delete operation for section
//node_state[@uname='h4']//lrm_resource[@id='lxc_unifi'] to master
(origin=local/crmd/402)
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Forwarding
cib_modify operation for section crm_config to master
(origin=local/crmd/404)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1448.0 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1448.1 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='lxc_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=1
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h4']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
origin=h4/crmd/402, version=0.1448.1)
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section crm_config: OK (rc=0, origin=h4/crmd/404,
version=0.1448.1)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1448.1 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1448.2 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='rbd_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=2
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h3']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
origin=h3/crmd/771, version=0.1448.1)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1448.1 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1448.2 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='rbd_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=2
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h3']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
origin=h3/crmd/772, version=0.1448.2)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1448.2 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.0 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@epoch=1449, @num_updates=0
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: +
/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']:
@value=1478626267
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section crm_config: OK (rc=0, origin=h3/crmd/774,
version=0.1449.0)
Nov 08 09:35:01 [3562] h4 cib: info: cib_file_backup: Archived
previous version as /var/lib/pacemaker/cib/cib-65.raw
Nov 08 09:35:01 [3562] h4 cib: info:
cib_file_write_with_digest: Wrote
version 0.1449.0 of the CIB to disk (digest:
f6be1cc446b06cf40d434f709a1e660e)
Nov 08 09:35:01 [3562] h4 cib: info:
cib_file_write_with_digest: Reading
cluster configuration file /var/lib/pacemaker/cib/cib.dq077Z (digest:
/var/lib/pacemaker/cib/cib.HTyLSy)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1449.0 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.1 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='fs_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=1
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h3']//lrm_resource[@id='fs_unifi']: OK (rc=0,
origin=h3/crmd/789, version=0.1449.0)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1449.0 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.1 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='fs_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=1
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h3']//lrm_resource[@id='fs_unifi']: OK (rc=0,
origin=h3/crmd/790, version=0.1449.1)
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section crm_config: OK (rc=0, origin=h3/crmd/792,
version=0.1449.1)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1449.1 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.2 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='lxc_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=2
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h3']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
origin=h3/crmd/796, version=0.1449.1)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1449.1 2
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.2 (null)
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
/cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='lxc_unifi']
Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=2
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='h3']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
origin=h3/crmd/797, version=0.1449.2)
Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section crm_config: OK (rc=0, origin=h3/crmd/799,
version=0.1449.2)
Nov 08 09:35:05 [3564] h4 lrmd: info:
process_lrmd_get_rsc_info: Resource
'rbd_unifi' not found (34 active resources)
Nov 08 09:35:05 [3564] h4 lrmd: info:
process_lrmd_rsc_register: Added
'rbd_unifi' to the rsc list (35 active resources)
Nov 08 09:35:05 [3567] h4 crmd: info: do_lrm_rsc_op: Performing
key=12:121:7:2bc5b523-b947-4035-b296-4c099646298b op=rbd_unifi_monitor_0
Nov 08 09:35:05 [3564] h4 lrmd: info:
process_lrmd_get_rsc_info: Resource
'fs_unifi' not found (35 active resources)
Nov 08 09:35:05 [3564] h4 lrmd: info:
process_lrmd_rsc_register: Added
'fs_unifi' to the rsc list (36 active resources)
Nov 08 09:35:05 [3567] h4 crmd: info: do_lrm_rsc_op: Performing
key=13:121:7:2bc5b523-b947-4035-b296-4c099646298b op=fs_unifi_monitor_0
Nov 08 09:35:05 [3564] h4 lrmd: info:
process_lrmd_get_rsc_info: Resource
'lxc_unifi' not found (36 active resources)
Nov 08 09:35:05 [3564] h4 lrmd: info:
process_lrmd_rsc_register: Added
'lxc_unifi' to the rsc list (37 active resources)
Nov 08 09:35:05 [3567] h4 crmd: info: do_lrm_rsc_op: Performing
key=14:121:7:2bc5b523-b947-4035-b296-4c099646298b op=lxc_unifi_monitor_0
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1449.2 2
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.3 (null)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=3
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources:
<lrm_resource id="rbd_unifi" type="rbd" class="ocf" provider="ceph"/>
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
<lrm_rsc_op id="rbd_unifi_last_0" operation_key="rbd_unifi_monitor_0"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="3:121:7:2bc5b523-b947-4035-b296-4c099646298b"
transition-magic="0:7;3:121:7:2bc5b523-b947-4035-b296-4c099646298b"
on_node="h1" call-id="441" rc-code="7" op-status="0" interval="0"
last-run="147862
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
</lrm_resource>
Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=h1/crmd/357,
version=0.1449.3)
Nov 08 09:35:05 [3564] h4 lrmd: notice: operation_finished:
lxc_unifi_monitor_0:23285:stderr
[ unifi doesn't exist ]
Nov 08 09:35:05 [3567] h4 crmd: notice: process_lrm_event: Operation
lxc_unifi_monitor_0: not running (node=h4, call=499, rc=7, cib-update=405,
confirmed=true)
Nov 08 09:35:05 [3567] h4 crmd: notice: process_lrm_event:
h4-lxc_unifi_monitor_0:499
[ unifi doesn't exist\n ]
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1449.3 2
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.4 (null)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=4
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources:
<lrm_resource id="lxc_unifi" type="lxc" class="ocf" provider="heartbeat"/>
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
<lrm_rsc_op id="lxc_unifi_last_0" operation_key="lxc_unifi_monitor_0"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="8:121:7:2bc5b523-b947-4035-b296-4c099646298b"
transition-magic="0:7;8:121:7:2bc5b523-b947-4035-b296-4c099646298b"
on_node="h2" call-id="451" rc-code="7" op-status="0" interval="0"
last-run="147862
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
</lrm_resource>
Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=h2/crmd/358,
version=0.1449.4)
Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Forwarding
cib_modify operation for section status to master (origin=local/crmd/405)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1449.4 2
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.5 (null)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=5
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources:
<lrm_resource id="lxc_unifi" type="lxc" class="ocf" provider="heartbeat"/>
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
<lrm_rsc_op id="lxc_unifi_last_0" operation_key="lxc_unifi_monitor_0"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="14:121:7:2bc5b523-b947-4035-b296-4c099646298b"
transition-magic="0:7;14:121:7:2bc5b523-b947-4035-b296-4c099646298b"
on_node="h4" call-id="499" rc-code="7" op-status="0" interval="0"
last-run="1478
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
</lrm_resource>
Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=h4/crmd/405,
version=0.1449.5)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1449.5 2
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.6 (null)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=6
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources:
<lrm_resource id="fs_unifi" type="Filesystem" class="ocf"
provider="heartbeat"/>
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
<lrm_rsc_op id="fs_unifi_last_0" operation_key="fs_unifi_monitor_0"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="7:121:7:2bc5b523-b947-4035-b296-4c099646298b"
transition-magic="0:7;7:121:7:2bc5b523-b947-4035-b296-4c099646298b"
on_node="h2" call-id="447" rc-code="7" op-status="0" interval="0"
last-run="14786262
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
</lrm_resource>
Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=h2/crmd/359,
version=0.1449.6)
Nov 08 09:35:05 [3567] h4 crmd: notice: process_lrm_event: Operation
fs_unifi_monitor_0: not running (node=h4, call=495, rc=7, cib-update=406,
confirmed=true)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1449.6 2
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.7 (null)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=7
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources:
<lrm_resource id="lxc_unifi" type="lxc" class="ocf" provider="heartbeat"/>
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
<lrm_rsc_op id="lxc_unifi_last_0" operation_key="lxc_unifi_monitor_0"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="5:121:7:2bc5b523-b947-4035-b296-4c099646298b"
transition-magic="0:7;5:121:7:2bc5b523-b947-4035-b296-4c099646298b"
on_node="h1" call-id="449" rc-code="7" op-status="0" interval="0"
last-run="147862
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
</lrm_resource>
Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=h1/crmd/358,
version=0.1449.7)
Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Forwarding
cib_modify operation for section status to master (origin=local/crmd/406)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1449.7 2
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.8 (null)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=8
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources:
<lrm_resource id="fs_unifi" type="Filesystem" class="ocf"
provider="heartbeat"/>
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
<lrm_rsc_op id="fs_unifi_last_0" operation_key="fs_unifi_monitor_0"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="13:121:7:2bc5b523-b947-4035-b296-4c099646298b"
transition-magic="0:7;13:121:7:2bc5b523-b947-4035-b296-4c099646298b"
on_node="h4" call-id="495" rc-code="7" op-status="0" interval="0"
last-run="147862
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
</lrm_resource>
Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=h4/crmd/406,
version=0.1449.8)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1449.8 2
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.9 (null)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=9
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources:
<lrm_resource id="lxc_unifi" type="lxc" class="ocf" provider="heartbeat"/>
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
<lrm_rsc_op id="lxc_unifi_last_failure_0"
operation_key="lxc_unifi_monitor_0" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
transition-key="11:121:7:2bc5b523-b947-4035-b296-4c099646298b"
transition-magic="0:0;11:121:7:2bc5b523-b947-4035-b296-4c099646298b"
on_node="h3" call-id="465" rc-code="0" op-status="0" interval="0" last-r
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
<lrm_rsc_op id="lxc_unifi_last_0" operation_key="lxc_unifi_monitor_0"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="11:121:7:2bc5b523-b947-4035-b296-4c099646298b"
transition-magic="0:0;11:121:7:2bc5b523-b947-4035-b296-4c099646298b"
on_node="h3" call-id="465" rc-code="0" op-status="0" interval="0"
last-run="1478
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
</lrm_resource>
Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=h3/crmd/802,
version=0.1449.9)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1449.9 2
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.10 (null)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=10
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources:
<lrm_resource id="fs_unifi" type="Filesystem" class="ocf"
provider="heartbeat"/>
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
<lrm_rsc_op id="fs_unifi_last_0" operation_key="fs_unifi_monitor_0"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="4:121:7:2bc5b523-b947-4035-b296-4c099646298b"
transition-magic="0:7;4:121:7:2bc5b523-b947-4035-b296-4c099646298b"
on_node="h1" call-id="445" rc-code="7" op-status="0" interval="0"
last-run="14786262
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
</lrm_resource>
Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=h1/crmd/359,
version=0.1449.10)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1449.10 2
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.11 (null)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=11
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources:
<lrm_resource id="fs_unifi" type="Filesystem" class="ocf"
provider="heartbeat"/>
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
<lrm_rsc_op id="fs_unifi_last_failure_0"
operation_key="fs_unifi_monitor_0" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
transition-key="10:121:7:2bc5b523-b947-4035-b296-4c099646298b"
transition-magic="0:0;10:121:7:2bc5b523-b947-4035-b296-4c099646298b"
on_node="h3" call-id="461" rc-code="0" op-status="0" interval="0" last-run
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
<lrm_rsc_op id="fs_unifi_last_0" operation_key="fs_unifi_monitor_0"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="10:121:7:2bc5b523-b947-4035-b296-4c099646298b"
transition-magic="0:0;10:121:7:2bc5b523-b947-4035-b296-4c099646298b"
on_node="h3" call-id="461" rc-code="0" op-status="0" interval="0"
last-run="147862
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
</lrm_resource>
Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=h3/crmd/803,
version=0.1449.11)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1449.11 2
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.12 (null)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=12
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources:
<lrm_resource id="rbd_unifi" type="rbd" class="ocf" provider="ceph"/>
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
<lrm_rsc_op id="rbd_unifi_last_0" operation_key="rbd_unifi_monitor_0"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="6:121:7:2bc5b523-b947-4035-b296-4c099646298b"
transition-magic="0:7;6:121:7:2bc5b523-b947-4035-b296-4c099646298b"
on_node="h2" call-id="443" rc-code="7" op-status="0" interval="0"
last-run="147862
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
</lrm_resource>
Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=h2/crmd/360,
version=0.1449.12)
Nov 08 09:35:05 [3567] h4 crmd: notice: process_lrm_event: Operation
rbd_unifi_monitor_0: not running (node=h4, call=491, rc=7, cib-update=407,
confirmed=true)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1449.12 2
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.13 (null)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=13
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources:
<lrm_resource id="rbd_unifi" type="rbd" class="ocf" provider="ceph"/>
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
<lrm_rsc_op id="rbd_unifi_last_failure_0"
operation_key="rbd_unifi_monitor_0" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
transition-key="9:121:7:2bc5b523-b947-4035-b296-4c099646298b"
transition-magic="0:0;9:121:7:2bc5b523-b947-4035-b296-4c099646298b"
on_node="h3" call-id="457" rc-code="0" op-status="0" interval="0" last-run
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
<lrm_rsc_op id="rbd_unifi_last_0" operation_key="rbd_unifi_monitor_0"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="9:121:7:2bc5b523-b947-4035-b296-4c099646298b"
transition-magic="0:0;9:121:7:2bc5b523-b947-4035-b296-4c099646298b"
on_node="h3" call-id="457" rc-code="0" op-status="0" interval="0"
last-run="147862
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
</lrm_resource>
Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=h3/crmd/804,
version=0.1449.13)
Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Forwarding
cib_modify operation for section status to master (origin=local/crmd/407)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
0.1449.13 2
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
0.1449.14 (null)
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
@num_updates=14
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources:
<lrm_resource id="rbd_unifi" type="rbd" class="ocf" provider="ceph"/>
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
<lrm_rsc_op id="rbd_unifi_last_0" operation_key="rbd_unifi_monitor_0"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="12:121:7:2bc5b523-b947-4035-b296-4c099646298b"
transition-magic="0:7;12:121:7:2bc5b523-b947-4035-b296-4c099646298b"
on_node="h4" call-id="491" rc-code="7" op-status="0" interval="0"
last-run="1478
Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
</lrm_resource>
Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=h4/crmd/407,
version=0.1449.14)
Nov 08 09:35:10 [3562] h4 cib: info: cib_process_ping: Reporting
our current digest to h3: b5a63d2ac8b4835f8c8a2f310eb6648c for 0.1449.14
(0x12ddc90 0)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20161108/8dae3608/attachment-0002.html>
More information about the Users
mailing list