[ClusterLabs] Resources wont start on new node unless it is the only active node

Ken Gaillot kgaillot at redhat.com
Tue Nov 8 16:59:44 EST 2016


On 11/08/2016 12:54 PM, Ryan Anstey wrote:
> I've been running a ceph cluster with pacemaker for a few months now.
> Everything has been working normally, but when I added a fourth node it
> won't work like the others, even though their OS is the same and the
> configs are all synced via salt. I also don't understand pacemaker that
> well since I followed a guide for it. If anyone could steer me in the
> right direction I would greatly appreciate it. Thank you!
> 
> - My resources only start if the new node is the only active node.
> - Once started on new node, if they are moved back to one of the
> original nodes, it won't go back to the new one.
> - My resources work 100% if I start them manually (without pacemaker).
> - (In the logs/configs below, my resources are named "unifi",
> "rbd_unifi" being the main one that's not working.)

The key is all the location constraints starting with "cli-" in your
configuration. Such constraints were added automatically by command-line
tools, rather than added by you explicitly.

For example, Pacemaker has no concept of "moving" a resource. It places
all resources where they can best run, as specified by the
configuration. So, to move a resource, command-line tools add a location
constraint making the resource prefer a different node.

The downside is that the preference doesn't automatically go away. The
resource will continue to prefer the other node until you explicitly
remove the constraint.

Command-line tools that add such constraints generally provide some way
to clear them. If you clear all those constraints, resources will again
be placed on any node equally.

Separately, you also have a default resource stickiness of 100. That
means that even after you remove the constraints, resources that are
already running will tend to stay where they are. But if you stop and
start a resource, or add a new resource, it could start on a different node.

> 
> Log when running cleaning up the resource on the NEW node:
> 
> Nov 08 09:25:20 h4 Filesystem(fs_unifi)[18044]: WARNING: Couldn't find
> device [/dev/rbd/rbd/unifi]. Expected /dev/??? to exist
> Nov 08 09:25:20 h4 lrmd[3564]: notice: lxc_unifi_monitor_0:18018:stderr
> [ unifi doesn't exist ]
> Nov 08 09:25:20 h4 crmd[3567]: notice: Operation lxc_unifi_monitor_0:
> not running (node=h4, call=484, rc=7, cib-update=390, confirmed=true)
> Nov 08 09:25:20 h4 crmd[3567]: notice: h4-lxc_unifi_monitor_0:484 [
> unifi doesn't exist\n ]
> Nov 08 09:25:20 h4 crmd[3567]: notice: Operation fs_unifi_monitor_0: not
> running (node=h4, call=480, rc=7, cib-update=391, confirmed=true)
> Nov 08 09:25:20 h4 crmd[3567]: notice: Operation rbd_unifi_monitor_0:
> not running (node=h4, call=476, rc=7, cib-update=392, confirmed=true)
> 
> Log when running cleaning up the resource on the OLD node:  
> 
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838209
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838210
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838212
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838209
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838209
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838210
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838212
> Nov 08 09:21:18 h3 crmd[11394]: notice: State transition S_IDLE ->
> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> origin=abort_transition_graph ]
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838210
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838212
> Nov 08 09:21:18 h3 cib[11389]: warning: A-Sync reply to crmd failed: No
> message of desired type
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838211
> Nov 08 09:21:22 h3 crmd[11394]: notice: Notifications disabled
> Nov 08 09:21:24 h3 crmd[11394]: notice: Notifications disabled
> Nov 08 09:21:24 h3 crmd[11394]: warning: No match for shutdown action on
> 167838211
> Nov 08 09:21:25 h3 crmd[11394]: notice: Notifications disabled
> Nov 08 09:21:25 h3 crmd[11394]: warning: No match for shutdown action on
> 167838211
> Nov 08 09:21:26 h3 pengine[11393]: notice: Start   rbd_unifi        (h3)
> Nov 08 09:21:26 h3 pengine[11393]: notice: Start   fs_unifi        (h3)
> Nov 08 09:21:26 h3 pengine[11393]: notice: Start   lxc_unifi        (h3)
> Nov 08 09:21:26 h3 pengine[11393]: notice: Calculated Transition 119:
> /var/lib/pacemaker/pengine/pe-input-739.bz2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Processing graph 119
> (ref=pe_calc-dc-1478625686-648) derived from
> /var/lib/pacemaker/pengine/pe-input-739.bz2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 12: monitor
> rbd_unifi_monitor_0 on h4
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 9: monitor
> rbd_unifi_monitor_0 on h3 (local)
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 6: monitor
> rbd_unifi_monitor_0 on h2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 3: monitor
> rbd_unifi_monitor_0 on h1
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 13: monitor
> fs_unifi_monitor_0 on h4
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 10: monitor
> fs_unifi_monitor_0 on h3 (local)
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 7: monitor
> fs_unifi_monitor_0 on h2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 4: monitor
> fs_unifi_monitor_0 on h1
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 14: monitor
> lxc_unifi_monitor_0 on h4
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 11: monitor
> lxc_unifi_monitor_0 on h3 (local)
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 8: monitor
> lxc_unifi_monitor_0 on h2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 5: monitor
> lxc_unifi_monitor_0 on h1
> Nov 08 09:21:26 h3 crmd[11394]: notice: Operation fs_unifi_monitor_0: ok
> (node=h3, call=446, rc=0, cib-update=767, confirmed=true)
> Nov 08 09:21:26 h3 crmd[11394]: warning: Action 10 (fs_unifi_monitor_0)
> on h3 failed (target: 7 vs. rc: 0): Error
> Nov 08 09:21:26 h3 crmd[11394]: notice: Transition aborted by
> fs_unifi_monitor_0 'create' on h3: Event failed
> (magic=0:0;10:119:7:2bc5b523-b947-4035-b296-4c099646298b, cib=0.1445.8,
> source=match_graph_event:381, 0)
> Nov 08 09:21:26 h3 crmd[11394]: warning: Action 10 (fs_unifi_monitor_0)
> on h3 failed (target: 7 vs. rc: 0): Error
> Nov 08 09:21:26 h3 crmd[11394]: notice: Operation lxc_unifi_monitor_0:
> ok (node=h3, call=450, rc=0, cib-update=768, confirmed=true)
> Nov 08 09:21:26 h3 crmd[11394]: warning: Action 11 (lxc_unifi_monitor_0)
> on h3 failed (target: 7 vs. rc: 0): Error
> Nov 08 09:21:26 h3 crmd[11394]: warning: Action 11 (lxc_unifi_monitor_0)
> on h3 failed (target: 7 vs. rc: 0): Error
> Nov 08 09:21:26 h3 crmd[11394]: notice: Operation rbd_unifi_monitor_0:
> ok (node=h3, call=442, rc=0, cib-update=769, confirmed=true)
> Nov 08 09:21:26 h3 crmd[11394]: warning: Action 9 (rbd_unifi_monitor_0)
> on h3 failed (target: 7 vs. rc: 0): Error
> Nov 08 09:21:26 h3 crmd[11394]: warning: Action 9 (rbd_unifi_monitor_0)
> on h3 failed (target: 7 vs. rc: 0): Error
> Nov 08 09:21:26 h3 crmd[11394]: notice: Transition 119 (Complete=13,
> Pending=0, Fired=0, Skipped=0, Incomplete=4,
> Source=/var/lib/pacemaker/pengine/pe-input-739.bz2): Complete
> Nov 08 09:21:26 h3 pengine[11393]: notice: Calculated Transition 120:
> /var/lib/pacemaker/pengine/pe-input-740.bz2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Processing graph 120
> (ref=pe_calc-dc-1478625686-661) derived from
> /var/lib/pacemaker/pengine/pe-input-740.bz2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Transition 120 (Complete=0,
> Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-740.bz2): Complete
> Nov 08 09:21:26 h3 crmd[11394]: notice: State transition
> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
> origin=notify_crmd ]
> 
> Output of crm configure show:
> 
> node 167838209: h1 \
>         attributes standby=off
> node 167838210: h2 \
>         attributes standby=off
> node 167838211: h3 \
>         attributes standby=off
> node 167838212: h4 \
>         attributes standby=off
> rsc_template fs Filesystem \
>         op start timeout=60s interval=0 \
>         op stop timeout=60s interval=0
> rsc_template kvm VirtualDomain \
>         params hypervisor="qemu:///system" \
>         meta allow-migrate=true \
>         op start timeout=120s interval=0 \
>         op stop timeout=120s interval=0 \
>         op monitor timeout=30 interval=10 depth=0
> rsc_template lxc lxc \
>         params use_screen=true \
>         op stop timeout=300s interval=0
> primitive fs_first @fs \
>         params device="/dev/rbd/rbd/first" fstype=xfs
> directory="/srv/first" \
>         meta target-role=Stopped
> primitive fs_fs1 @fs \
>         params device="/dev/rbd/rbd/fs1" fstype=xfs
> directory="/var/lib/lxc/fs1"
> primitive fs_ld1 @fs \
>         params device="/dev/rbd/rbd/ld1" fstype=xfs
> directory="/var/lib/lxc/ld1"
> primitive fs_ld2 @fs \
>         params device="/dev/rbd/rbd/ld2" fstype=xfs
> directory="/var/lib/lxc/ld2"
> primitive fs_ld3 @fs \
>         params device="/dev/rbd/rbd/ld3" fstype=xfs
> directory="/var/lib/lxc/ld3"
> primitive fs_ls1 @fs \
>         params device="/dev/rbd/rbd/ls1" fstype=xfs
> directory="/var/lib/lxc/ls1"
> primitive fs_ls2 @fs \
>         params device="/dev/rbd/rbd/ls2" fstype=xfs
> directory="/var/lib/lxc/ls2"
> primitive fs_ls3 @fs \
>         params device="/dev/rbd/rbd/ls3" fstype=xfs
> directory="/var/lib/lxc/ls3"
> primitive fs_ls4 @fs \
>         params device="/dev/rbd/rbd/ls4" fstype=xfs
> directory="/var/lib/lxc/ls4"
> primitive fs_ls5 @fs \
>         params device="/dev/rbd/rbd/ls5" fstype=xfs
> directory="/var/lib/lxc/ls5"
> primitive fs_rdosrv @fs \
>         params device="/dev/rbd/rbd/rdosrv" fstype=xfs
> directory="/var/lib/lxc/rdosrv"
> primitive fs_unifi @fs \
>         params device="/dev/rbd/rbd/unifi" fstype=xfs
> directory="/var/lib/lxc/unifi" \
>         meta target-role=Stopped is-managed=false
> primitive kvm_foxy @kvm \
>         params config="/srv/cluster/containers/foxy.xml" \
>         utilization cpu=8 hv_memory=4000 \
>         meta target-role=Stopped is-managed=true
> primitive kvm_ws5 @kvm \
>         params config="/srv/cluster/containers/ws5.xml" \
>         utilization cpu=4 hv_memory=4096 \
>         meta target-role=Started
> primitive lxc_fs1 @lxc \
>         params container=fs1 config="/var/lib/lxc/fs1/config"
> primitive lxc_ld1 @lxc \
>         params container=ld1 config="/var/lib/lxc/ld1/config" \
>         meta target-role=Stopped
> primitive lxc_ld2 @lxc \
>         params container=ld2 config="/var/lib/lxc/ld2/config"
> primitive lxc_ld3 @lxc \
>         params container=ld3 config="/var/lib/lxc/ld3/config"
> primitive lxc_ls1 @lxc \
>         params container=ls1 config="/var/lib/lxc/ls1/config"
> primitive lxc_ls2 @lxc \
>         params container=ls2 config="/var/lib/lxc/ls2/config"
> primitive lxc_ls3 @lxc \
>         params container=ls3 config="/var/lib/lxc/ls3/config"
> primitive lxc_ls4 @lxc \
>         params container=ls4 config="/var/lib/lxc/ls4/config"
> primitive lxc_ls5 @lxc \
>         params container=ls5 config="/var/lib/lxc/ls5/config" \
>         meta target-role=Started
> primitive lxc_rdosrv @lxc \
>         params container=rdosrv config="/var/lib/lxc/rdosrv/config"
> primitive lxc_unifi @lxc \
>         params container=unifi config="/var/lib/lxc/unifi/config" \
>         meta target-role=Stopped is-managed=false
> primitive rbd_first ocf:ceph:rbd \
>         params name=first \
>         meta target-role=Stopped
> primitive rbd_fs1 ocf:ceph:rbd \
>         params name=fs1
> primitive rbd_ld1 ocf:ceph:rbd \
>         params name=ld1
> primitive rbd_ld2 ocf:ceph:rbd \
>         params name=ld2
> primitive rbd_ld3 ocf:ceph:rbd \
>         params name=ld3
> primitive rbd_ls1 ocf:ceph:rbd \
>         params name=ls1
> primitive rbd_ls2 ocf:ceph:rbd \
>         params name=ls2
> primitive rbd_ls3 ocf:ceph:rbd \
>         params name=ls3
> primitive rbd_ls4 ocf:ceph:rbd \
>         params name=ls4
> primitive rbd_ls5 ocf:ceph:rbd \
>         params name=ls5
> primitive rbd_rdosrv ocf:ceph:rbd \
>         params name=rdosrv \
>         meta target-role=Started
> primitive rbd_unifi ocf:ceph:rbd \
>         params name=unifi \
>         meta target-role=Started
> group grp_first rbd_first fs_first \
>         meta is-managed=true
> group grp_fs1 rbd_fs1 fs_fs1 lxc_fs1 \
>         meta is-managed=true target-role=Started
> group grp_ld1 rbd_ld1 fs_ld1 lxc_ld1 \
>         meta target-role=Stopped
> group grp_ld2 rbd_ld2 fs_ld2 lxc_ld2 \
>         meta target-role=Stopped
> group grp_ld3 rbd_ld3 fs_ld3 lxc_ld3 \
>         meta target-role=Started
> group grp_ls1 rbd_ls1 fs_ls1 lxc_ls1 \
>         meta target-role=Started
> group grp_ls2 rbd_ls2 fs_ls2 lxc_ls2 \
>         meta is-managed=true
> group grp_ls3 rbd_ls3 fs_ls3 lxc_ls3 \
>         meta target-role=Started
> group grp_ls4 rbd_ls4 fs_ls4 lxc_ls4 \
>         meta target-role=Stopped
> group grp_ls5 rbd_ls5 fs_ls5 lxc_ls5 \
>         meta target-role=Started
> group grp_rdosrv rbd_rdosrv fs_rdosrv lxc_rdosrv \
>         meta target-role=Started is-managed=true
> group grp_unifi rbd_unifi fs_unifi lxc_unifi \
>         meta target-role=Started
> location cli-prefer-fs_ls5 fs_ls5 role=Started inf: h3
> location cli-prefer-grp_fs1 grp_fs1 role=Started inf: h4
> location cli-prefer-grp_ld1 grp_ld1 role=Started inf: h3
> location cli-prefer-grp_ld2 grp_ld2 role=Started inf: h2
> location cli-prefer-grp_ld3 grp_ld3 role=Started inf: h4
> location cli-prefer-grp_ls1 grp_ls1 role=Started inf: h2
> location cli-prefer-grp_ls2 grp_ls2 role=Started inf: h2
> location cli-prefer-grp_ls3 grp_ls3 role=Started inf: h4
> location cli-prefer-grp_ls4 grp_ls4 role=Started inf: h3
> location cli-prefer-grp_ls5 grp_ls5 role=Started inf: h4
> location cli-prefer-grp_rdosrv grp_rdosrv role=Started inf: h4
> location cli-prefer-grp_unifi grp_unifi role=Started inf: h4
> location cli-prefer-kvm_foxy kvm_foxy role=Started inf: h1
> location cli-prefer-kvm_ws5 kvm_ws5 role=Started inf: h1
> location cli-prefer-lxc_fs1 lxc_fs1 role=Started inf: h3
> location cli-prefer-lxc_ld3 lxc_ld3 role=Started inf: h3
> location cli-prefer-lxc_ls1 lxc_ls1 role=Started inf: h2
> location cli-prefer-lxc_ls3 lxc_ls3 role=Started inf: h2
> location cli-prefer-lxc_ls4 lxc_ls4 role=Started inf: h3
> location cli-prefer-lxc_ls5 lxc_ls5 role=Started inf: h3
> location cli-prefer-lxc_rdosrv lxc_rdosrv role=Started inf: h3
> location cli-prefer-lxc_unifi lxc_unifi role=Started inf: h3
> location cli-prefer-rbd_ls5 rbd_ls5 role=Started inf: h3
> location cli-prefer-rbd_rdosrv rbd_rdosrv role=Started inf: h4
> property cib-bootstrap-options: \
>         have-watchdog=false \
>         dc-version=1.1.13-20.1-6f22ad7 \
>         cluster-infrastructure=corosync \
>         cluster-name=hosteria \
>         stonith-enabled=false \
>         last-lrm-refresh=1478629707
> rsc_defaults rsc-options: \
>         resource-stickiness=100
> 
> Pacemaker logs:
> 
> Nov 08 09:35:00 [3562] h4        cib:     info:
> cib_process_request:Forwarding cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='rbd_unifi'] to master
> (origin=local/crmd/393)
> Nov 08 09:35:00 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1445.14 2
> Nov 08 09:35:00 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1445.15 0baf07ee77ccf8d61824f54c68f116b7
> Nov 08 09:35:00 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='rbd_unifi']
> Nov 08 09:35:00 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=15
> Nov 08 09:35:00 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h1']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
> origin=h1/crmd/345, version=0.1445.14)
> Nov 08 09:35:00 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1445.14 2
> Nov 08 09:35:00 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1445.15 (null)
> Nov 08 09:35:00 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='rbd_unifi']
> Nov 08 09:35:00 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=15
> Nov 08 09:35:00 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
> origin=h4/crmd/393, version=0.1445.14)
> Nov 08 09:35:00 [3567] h4       crmd:     info: delete_resource:Removing
> resource rbd_unifi for d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 (root) on h4
> Nov 08 09:35:00 [3567] h4       crmd:     info: notify_deleted:Notifying
> d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 on h4 that rbd_unifi was deleted
> Nov 08 09:35:00 [3562] h4        cib:     info:
> cib_process_request:Forwarding cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='rbd_unifi'] to master
> (origin=local/crmd/394)
> Nov 08 09:35:00 [3562] h4        cib:     info:
> cib_process_request:Forwarding cib_modify operation for section
> crm_config to master (origin=local/crmd/396)
> Nov 08 09:35:00 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1445.14 2
> Nov 08 09:35:00 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1445.15 (null)
> Nov 08 09:35:00 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='rbd_unifi']
> Nov 08 09:35:00 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=15
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
> origin=h4/crmd/394, version=0.1445.15)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1445.15 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1446.0 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @epoch=1446, @num_updates=0
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+
>  /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']:
>  @value=1478626500
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section
> crm_config: OK (rc=0, origin=h4/crmd/396, version=0.1446.0)
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Forwarding cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='fs_unifi'] to master
> (origin=local/crmd/397)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1446.0 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1446.1 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='fs_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=1
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='fs_unifi']: OK (rc=0,
> origin=h4/crmd/397, version=0.1446.0)
> Nov 08 09:35:01 [3567] h4       crmd:     info: delete_resource:Removing
> resource fs_unifi for d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 (root) on h4
> Nov 08 09:35:01 [3567] h4       crmd:     info: notify_deleted:Notifying
> d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 on h4 that fs_unifi was deleted
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1446.0 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1446.1 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='rbd_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=1
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_file_backup:Archived
> previous version as /var/lib/pacemaker/cib/cib-64.raw
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h1']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
> origin=h1/crmd/346, version=0.1446.1)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1446.1 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1447.0 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @epoch=1447, @num_updates=0
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+
>  /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']:
>  @value=1478626267
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section
> crm_config: OK (rc=0, origin=h1/crmd/348, version=0.1447.0)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1447.0 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1447.1 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='fs_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=1
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h1']//lrm_resource[@id='fs_unifi']: OK (rc=0,
> origin=h1/crmd/349, version=0.1447.0)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1447.0 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1447.1 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='fs_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=1
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h1']//lrm_resource[@id='fs_unifi']: OK (rc=0,
> origin=h1/crmd/350, version=0.1447.1)
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_file_write_with_digest:Wrote version 0.1446.0 of the CIB to disk
> (digest: 13247e2b093b5d2dee902604bc9a3208)
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section
> crm_config: OK (rc=0, origin=h1/crmd/352, version=0.1447.1)
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_file_write_with_digest:Reading cluster configuration file
> /var/lib/pacemaker/cib/cib.111AJI (digest:
> /var/lib/pacemaker/cib/cib.Zv2YLg)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1447.1 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1447.2 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='lxc_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=2
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h1']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
> origin=h1/crmd/353, version=0.1447.1)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1447.1 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1447.2 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='lxc_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=2
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h1']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
> origin=h1/crmd/354, version=0.1447.2)
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section
> crm_config: OK (rc=0, origin=h1/crmd/356, version=0.1447.2)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1447.2 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1447.3 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='rbd_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=3
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h2']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
> origin=h2/crmd/346, version=0.1447.2)
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Forwarding cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='fs_unifi'] to master
> (origin=local/crmd/398)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1447.2 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1447.3 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='fs_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=3
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='fs_unifi']: OK (rc=0,
> origin=h4/crmd/398, version=0.1447.3)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1447.3 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1447.4 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='rbd_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=4
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h2']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
> origin=h2/crmd/347, version=0.1447.4)
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section
> crm_config: OK (rc=0, origin=h2/crmd/349, version=0.1447.4)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1447.4 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1447.5 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='fs_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=5
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h2']//lrm_resource[@id='fs_unifi']: OK (rc=0,
> origin=h2/crmd/350, version=0.1447.4)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1447.4 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1447.5 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='fs_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=5
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h2']//lrm_resource[@id='fs_unifi']: OK (rc=0,
> origin=h2/crmd/351, version=0.1447.5)
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section
> crm_config: OK (rc=0, origin=h2/crmd/353, version=0.1447.5)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1447.5 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1447.6 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='lxc_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=6
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h2']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
> origin=h2/crmd/354, version=0.1447.5)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1447.5 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1447.6 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='lxc_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=6
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h2']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
> origin=h2/crmd/355, version=0.1447.6)
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section
> crm_config: OK (rc=0, origin=h2/crmd/357, version=0.1447.6)
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Forwarding cib_modify operation for section
> crm_config to master (origin=local/crmd/400)
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Forwarding cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='lxc_unifi'] to master
> (origin=local/crmd/401)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1447.6 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1448.0 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @epoch=1448, @num_updates=0
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+
>  /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']:
>  @value=1478626501
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section
> crm_config: OK (rc=0, origin=h4/crmd/400, version=0.1448.0)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1448.0 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1448.1 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='lxc_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=1
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
> origin=h4/crmd/401, version=0.1448.0)
> Nov 08 09:35:01 [3567] h4       crmd:     info: delete_resource:Removing
> resource lxc_unifi for d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 (root) on h4
> Nov 08 09:35:01 [3567] h4       crmd:     info: notify_deleted:Notifying
> d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 on h4 that lxc_unifi was deleted
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Forwarding cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='lxc_unifi'] to master
> (origin=local/crmd/402)
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Forwarding cib_modify operation for section
> crm_config to master (origin=local/crmd/404)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1448.0 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1448.1 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='lxc_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=1
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
> origin=h4/crmd/402, version=0.1448.1)
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section
> crm_config: OK (rc=0, origin=h4/crmd/404, version=0.1448.1)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1448.1 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1448.2 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='rbd_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=2
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h3']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
> origin=h3/crmd/771, version=0.1448.1)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1448.1 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1448.2 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='rbd_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=2
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h3']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
> origin=h3/crmd/772, version=0.1448.2)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1448.2 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.0 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @epoch=1449, @num_updates=0
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+
>  /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']:
>  @value=1478626267
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section
> crm_config: OK (rc=0, origin=h3/crmd/774, version=0.1449.0)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_file_backup:Archived
> previous version as /var/lib/pacemaker/cib/cib-65.raw
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_file_write_with_digest:Wrote version 0.1449.0 of the CIB to disk
> (digest: f6be1cc446b06cf40d434f709a1e660e)
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_file_write_with_digest:Reading cluster configuration file
> /var/lib/pacemaker/cib/cib.dq077Z (digest:
> /var/lib/pacemaker/cib/cib.HTyLSy)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1449.0 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.1 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='fs_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=1
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h3']//lrm_resource[@id='fs_unifi']: OK (rc=0,
> origin=h3/crmd/789, version=0.1449.0)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1449.0 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.1 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='fs_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=1
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h3']//lrm_resource[@id='fs_unifi']: OK (rc=0,
> origin=h3/crmd/790, version=0.1449.1)
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section
> crm_config: OK (rc=0, origin=h3/crmd/792, version=0.1449.1)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1449.1 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.2 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='lxc_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=2
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h3']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
> origin=h3/crmd/796, version=0.1449.1)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1449.1 2
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.2 (null)
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:--
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='lxc_unifi']
> Nov 08 09:35:01 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=2
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_delete operation for section
> //node_state[@uname='h3']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
> origin=h3/crmd/797, version=0.1449.2)
> Nov 08 09:35:01 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section
> crm_config: OK (rc=0, origin=h3/crmd/799, version=0.1449.2)
> Nov 08 09:35:05 [3564] h4       lrmd:     info:
> process_lrmd_get_rsc_info:Resource 'rbd_unifi' not found (34 active
> resources)
> Nov 08 09:35:05 [3564] h4       lrmd:     info:
> process_lrmd_rsc_register:Added 'rbd_unifi' to the rsc list (35 active
> resources)
> Nov 08 09:35:05 [3567] h4       crmd:     info: do_lrm_rsc_op:Performing
> key=12:121:7:2bc5b523-b947-4035-b296-4c099646298b op=rbd_unifi_monitor_0
> Nov 08 09:35:05 [3564] h4       lrmd:     info:
> process_lrmd_get_rsc_info:Resource 'fs_unifi' not found (35 active
> resources)
> Nov 08 09:35:05 [3564] h4       lrmd:     info:
> process_lrmd_rsc_register:Added 'fs_unifi' to the rsc list (36 active
> resources)
> Nov 08 09:35:05 [3567] h4       crmd:     info: do_lrm_rsc_op:Performing
> key=13:121:7:2bc5b523-b947-4035-b296-4c099646298b op=fs_unifi_monitor_0
> Nov 08 09:35:05 [3564] h4       lrmd:     info:
> process_lrmd_get_rsc_info:Resource 'lxc_unifi' not found (36 active
> resources)
> Nov 08 09:35:05 [3564] h4       lrmd:     info:
> process_lrmd_rsc_register:Added 'lxc_unifi' to the rsc list (37 active
> resources)
> Nov 08 09:35:05 [3567] h4       crmd:     info: do_lrm_rsc_op:Performing
> key=14:121:7:2bc5b523-b947-4035-b296-4c099646298b op=lxc_unifi_monitor_0
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1449.2 2
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.3 (null)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=3
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources:
>  <lrm_resource id="rbd_unifi" type="rbd" class="ocf" provider="ceph"/>
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                        
>  <lrm_rsc_op id="rbd_unifi_last_0" operation_key="rbd_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="3:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;3:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h1" call-id="441" rc-code="7" op-status="0" interval="0"
> last-run="147862
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                      
>  </lrm_resource>
> Nov 08 09:35:05 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section status:
> OK (rc=0, origin=h1/crmd/357, version=0.1449.3)
> Nov 08 09:35:05 [3564] h4       lrmd:   notice:
> operation_finished:lxc_unifi_monitor_0:23285:stderr [ unifi doesn't exist ]
> Nov 08 09:35:05 [3567] h4       crmd:   notice:
> process_lrm_event:Operation lxc_unifi_monitor_0: not running (node=h4,
> call=499, rc=7, cib-update=405, confirmed=true)
> Nov 08 09:35:05 [3567] h4       crmd:   notice:
> process_lrm_event:h4-lxc_unifi_monitor_0:499 [ unifi doesn't exist\n ]
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1449.3 2
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.4 (null)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=4
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources:
>  <lrm_resource id="lxc_unifi" type="lxc" class="ocf" provider="heartbeat"/>
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                        
>  <lrm_rsc_op id="lxc_unifi_last_0" operation_key="lxc_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="8:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;8:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h2" call-id="451" rc-code="7" op-status="0" interval="0"
> last-run="147862
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                      
>  </lrm_resource>
> Nov 08 09:35:05 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section status:
> OK (rc=0, origin=h2/crmd/358, version=0.1449.4)
> Nov 08 09:35:05 [3562] h4        cib:     info:
> cib_process_request:Forwarding cib_modify operation for section status
> to master (origin=local/crmd/405)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1449.4 2
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.5 (null)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=5
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources:
>  <lrm_resource id="lxc_unifi" type="lxc" class="ocf" provider="heartbeat"/>
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                        
>  <lrm_rsc_op id="lxc_unifi_last_0" operation_key="lxc_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="14:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;14:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h4" call-id="499" rc-code="7" op-status="0" interval="0"
> last-run="1478
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                      
>  </lrm_resource>
> Nov 08 09:35:05 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section status:
> OK (rc=0, origin=h4/crmd/405, version=0.1449.5)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1449.5 2
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.6 (null)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=6
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources:
>  <lrm_resource id="fs_unifi" type="Filesystem" class="ocf"
> provider="heartbeat"/>
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                        
>  <lrm_rsc_op id="fs_unifi_last_0" operation_key="fs_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="7:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;7:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h2" call-id="447" rc-code="7" op-status="0" interval="0"
> last-run="14786262
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                      
>  </lrm_resource>
> Nov 08 09:35:05 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section status:
> OK (rc=0, origin=h2/crmd/359, version=0.1449.6)
> Nov 08 09:35:05 [3567] h4       crmd:   notice:
> process_lrm_event:Operation fs_unifi_monitor_0: not running (node=h4,
> call=495, rc=7, cib-update=406, confirmed=true)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1449.6 2
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.7 (null)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=7
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources:
>  <lrm_resource id="lxc_unifi" type="lxc" class="ocf" provider="heartbeat"/>
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                        
>  <lrm_rsc_op id="lxc_unifi_last_0" operation_key="lxc_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="5:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;5:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h1" call-id="449" rc-code="7" op-status="0" interval="0"
> last-run="147862
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                      
>  </lrm_resource>
> Nov 08 09:35:05 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section status:
> OK (rc=0, origin=h1/crmd/358, version=0.1449.7)
> Nov 08 09:35:05 [3562] h4        cib:     info:
> cib_process_request:Forwarding cib_modify operation for section status
> to master (origin=local/crmd/406)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1449.7 2
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.8 (null)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=8
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources:
>  <lrm_resource id="fs_unifi" type="Filesystem" class="ocf"
> provider="heartbeat"/>
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                        
>  <lrm_rsc_op id="fs_unifi_last_0" operation_key="fs_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="13:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;13:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h4" call-id="495" rc-code="7" op-status="0" interval="0"
> last-run="147862
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                      
>  </lrm_resource>
> Nov 08 09:35:05 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section status:
> OK (rc=0, origin=h4/crmd/406, version=0.1449.8)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1449.8 2
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.9 (null)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=9
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources:
>  <lrm_resource id="lxc_unifi" type="lxc" class="ocf" provider="heartbeat"/>
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                        
>  <lrm_rsc_op id="lxc_unifi_last_failure_0"
> operation_key="lxc_unifi_monitor_0" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
> transition-key="11:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:0;11:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h3" call-id="465" rc-code="0" op-status="0" interval="0" last-r
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                        
>  <lrm_rsc_op id="lxc_unifi_last_0" operation_key="lxc_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="11:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:0;11:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h3" call-id="465" rc-code="0" op-status="0" interval="0"
> last-run="1478
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                      
>  </lrm_resource>
> Nov 08 09:35:05 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section status:
> OK (rc=0, origin=h3/crmd/802, version=0.1449.9)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1449.9 2
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.10 (null)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=10
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources:
>  <lrm_resource id="fs_unifi" type="Filesystem" class="ocf"
> provider="heartbeat"/>
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                        
>  <lrm_rsc_op id="fs_unifi_last_0" operation_key="fs_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="4:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;4:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h1" call-id="445" rc-code="7" op-status="0" interval="0"
> last-run="14786262
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                      
>  </lrm_resource>
> Nov 08 09:35:05 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section status:
> OK (rc=0, origin=h1/crmd/359, version=0.1449.10)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1449.10 2
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.11 (null)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=11
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources:
>  <lrm_resource id="fs_unifi" type="Filesystem" class="ocf"
> provider="heartbeat"/>
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                        
>  <lrm_rsc_op id="fs_unifi_last_failure_0"
> operation_key="fs_unifi_monitor_0" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
> transition-key="10:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:0;10:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h3" call-id="461" rc-code="0" op-status="0" interval="0" last-run
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                        
>  <lrm_rsc_op id="fs_unifi_last_0" operation_key="fs_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="10:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:0;10:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h3" call-id="461" rc-code="0" op-status="0" interval="0"
> last-run="147862
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                      
>  </lrm_resource>
> Nov 08 09:35:05 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section status:
> OK (rc=0, origin=h3/crmd/803, version=0.1449.11)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1449.11 2
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.12 (null)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=12
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources:
>  <lrm_resource id="rbd_unifi" type="rbd" class="ocf" provider="ceph"/>
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                        
>  <lrm_rsc_op id="rbd_unifi_last_0" operation_key="rbd_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="6:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;6:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h2" call-id="443" rc-code="7" op-status="0" interval="0"
> last-run="147862
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                      
>  </lrm_resource>
> Nov 08 09:35:05 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section status:
> OK (rc=0, origin=h2/crmd/360, version=0.1449.12)
> Nov 08 09:35:05 [3567] h4       crmd:   notice:
> process_lrm_event:Operation rbd_unifi_monitor_0: not running (node=h4,
> call=491, rc=7, cib-update=407, confirmed=true)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1449.12 2
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.13 (null)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=13
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources:
>  <lrm_resource id="rbd_unifi" type="rbd" class="ocf" provider="ceph"/>
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                        
>  <lrm_rsc_op id="rbd_unifi_last_failure_0"
> operation_key="rbd_unifi_monitor_0" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
> transition-key="9:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:0;9:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h3" call-id="457" rc-code="0" op-status="0" interval="0" last-run
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                        
>  <lrm_rsc_op id="rbd_unifi_last_0" operation_key="rbd_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="9:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:0;9:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h3" call-id="457" rc-code="0" op-status="0" interval="0"
> last-run="147862
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                      
>  </lrm_resource>
> Nov 08 09:35:05 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section status:
> OK (rc=0, origin=h3/crmd/804, version=0.1449.13)
> Nov 08 09:35:05 [3562] h4        cib:     info:
> cib_process_request:Forwarding cib_modify operation for section status
> to master (origin=local/crmd/407)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: ---
> 0.1449.13 2
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:Diff: +++
> 0.1449.14 (null)
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:+  /cib:
>  @num_updates=14
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources:
>  <lrm_resource id="rbd_unifi" type="rbd" class="ocf" provider="ceph"/>
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                        
>  <lrm_rsc_op id="rbd_unifi_last_0" operation_key="rbd_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="12:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;12:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h4" call-id="491" rc-code="7" op-status="0" interval="0"
> last-run="1478
> Nov 08 09:35:05 [3562] h4        cib:     info: cib_perform_op:++      
>                                                                      
>  </lrm_resource>
> Nov 08 09:35:05 [3562] h4        cib:     info:
> cib_process_request:Completed cib_modify operation for section status:
> OK (rc=0, origin=h4/crmd/407, version=0.1449.14)
> Nov 08 09:35:10 [3562] h4        cib:     info:
> cib_process_ping:Reporting our current digest to h3:
> b5a63d2ac8b4835f8c8a2f310eb6648c for 0.1449.14 (0x12ddc90 0)




More information about the Users mailing list