[ClusterLabs] Resources not always stopped when quorum lost
Vladislav Bogdanov
bubble at hoster-ok.com
Thu May 28 18:25:25 UTC 2015
28 мая 2015 г. 18:39:27 GMT+03:00, Matt Rideout <mrideout at windserve.com> пишет:
>I'm attempting to upgrade a two node cluster with no quorum requirement
>
>to a three node cluster with a two member quorum requirement. Each node
>
>is running CentOS 7, Pacemaker 1.1.12-22 and Crosync 2.3.4-4.
>
>If a node that's running resources loses quorum, then I want it to stop
>
>all of its resources. The goal was partially accomplished by setting
>the following in corosync.conf:
>
>quorum {
> provider: corosync_votequorum
> two_node: 1
>}
>
>...and updating Pacemaker's configuration with:
>
>pcs property set no-quorum-policy=stop
>
>With the above configuration, Two failure scenarios work as I would
>expect:
>
>1. If I power up a single node, it sees that there is no quorum, and
>refuses to start any resources until it sees a second node come up.
>
>2. If there are two nodes running, and I power down a node that's
>running resources, the other node sees that it lost quorum, and refuses
>
>to start any resources.
>
>However, a third failure scenario does not work as I would expect:
>
>3. If there are two nodes running, and I power down a node that's not
>running resources, the node that is running resources notes in its log
>that it lost quorum, but does not actually shutdown any of its running
>services.
>
>Any ideas on what the problem may be would be greatly appreciated. It
>in
>case it helps, I included the output of "pcs status", "pcs config
>show",
>the contents of "corosync.conf", and the pacemaker and corosync logs
>from the period during which resources were not stopped.
>
>*"pcs status" shows the resources still running after quorum is lost:*
>
>Cluster name:
>Last updated: Thu May 28 10:27:47 2015
>Last change: Thu May 28 10:03:05 2015
>Stack: corosync
>Current DC: node1 (1) - partition WITHOUT quorum
>Version: 1.1.12-a14efad
>3 Nodes configured
>12 Resources configured
>
>
>Node node3 (3): OFFLINE (standby)
>Online: [ node1 ]
>OFFLINE: [ node2 ]
>
>Full list of resources:
>
> Resource Group: primary
> virtual_ip_primary (ocf::heartbeat:IPaddr2): Started node1
> GreenArrowFS (ocf::heartbeat:Filesystem): Started node1
> GreenArrow (ocf::drh:greenarrow): Started node1
> virtual_ip_1 (ocf::heartbeat:IPaddr2): Started node1
> virtual_ip_2 (ocf::heartbeat:IPaddr2): Started node1
> Resource Group: secondary
> virtual_ip_secondary (ocf::heartbeat:IPaddr2): Stopped
> GreenArrow-Secondary (ocf::drh:greenarrow-secondary): Stopped
> Clone Set: ping-clone [ping]
> Started: [ node1 ]
> Stopped: [ node2 node3 ]
> Master/Slave Set: GreenArrowDataClone [GreenArrowData]
> Masters: [ node1 ]
> Stopped: [ node2 node3 ]
>
>PCSD Status:
> node1: Online
> node2: Offline
> node3: Offline
>
>Daemon Status:
> corosync: active/enabled
> pacemaker: active/enabled
> pcsd: active/enabled
>
>*"pcs config show"**shows that the "no-quorum-policy: stop" setting is
>in place:*
>
>Cluster Name:
>Corosync Nodes:
> node1 node2 node3
>Pacemaker Nodes:
> node1 node2 node3
>
>Resources:
> Group: primary
>Resource: virtual_ip_primary (class=ocf provider=heartbeat
>type=IPaddr2)
> Attributes: ip=10.10.10.1 cidr_netmask=32
> Operations: start interval=0s timeout=20s
>(virtual_ip_primary-start-timeout-20s)
> stop interval=0s timeout=20s
>(virtual_ip_primary-stop-timeout-20s)
> monitor interval=30s
>(virtual_ip_primary-monitor-interval-30s)
> Resource: GreenArrowFS (class=ocf provider=heartbeat type=Filesystem)
> Attributes: device=/dev/drbd1 directory=/media/drbd1 fstype=xfs
>options=noatime,discard
>Operations: start interval=0s timeout=60
>(GreenArrowFS-start-timeout-60)
> stop interval=0s timeout=60 (GreenArrowFS-stop-timeout-60)
> monitor interval=20 timeout=40
>(GreenArrowFS-monitor-interval-20)
> Resource: GreenArrow (class=ocf provider=drh type=greenarrow)
> Operations: start interval=0s timeout=30 (GreenArrow-start-timeout-30)
> stop interval=0s timeout=240 (GreenArrow-stop-timeout-240)
> monitor interval=10 timeout=20
>(GreenArrow-monitor-interval-10)
> Resource: virtual_ip_1 (class=ocf provider=heartbeat type=IPaddr2)
> Attributes: ip=64.21.76.51 cidr_netmask=32
> Operations: start interval=0s timeout=20s
>(virtual_ip_1-start-timeout-20s)
> stop interval=0s timeout=20s (virtual_ip_1-stop-timeout-20s)
> monitor interval=30s (virtual_ip_1-monitor-interval-30s)
> Resource: virtual_ip_2 (class=ocf provider=heartbeat type=IPaddr2)
> Attributes: ip=64.21.76.63 cidr_netmask=32
> Operations: start interval=0s timeout=20s
>(virtual_ip_2-start-timeout-20s)
> stop interval=0s timeout=20s (virtual_ip_2-stop-timeout-20s)
> monitor interval=30s (virtual_ip_2-monitor-interval-30s)
> Group: secondary
> Resource: virtual_ip_secondary (class=ocf provider=heartbeat
>type=IPaddr2)
> Attributes: ip=10.10.10.4 cidr_netmask=32
> Operations: start interval=0s timeout=20s
>(virtual_ip_secondary-start-timeout-20s)
> stop interval=0s timeout=20s
>(virtual_ip_secondary-stop-timeout-20s)
> monitor interval=30s
>(virtual_ip_secondary-monitor-interval-30s)
> Resource: GreenArrow-Secondary (class=ocf provider=drh
>type=greenarrow-secondary)
> Operations: start interval=0s timeout=30
>(GreenArrow-Secondary-start-timeout-30)
> stop interval=0s timeout=240
>(GreenArrow-Secondary-stop-timeout-240)
> monitor interval=10 timeout=20
>(GreenArrow-Secondary-monitor-interval-10)
> Clone: ping-clone
> Resource: ping (class=ocf provider=pacemaker type=ping)
> Attributes: dampen=30s multiplier=1000 host_list=64.21.76.1
> Operations: start interval=0s timeout=60 (ping-start-timeout-60)
> stop interval=0s timeout=20 (ping-stop-timeout-20)
> monitor interval=10 timeout=60 (ping-monitor-interval-10)
> Master: GreenArrowDataClone
> Meta Attrs: master-max=1 master-node-max=1 clone-max=2
>clone-node-max=1 notify=true
> Resource: GreenArrowData (class=ocf provider=linbit type=drbd)
> Attributes: drbd_resource=r0
> Operations: start interval=0s timeout=240
>(GreenArrowData-start-timeout-240)
> promote interval=0s timeout=90
>(GreenArrowData-promote-timeout-90)
> demote interval=0s timeout=90
>(GreenArrowData-demote-timeout-90)
> stop interval=0s timeout=100
>(GreenArrowData-stop-timeout-100)
> monitor interval=60s (GreenArrowData-monitor-interval-60s)
>
>Stonith Devices:
>Fencing Levels:
>
>Location Constraints:
> Resource: primary
>Enabled on: node1 (score:INFINITY) (id:location-primary-node1-INFINITY)
> Constraint: location-primary
> Rule: score=-INFINITY boolean-op=or (id:location-primary-rule)
> Expression: pingd lt 1 (id:location-primary-rule-expr)
> Expression: not_defined pingd (id:location-primary-rule-expr-1)
>Ordering Constraints:
> promote GreenArrowDataClone then start GreenArrowFS (kind:Mandatory)
>(id:order-GreenArrowDataClone-GreenArrowFS-mandatory)
> stop GreenArrowFS then demote GreenArrowDataClone (kind:Mandatory)
>(id:order-GreenArrowFS-GreenArrowDataClone-mandatory)
>Colocation Constraints:
> GreenArrowFS with GreenArrowDataClone (score:INFINITY)
>(with-rsc-role:Master)
>(id:colocation-GreenArrowFS-GreenArrowDataClone-INFINITY)
> virtual_ip_secondary with GreenArrowDataClone (score:INFINITY)
>(with-rsc-role:Slave)
>(id:colocation-virtual_ip_secondary-GreenArrowDataClone-INFINITY)
> virtual_ip_primary with GreenArrowDataClone (score:INFINITY)
>(with-rsc-role:Master)
>(id:colocation-virtual_ip_primary-GreenArrowDataClone-INFINITY)
>
>Cluster Properties:
> cluster-infrastructure: corosync
> cluster-name: cluster_greenarrow
> dc-version: 1.1.12-a14efad
> have-watchdog: false
> no-quorum-policy: stop
> stonith-enabled: false
>Node Attributes:
> node3: standby=on
>
>*Here's what was logged*:
>
>May 28 10:19:51 node1 pengine[1296]: notice: stage6: Scheduling Node
>node3 for shutdown
>May 28 10:19:51 node1 pengine[1296]: notice: process_pe_message:
>Calculated Transition 7: /var/lib/pacemaker/pengine/pe-input-992.bz2
>May 28 10:19:51 node1 crmd[1297]: notice: run_graph: Transition 7
>(Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0,
>Source=/var/lib/pacemaker/pengine/pe-input-992.bz2): Complete
>May 28 10:19:51 node1 crmd[1297]: notice: do_state_transition: State
>transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
>cause=C_FSA_INTERNAL origin=notify_crmd ]
>May 28 10:19:51 node1 crmd[1297]: notice: peer_update_callback:
>do_shutdown of node3 (op 64) is complete
>May 28 10:19:51 node1 attrd[1295]: notice: crm_update_peer_state:
>attrd_peer_change_cb: Node node3[3] - state is now lost (was member)
>May 28 10:19:51 node1 attrd[1295]: notice: attrd_peer_remove: Removing
>all node3 attributes for attrd_peer_change_cb
>May 28 10:19:51 node1 attrd[1295]: notice: attrd_peer_change_cb: Lost
>attribute writer node3
>May 28 10:19:51 node1 corosync[1040]: [TOTEM ] Membership left list
>contains incorrect address. This is sign of misconfiguration between
>nodes!
>May 28 10:19:51 node1 corosync[1040]: [TOTEM ] A new membership
>(64.21.76.61:25740) was formed. Members left: 3
>May 28 10:19:51 node1 corosync[1040]: [QUORUM] This node is within the
>non-primary component and will NOT provide any services.
>May 28 10:19:51 node1 corosync[1040]: [QUORUM] Members[1]: 1
>May 28 10:19:51 node1 corosync[1040]: [MAIN ] Completed service
>synchronization, ready to provide service.
>May 28 10:19:51 node1 crmd[1297]: notice: pcmk_quorum_notification:
>Membership 25740: quorum lost (1)
>May 28 10:19:51 node1 crmd[1297]: notice: crm_update_peer_state:
>pcmk_quorum_notification: Node node3[3] - state is now lost (was
>member)
>May 28 10:19:51 node1 crmd[1297]: notice: peer_update_callback:
>do_shutdown of node3 (op 64) is complete
>May 28 10:19:51 node1 pacemakerd[1254]: notice:
>pcmk_quorum_notification: Membership 25740: quorum lost (1)
>May 28 10:19:51 node1 pacemakerd[1254]: notice: crm_update_peer_state:
>pcmk_quorum_notification: Node node3[3] - state is now lost (was
>member)
>May 28 10:19:52 node1 corosync[1040]: [TOTEM ] Automatically recovered
>ring 1
>
>*H**ere's corosync.conf:*
>
>totem {
> version: 2
> secauth: off
> cluster_name: cluster_greenarrow
> rrp_mode: passive
> transport: udpu
>}
>
>nodelist {
> node {
> ring0_addr: node1
> ring1_addr: 10.10.10.2
> nodeid: 1
> }
> node {
> ring0_addr: node2
> ring1_addr: 10.10.10.3
> nodeid: 2
> }
> node {
> ring0_addr: node3
> nodeid: 3
> }
>}
>
>quorum {
> provider: corosync_votequorum
> two_node: 0
>}
>
>logging {
> to_syslog: yes
>}
>
>Thanks,
>
>Matt
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Users mailing list: Users at clusterlabs.org
>http://clusterlabs.org/mailman/listinfo/users
>
>Project Home: http://www.clusterlabs.org
>Getting started:
>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>Bugs: http://bugs.clusterlabs.org
Hi,
you probably need to replace two_node with wait_for_all to achieve what you want. Two_node mode implies latter but weakens quorum requirement from 50%+1 to just 50%. So pacemaker sees that your cluster is quorate.
Best,
Vladislav
More information about the Users
mailing list