[ClusterLabs] Stonith two-node cluster shot each other
Klaus Wenninger
kwenning at redhat.com
Wed Dec 5 05:01:47 EST 2018
If you are not so sure which of the nodes you want to give
precedence then you can at least add some random-delay
to it as to at least prevent that they at the same time
decide to kill each other (fence-race).
If your fencing-agent doesn't support a delay or you don't
want to use that for some reason, pacemaker offers a
generic approach with the 2 meta attributes pcmk_delay_max
and pcmk_delay_base that will give you a random delay
between pcmk_delay_base and pcmk_delay_max (either of them
can be 0 if you like - not at the same time for obvious reasons ;-) ).
Regards,
Klaus
On 12/04/2018 07:32 PM, Digimer wrote:
> You need to set a fence delay on the node you want to win in a case like
> this. So say, for example, node 1 is hosting services. You will want to
> add 'delay="15"' to the stonith config for node 1.
>
> This way, when both nodes try to fence each other, node 2 looks up how
> to fence node 1, sees a delay and pauses for 15 seconds. Node 1 looks up
> how to fence node 2, sees no delay, and fences immediately. Node 1
> lives, node 2 gets fenced.
>
> digimer
>
> On 2018-12-04 12:48 p.m., Daniel Ragle wrote:
>> I *think* the two nodes of my cluster shot each other in the head this
>> weekend and I can't figure out why.
>>
>> Looking at corosync.log on node1 I see this:
>>
>> [143747] node1.mydomain.com corosyncnotice [TOTEM ] A processor failed,
>> forming new configuration.
>> [143747] node1.mydomain.com corosyncnotice [TOTEM ] A new membership
>> (192.168.10.25:236) was formed. Members joined: 2 left: 2
>> [143747] node1.mydomain.com corosyncnotice [TOTEM ] Failed to receive
>> the leave message. failed: 2
>> [143747] node1.mydomain.com corosyncnotice [TOTEM ] Retransmit List: 1
>> Dec 01 07:03:50 [143768] node1.mydomain.com crmd: info:
>> pcmk_cpg_membership: Node 2 left group crmd (peer=node2.mydomain.com,
>> counter=1.0)
>> Dec 01 07:03:50 [143766] node1.mydomain.com attrd: info:
>> pcmk_cpg_membership: Node 2 left group attrd (peer=node2.mydomain.com,
>> counter=1.0)
>> Dec 01 07:03:50 [143764] node1.mydomain.com stonith-ng: info:
>> pcmk_cpg_membership: Node 2 left group stonith-ng
>> (peer=node2.vselect.com, counter=1.0)
>> Dec 01 07:03:50 [143762] node1.mydomain.com pacemakerd: info:
>> pcmk_cpg_membership: Node 2 left group pacemakerd
>> (peer=node2.vselect.com, counter=1.0)
>>
>> Followed by a whole slew of messages generally saying node2 was
>> dead/could not be reached, culminating in:
>>
>> Dec 01 07:04:20 [143764] node1.mydomain.com stonith-ng: notice:
>> initiate_remote_stonith_op: Requesting peer fencing (reboot) of
>> node2.mydomain.com | id=a041d1df-e857-4815-91db-00f448106a33 state=0
>> Dec 01 07:04:20 [143764] node1.mydomain.com stonith-ng: info:
>> process_remote_stonith_query: Query result 1 of 2 from
>> node1.mydomain.com for node2.mydomain.com/reboot (1 devices)
>> a041d1df-e857-4815-91db-00f448106a33
>> Dec 01 07:04:20 [143764] node1.mydomain.com stonith-ng: info:
>> call_remote_stonith: Total timeout set to 300 for peer's fencing of
>> node2.mydomain.com for
>> stonith-api.139901|id=a041d1df-e857-4815-91db-00f448106a33
>> Dec 01 07:04:20 [143764] node1.mydomain.com stonith-ng: info:
>> call_remote_stonith: Requesting that 'node1.mydomain.com' perform op
>> 'node2.mydomain.com reboot' for stonith-api.139901 (360s, 0s)
>> Dec 01 07:04:20 [143764] node1.mydomain.com stonith-ng: info:
>> process_remote_stonith_query: Query result 2 of 2 from
>> node2.mydomain.com for node2.mydomain.com/reboot (1 devices)
>> a041d1df-e857-4815-91db-00f448106a33
>> Dec 01 07:04:20 [143764] node1.mydomain.com stonith-ng: info:
>> stonith_fence_get_devices_cb: Found 1 matching devices for
>> 'node2.mydomain.com'
>> Dec 01 07:04:21 [143768] node1.mydomain.com crmd: info:
>> crm_update_peer_expected: handle_request: Node node2.mydomain.com[2]
>> - expected state is now down (was member)
>> Dec 01 07:04:21 [143766] node1.mydomain.com attrd: info:
>> attrd_peer_update: Setting shutdown[node2.mydomain.com]: (null) ->
>> 1543665861 from node2.mydomain.com
>> Dec 01 07:04:21 [143763] node1.mydomain.com cib: info:
>> cib_perform_op: Diff: --- 0.188.66 2
>> Dec 01 07:04:21 [143763] node1.mydomain.com cib: info:
>> cib_perform_op: Diff: +++ 0.188.67 (null)
>> Dec 01 07:04:21 [143763] node1.mydomain.com cib: info:
>> cib_perform_op: + /cib: @num_updates=67
>> Dec 01 07:04:21 [143763] node1.mydomain.com cib: info:
>> cib_perform_op: ++ /cib/status/node_state[@id='2']:
>> <transient_attributes id="2"/>
>> Dec 01 07:04:21 [143763] node1.mydomain.com cib: info:
>> cib_perform_op: ++ <instance_attributes id="status-2">
>> Dec 01 07:04:21 [143763] node1.mydomain.com cib: info:
>> cib_perform_op: ++ <nvpair
>> id="status-2-shutdown" name="shutdown" value="1543665861"/>
>> Dec 01 07:04:21 [143763] node1.mydomain.com cib: info:
>> cib_perform_op: ++ </instance_attributes>
>> Dec 01 07:04:21 [143763] node1.mydomain.com cib: info:
>> cib_perform_op: ++ </transient_attributes>
>> Dec 01 07:04:21 [143763] node1.mydomain.com cib: info:
>> cib_process_request: Completed cib_modify operation for section status:
>> OK (rc=0, origin=node2.mydomain.com/attrd/6, version=0.188.67)
>>
>> And on node2 I see this:
>>
>> [50215] node2.mydomain.com corosyncnotice [TOTEM ] A new membership
>> (192.168.10.25:228) was formed. Members
>> [50215] node2.mydomain.com corosyncnotice [TOTEM ] A new membership
>> (192.168.10.25:236) was formed. Members joined: 1 left: 1
>> [50215] node2.mydomain.com corosyncnotice [TOTEM ] Failed to receive
>> the leave message. failed: 1
>> Dec 01 07:03:50 [50224] node2.mydomain.com cib: info:
>> pcmk_cpg_membership: Node 1 left group cib (peer=node1.mydomain.com,
>> counter=2.0)
>> Dec 01 07:03:50 [50224] node2.mydomain.com cib: info:
>> crm_update_peer_proc: pcmk_cpg_membership: Node node1.mydomain.com[1] -
>> corosync-cpg is now offline
>> Dec 01 07:03:50 [50229] node2.mydomain.com crmd: info:
>> pcmk_cpg_membership: Node 1 left group crmd (peer=node1.mydomain.com,
>> counter=2.0)
>> Dec 01 07:03:50 [50229] node2.mydomain.com crmd: info:
>> crm_update_peer_proc: pcmk_cpg_membership: Node node1.mydomain.com[1] -
>> corosync-cpg is now offline
>> Dec 01 07:03:50 [50229] node2.mydomain.com crmd: info:
>> peer_update_callback: Client node1.mydomain.com/peer now has status
>> [offline] (DC=true, changed=4000000)
>>
>> and then later
>>
>> Dec 01 07:04:20 [50225] node2.mydomain.com stonith-ng: notice:
>> handle_request: Client stonith-api.170881.b598a6f3 wants to fence
>> (reboot) '1' with device '(any)'
>> Dec 01 07:04:20 [50225] node2.mydomain.com stonith-ng: notice:
>> initiate_remote_stonith_op: Requesting peer fencing (reboot) of
>> node1.mydomain.com | id=2b08eff2-1555-46fa-8a88-fe500f3fca87 state=0
>> Dec 01 07:04:20 [50225] node2.mydomain.com stonith-ng: info:
>> process_remote_stonith_query: Query result 1 of 2 from
>> node1.mydomain.com for node1.mydomain.com/reboot (1 devices)
>> 2b08eff2-1555-46fa-8a88-fe500f3fca87
>> Dec 01 07:04:20 [50225] node2.mydomain.com stonith-ng: info:
>> process_remote_stonith_query: Query result 2 of 2 from
>> node2.mydomain.com for node1.mydomain.com/reboot (1 devices)
>> 2b08eff2-1555-46fa-8a88-fe500f3fca87
>> Dec 01 07:04:20 [50225] node2.mydomain.com stonith-ng: info:
>> call_remote_stonith: Total timeout set to 300 for peer's fencing of
>> node1.mydomain.com for
>> stonith-api.170881|id=2b08eff2-1555-46fa-8a88-fe500f3fca87
>> Dec 01 07:04:20 [50225] node2.mydomain.com stonith-ng: info:
>> call_remote_stonith: Requesting that 'node2.mydomain.com' perform op
>> 'node1.mydomain.com reboot' for stonith-api.170881 (360s, 0s)
>> Dec 01 07:04:21 [50225] node2.mydomain.com stonith-ng: info:
>> stonith_fence_get_devices_cb: Found 1 matching devices for
>> 'node1.mydomain.com'
>>
>> What is wrong with my config that they would want to kill each other?
>> Shouldn't one always survive?
>>
>> # pcs stonith show --full
>> Resource: FenceNode2 (class=stonith type=fence_ipmilan)
>> Attributes: hexadecimal_kg=<KEY> ipaddr=192.168.10.29 lanplus=1
>> login=ipmiUser method=onoff passwd=<BLAH> power_timeout=30 power_wait=4
>> Operations: monitor interval=60s (FenceNode2-monitor-interval-60s)
>> Resource: FenceNode1 (class=stonith type=fence_ipmilan)
>> Attributes: hexadecimal_kg=<KEY> ipaddr=192.168.100.28 lanplus=1
>> login=ipmiUser method=onoff passwd=<BLAH> power_timeout=30 power_wait=4
>> Operations: monitor interval=60s (FenceNode1-monitor-interval-60s)
>>
>> The corresponding constraints:
>>
>> Resource: FenceNode1
>> Disabled on: node1.mydomain.com (score:-INFINITY)
>> Resource: FenceNode2
>> Disabled on: node2.mydomain.com (score:-INFINITY)
>>
>> And corosync.conf:
>>
>> # cat /etc/corosync/corosync.conf
>> totem {
>> version: 2
>> cluster_name: MyCluster
>> secauth: off
>> transport: udp
>>
>> interface {
>> ringnumber: 0
>> bindnetaddr: 192.168.10.0
>> broadcast: no
>> mcastport: 5405
>> ttl: 1
>> }
>> }
>>
>> nodelist {
>> node {
>> ring0_addr: node1.vselect.com
>> nodeid: 1
>> }
>>
>> node {
>> ring0_addr: node2.vselect.com
>> nodeid: 2
>> }
>> }
>>
>> quorum {
>> provider: corosync_votequorum
>> two_node: 1
>> }
>>
>> logging {
>> to_logfile: yes
>> logfile: /var/log/cluster/corosync.log
>> to_syslog: yes
>> }
>>
>> TIA,
>>
>> Dan
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
More information about the Users
mailing list