<tt><font size=2>"Users" <users-bounces@clusterlabs.org>
schrieb am 18.07.2018 15:46:09:<br>
<br>
> Von: Jan Pokornı <jpokorny@redhat.com></font></tt>
<br><tt><font size=2>> An: users@clusterlabs.org</font></tt>
<br><tt><font size=2>> Datum: 18.07.2018 15:46</font></tt>
<br><tt><font size=2>> Betreff: Re: [ClusterLabs] corosync/dlm fencing?</font></tt>
<br><tt><font size=2>> Gesendet von: "Users" <users-bounces@clusterlabs.org></font></tt>
<br><tt><font size=2>> <br>
> Just minor clarifications (without changing the validity) below:<br>
> <br>
> On 17/07/18 21:28 +0200, Jan Pokornı wrote:<br>
> > <br>
> > On 16/07/18 11:44 +0200, Philipp Achmüller wrote:<br>
> >> Unfortunatly it is not obvious for me - the "grep fence"
is attached<br>
> >> in my original message.<br>
> > <br>
> > Sifting your logs a bit:<br>
> > <br>
> >> -------------------<br>
> >> Node: siteb-2 (DC):<br>
> >> 2018-06-28T09:02:23.282153+02:00 siteb-2 pengine[189259]:
<br>
> notice: Move stonith-sbd#011(Started sitea-1 -> siteb-1)<br>
> >> [...]<br>
> >> 2018-06-28T09:02:23.284575+02:00 siteb-2 crmd[189260]:
notice: <br>
> Initiating stop operation stonith-sbd_stop_0 on sitea-1<br>
> >> [...]<br>
> >> 2018-06-28T09:02:23.288254+02:00 siteb-2 crmd[189260]:
notice: <br>
> Initiating start operation stonith-sbd_start_0 on siteb-1<br>
> >> [...]<br>
> >> 2018-06-28T09:02:38.414440+02:00 siteb-2 corosync[189245]:
<br>
> [TOTEM ] A processor failed, forming new configuration.<br>
> >> 2018-06-28T09:02:52.080141+02:00 siteb-2 corosync[189245]:
<br>
> [TOTEM ] A new membership (192.168.121.55:2012) was formed. Members
left: 2<br>
> >> 2018-06-28T09:02:52.080537+02:00 siteb-2 corosync[189245]:
<br>
> [TOTEM ] Failed to receive the leave message. failed: 2<br>
> >> 2018-06-28T09:02:52.083415+02:00 siteb-2 attrd[189258]:
notice:<br>
> Node siteb-1 state is now lost<br>
> >> [...]<br>
> >> 2018-06-28T09:02:52.084054+02:00 siteb-2 crmd[189260]: warning:
<br>
> No reason to expect node 2 to be down<br>
> >> [...]<br>
> >> 2018-06-28T09:02:52.084409+02:00 siteb-2 corosync[189245]:
<br>
> [QUORUM] Members[3]: 1 3 4<br>
> >> 2018-06-28T09:02:52.084492+02:00 siteb-2 corosync[189245]:
<br>
> [MAIN ] Completed service synchronization, ready to provide
service.<br>
> >> [...]<br>
> >> 2018-06-28T09:02:52.085210+02:00 siteb-2 kernel: [80872.012486]
<br>
> dlm: closing connection to node 2<br>
> >> [...]<br>
> >> 2018-06-28T09:02:53.098683+02:00 siteb-2 pengine[189259]:
<br>
> warning: Scheduling Node siteb-1 for STONITH<br>
> > <br>
> >> -------------------<br>
> >> Node sitea-1:<br>
> >> 2018-06-28T09:02:38.413748+02:00 sitea-1 corosync[6661]:
[TOTEM<br>
> ] A processor failed, forming new configuration.<br>
> >> 2018-06-28T09:02:52.079905+02:00 sitea-1 corosync[6661]:
[TOTEM<br>
> ] A new membership (192.168.121.55:2012) was formed. Members left:
2<br>
> >> 2018-06-28T09:02:52.080306+02:00 sitea-1 corosync[6661]:
[TOTEM<br>
> ] Failed to receive the leave message. failed: 2<br>
> >> 2018-06-28T09:02:52.082619+02:00 sitea-1 cib[9021]:
notice: <br>
> Node siteb-1 state is now lost<br>
> >> [...]<br>
> >> 2018-06-28T09:02:52.083429+02:00 sitea-1 corosync[6661]:
<br>
> [QUORUM] Members[3]: 1 3 4<br>
> >> 2018-06-28T09:02:52.083521+02:00 sitea-1 corosync[6661]:
[MAIN <br>
> ] Completed service synchronization, ready to provide service.<br>
> >> 2018-06-28T09:02:52.083606+02:00 sitea-1 crmd[9031]:
notice: <br>
> Node siteb-1 state is now lost<br>
> >> 2018-06-28T09:02:52.084290+02:00 sitea-1 dlm_controld[73416]:
<br>
> 59514 fence request 2 pid 171087 nodedown time 1530169372 fence_all
<br>
> dlm_stonith<br>
> >> 2018-06-28T09:02:52.085446+02:00 sitea-1 kernel: [59508.568940]
<br>
> dlm: closing connection to node 2<br>
> >> 2018-06-28T09:02:52.109393+02:00 sitea-1 dlm_stonith: <br>
> stonith_api_time: Found 0 entries for 2/(null): 0 in progress, 0 completed<br>
> >> 2018-06-28T09:02:52.110167+02:00 sitea-1 stonith-ng[9022]:
<br>
> notice: Client stonith-api.171087.d3c59fc2 wants to fence (reboot)
<br>
> '2' with device '(any)'<br>
> >> 2018-06-28T09:02:52.113257+02:00 sitea-1 stonith-ng[9022]:
<br>
> notice: Requesting peer fencing (reboot) of siteb-1<br>
> >> 2018-06-28T09:03:29.096714+02:00 sitea-1 stonith-ng[9022]:
<br>
> notice: Operation reboot of siteb-1 by sitea-2 for stonith-api.<br>
> 171087@sitea-1.9fe08723: OK<br>
> >> 2018-06-28T09:03:29.097152+02:00 sitea-1 stonith-api[171087]:
<br>
> stonith_api_kick: Node 2/(null) kicked: reboot<br>
> >> 2018-06-28T09:03:29.097426+02:00 sitea-1 crmd[9031]:
notice: <br>
> Peer lnx0361b was terminated (reboot) by sitea-2 on behalf of <br>
> stonith-api.171087: OK<br>
> >> 2018-06-28T09:03:30.098657+02:00 sitea-1 dlm_controld[73416]:
<br>
> 59552 fence result 2 pid 171087 result 0 exit status<br>
> >> 2018-06-28T09:03:30.099730+02:00 sitea-1 dlm_controld[73416]:
<br>
> 59552 fence status 2 receive 0 from 1 walltime 1530169410 local 59552<br>
> > <br>
> >> -------------------<br>
> >> Node sitea-2:<br>
> >> 2018-06-28T09:02:38.412808+02:00 sitea-2 corosync[6570]:
[TOTEM<br>
> ] A processor failed, forming new configuration.<br>
> >> 2018-06-28T09:02:52.078249+02:00 sitea-2 corosync[6570]:
[TOTEM<br>
> ] A new membership (192.168.121.55:2012) was formed. Members left:
2<br>
> >> 2018-06-28T09:02:52.078359+02:00 sitea-2 corosync[6570]:
[TOTEM<br>
> ] Failed to receive the leave message. failed: 2<br>
> >> 2018-06-28T09:02:52.081949+02:00 sitea-2 cib[9655]:
notice: <br>
> Node siteb-1 state is now lost<br>
> >> [...]<br>
> >> 2018-06-28T09:02:52.082653+02:00 sitea-2 corosync[6570]:
<br>
> [QUORUM] Members[3]: 1 3 4<br>
> >> 2018-06-28T09:02:52.082739+02:00 sitea-2 corosync[6570]:
[MAIN <br>
> ] Completed service synchronization, ready to provide service.<br>
> >> [...]<br>
> >> 2018-06-28T09:02:52.495697+02:00 sitea-2 stonith-ng[9656]:
<br>
> notice: stonith-sbd can fence (reboot) siteb-1: dynamic-list<br>
> >> 2018-06-28T09:02:52.495902+02:00 sitea-2 stonith-ng[9656]:
<br>
> notice: Delaying reboot on stonith-sbd for 25358ms (timeout=300s)<br>
> >> 2018-06-28T09:03:29.093957+02:00 sitea-2 stonith-ng[9656]:
<br>
> notice: Operation 'reboot' [231293] (call 2 from stonith-api.171087)<br>
> for host 'siteb-1' with device 'stonith-sbd' returned: 0 (OK)<br>
> >> 2018-06-28T09:03:29.096254+02:00 sitea-2 stonith-ng[9656]:
<br>
> notice: Operation reboot of siteb-1 by sitea-2 for stonith-api.<br>
> 171087@sitea-1.9fe08723: OK<br>
> >> 2018-06-28T09:03:29.096769+02:00 sitea-2 crmd[9660]:
notice: <br>
> Peer siteb-1 was terminated (reboot) by sitea-2 on behalf of <br>
> stonith-api.171087: OK<br>
> > <br>
> >> -------------------<br>
> >> Node sideb-1 has no entries during this timeframe<br>
> >> <br>
> >> during standby corosync should be up/running - so may the
"Failed to <br>
> >> receive the leave message" will be a Problem?<br>
> > <br>
> > True, it should normally stay running, and that's actually the
problem<br>
> > here and you just seem to be confusing consequence with consequence:<br>
> <br>
> "consequence with the cause", apparently<br>
> </font></tt>
<br><tt><font size=2>> > <br>
> > - first: the corosync membership got broken on siteb-1 node<br>
> > (hard to tell why, the only thing we can observe is that<br>
> > sbd monitoring was moved there from sitea-1)<br>
> > <br>
> > - then: dlm_controld on sitea-1, which already observed that
membership<br>
> > issue of siteb-1, proceed to ask pacemaker to fence siteb-1,
which<br>
> > was cheerfully executed by sitea-2 node<br>
> > <br>
> > I cannot really tell what the root cause was here, but observing
PID<br>
> > as high as 171087, I wonder if you have recent enough libqb (0.17.2):<br>
> <br>
> if it's unclear, 0.17.2 as the lowest version that's fixed</font></tt>
<br>
<br><tt><font size=2>following version is currently installed with SP3:</font></tt>
<br>
<br><tt><font size=2>libqb0-1.0.1-2.15.x86_64</font></tt>
<br>
<br><tt><font size=2>> <br>
> > </font></tt><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1114852"><tt><font size=2>https://bugzilla.redhat.com/show_bug.cgi?id=1114852</font></tt></a><tt><font size=2><br>
> > <br>
> > (and would perhaps fit: new sbd process/es with high enough PIDs,<br>
> <br>
> actually, crm_attribute processes arising from stonith:external/sbd<br>
> <br>
> (fun fact: fence_sbd agent from fence-agents project, using a bit<br>
> different API for fencing, doesn't need to resort to relying on<br>
> stonith-timeout cluster property, so it doesn't involved such<br>
> "calling home")<br>
> <br>
> > exercising libqb-backed IPC -> something bad happening like<br>
> > corosync getting stuck or crashing -> node kicked out...)<br>
> <br>
> -- <br>
> Nazdar,<br>
> Jan (Poki)<br>
> [Anhang "attlmf24.dat" gelöscht von Philipp Achmüller/ARZ/AT]
<br>
> _______________________________________________<br>
> Users mailing list: Users@clusterlabs.org<br>
> </font></tt><a href=https://lists.clusterlabs.org/mailman/listinfo/users><tt><font size=2>https://lists.clusterlabs.org/mailman/listinfo/users</font></tt></a><tt><font size=2><br>
> <br>
> Project Home: </font></tt><a href=http://www.clusterlabs.org/><tt><font size=2>http://www.clusterlabs.org</font></tt></a><tt><font size=2><br>
> Getting started: </font></tt><a href=http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf><tt><font size=2>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</font></tt></a><tt><font size=2><br>
> Bugs: </font></tt><a href=http://bugs.clusterlabs.org/><tt><font size=2>http://bugs.clusterlabs.org</font></tt></a><tt><font size=2><br>
</font></tt><font size=2 face="sans-serif"><br>
</font>