<div dir="ltr">Hi Ulrich,<div><br></div><div>You're right, it is as if stonithd selected the incorrect device to reboot the node. I'm using fence_ilo as the stonith agent, and reviewing the params it take is not clear which one (besides the name which is irrelevant for stonithd) should be used to fence each node. </div><div><br></div><div>In cman+rgmanager you can associate a fence device params to each node, for example:</div><div><br></div><div><div><clusternode name="e1b07" nodeid="2"></div><div> <fence></div><div> <method name="single"></div><div> <device name="fence_ilo" ipaddr="e1b07-ilo"/></div><div> </method></div><div> </fence></div><div> </clusternode></div></div><div><br></div><div>what's the equivalent of that in corosync+pacemaker (using crm)??</div><div><br></div><div>In general, in a cluster of more than 2 nodes and more than 2 stonith devices, how stonithd find which stonith device should be used to fence a specific node??</div><div><br></div><div>Regards,</div><div> Ali</div><div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 4, 2017 at 2:27 AM, Ulrich Windl <span dir="ltr"><<a href="mailto:Ulrich.Windl@rz.uni-regensburg.de" target="_blank">Ulrich.Windl@rz.uni-regensburg.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi!<br>
<br>
A few messages that look uncommon to me are:<br>
<br>
crm_reap_dead_member: Removing node with name unknown and id 1239211543 from membership cache<br>
<br>
A bit later the node name is known:<br>
info: crm_update_peer_proc: pcmk_cpg_membership: Node e1b13[1239211543] - corosync-cpg is now offline<br>
<br>
Another node seems to go offline also:<br>
crmd: info: peer_update_callback: Client e1b13/peer now has status [offline] (DC=e1b07, changed=4000000)<br>
<br>
This looks OK to me:<br>
stonith-ng: debug: get_capable_devices: Searching through 3 devices to see what is capable of action (reboot) for target e1b13<br>
stonith-ng: debug: stonith_action_create: Initiating action status for agent fence_ilo (target=e1b13)<br>
<br>
This looks odd to me:<br>
stonith-ng: debug: stonith_device_execute: Operation status for node e1b13 on fence-e1b03 now running with pid=25689, timeout=20s<br>
stonith-ng: debug: stonith_device_execute: Operation status for node e1b13 on fence-e1b07 now running with pid=25690, timeout=20s<br>
stonith-ng: debug: stonith_device_execute: Operation status for node e1b13 on fence-e1b13 now running with pid=25691, timeout=20s<br>
<br>
Maybe not, because you can fence the node oin three different ways it seems:<br>
stonith-ng: debug: stonith_query_capable_device_<wbr>cb: Found 3 matching devices for 'e1b13'<br>
<br>
Now it's geting odd:<br>
stonith-ng: debug: schedule_stonith_command: Scheduling reboot on fence-e1b07 for remote peer e1b07 with op id (ae1956b5-ffe1-4d6a-b5a2-<wbr>c7bba2c6d7fd) (timeout=60s)<br>
stonith-ng: debug: stonith_action_create: Initiating action reboot for agent fence_ilo (target=e1b13)<br>
stonith-ng: debug: stonith_device_execute: Operation reboot for node e1b13 on fence-e1b07 now running with pid=25784, timeout=60s<br>
crmd: info: crm_update_peer_expected: handle_request: Node e1b07[1239211582] - expected state is now down (was member)<br>
stonith-ng: debug: st_child_done: Operation 'reboot' on 'fence-e1b07' completed with rc=0 (2 remaining)<br>
stonith-ng: notice: log_operation: Operation 'reboot' [25784] (call 6 from crmd.1201) for host 'e1b13' with device 'fence-e1b07' returned: 0 (OK)<br>
attrd: info: crm_update_peer_proc: pcmk_cpg_membership: Node e1b07[1239211582] - corosync-cpg is now offline<br>
<br>
To me it looks as if your STONITH agents kill the wrong node for reasons unknown to me.<br>
<br>
(didn't inspect the whole logs)<br>
<br>
Regards,<br>
Ulrich<br>
<br>
<br>
>>> Alfonso Ali <<a href="mailto:alfonso.ali@gmail.com">alfonso.ali@gmail.com</a>> schrieb am 03.01.2017 um 18:54 in Nachricht<br>
<CANeoTMee-=_-Gtf_<wbr>vxigKsrXNQ0pWEUAg=<a href="mailto:7YJHrhvrWDNthsmg@mail.gmail.com">7YJHrhvrWDNt<wbr>hsmg@mail.gmail.com</a>>:<br>
<div class="gmail-HOEnZb"><div class="gmail-h5">> Hi Ulrich,<br>
><br>
> I'm using udpu and static node list. This is my corosync conf:<br>
><br>
> --------------------Corosync<br>
> configuration-----------------<wbr>------------------------------<wbr>--<br>
> totem {<br>
> version: 2<br>
> cluster_name: test-cluster<br>
> token: 3000<br>
> token_retransmits_before_loss_<wbr>const: 10<br>
> clear_node_high_bit: yes<br>
> crypto_cipher: aes256<br>
> crypto_hash: sha1<br>
> transport: udpu<br>
><br>
> interface {<br>
> ringnumber: 0<br>
> bindnetaddr: 201.220.222.0<br>
> mcastport: 5405<br>
> ttl: 1<br>
> }<br>
> }<br>
><br>
> logging {<br>
> fileline: off<br>
> to_stderr: no<br>
> to_logfile: no<br>
> to_syslog: yes<br>
> syslog_facility: daemon<br>
> debug: on<br>
> timestamp: on<br>
> logger_subsys {<br>
> subsys: QUORUM<br>
> debug: on<br>
> }<br>
> }<br>
><br>
> quorum {<br>
> provider: corosync_votequorum<br>
> expected_votes: 3<br>
> }<br>
><br>
> nodelist {<br>
> node: {<br>
> ring0_addr: 201.220.222.62<br>
> }<br>
> node: {<br>
> ring0_addr: 201.220.222.23<br>
> }<br>
> node: {<br>
> ring0_addr: 201.220.222.61<br>
> }<br>
> node: {<br>
> ring0_addr: 201.220.222.22<br>
> }<br>
> }<br>
> ------------------------------<wbr>--/Corosync<br>
> conf--------------------------<wbr>-----------------------------<br>
><br>
> The pacemaker log is very long, i'm sending it attached as a zip file,<br>
> don't know if the list will allow it, if not please tell me which sections<br>
> (stonith, crmd, lrmd, attrd, cib) should i post.<br>
><br>
> For a better understanding, the cluster have 4 nodes, e1b03, e1b07, e1b12<br>
> and e1b13. I simulated a crash on e1b13 with:<br>
><br>
> echo c > /proc/sysrq-trigger<br>
><br>
> the cluster detected e1b13 as crashed and reboot it, but after that e1b07<br>
> was restarted too, and later e1b03 did the same, the only node that remined<br>
> alive was e1b12. The attached log was taken from that node.<br>
><br>
> Let me know if any other info is needed to debug the problem.<br>
><br>
> Regards,<br>
> Ali<br>
><br>
><br>
><br>
> On Mon, Jan 2, 2017 at 3:30 AM, Ulrich Windl <<br>
> <a href="mailto:Ulrich.Windl@rz.uni-regensburg.de">Ulrich.Windl@rz.uni-<wbr>regensburg.de</a>> wrote:<br>
><br>
>> Hi!<br>
>><br>
>> Seeing the detailed log of events would be helpful. Despite of that we had<br>
>> a similar issue with using multicast (and after adding a new node to an<br>
>> existing cluster). Switching to UDPU helped in our case, but unless we see<br>
>> the details, it's all just guessing...<br>
>><br>
>> Ulrich<br>
>> P.S. A good new year to everyone!<br>
>><br>
>> >>> Alfonso Ali <<a href="mailto:alfonso.ali@gmail.com">alfonso.ali@gmail.com</a>> schrieb am 30.12.2016 um 21:40 in<br>
>> Nachricht<br>
>> <CANeoTMcuNGw_T9e4WNEEK-nmHnV-<wbr>NwiX2Ck0UBDnVeuoiC=<a href="mailto:r8A@mail.gmail.com">r8A@mail.<wbr>gmail.com</a>>:<br>
>> > Hi,<br>
>> ><br>
>> > I have a four node cluster that uses iLo as fencing agent. When i<br>
>> simulate<br>
>> > a node crash (either killing corosync or echo c > /proc/sysrq-trigger)<br>
>> the<br>
>> > node is marked as UNCLEAN and requested to be restarted by the stonith<br>
>> > agent, but everytime that happens another node in the cluster is also<br>
>> > marked as UNCLEAN and rebooted as well. After the nodes are rebooted they<br>
>> > are marked as online again and cluster resume operation without problem.<br>
>> ><br>
>> > I have reviewed corosync and pacemaker logs but found nothing that<br>
>> explain<br>
>> > why the other node is also rebooted.<br>
>> ><br>
>> > Any hint of what to check or what to look for would be appreciated.<br>
>> ><br>
>> > -----------------Cluster conf--------------------------<wbr>--------<br>
>> > node 1239211542: e1b12 \<br>
>> > attributes standby=off<br>
>> > node 1239211543: e1b13<br>
>> > node 1239211581: e1b03 \<br>
>> > attributes standby=off<br>
>> > node 1239211582: e1b07 \<br>
>> > attributes standby=off<br>
>> > primitive fence-e1b03 stonith:fence_ilo \<br>
>> > params ipaddr=e1b03-ilo login=fence_agent passwd=XXX ssl_insecure=1 \<br>
>> > op monitor interval=300 timeout=120 \<br>
>> > meta migration-threshold=2 target-role=Started<br>
>> > primitive fence-e1b07 stonith:fence_ilo \<br>
>> > params ipaddr=e1b07-ilo login=fence_agent passwd=XXX ssl_insecure=1 \<br>
>> > op monitor interval=300 timeout=120 \<br>
>> > meta migration-threshold=2 target-role=Started<br>
>> > primitive fence-e1b12 stonith:fence_ilo \<br>
>> > params ipaddr=e1b12-ilo login=fence_agent passwd=XXX ssl_insecure=1 \<br>
>> > op monitor interval=300 timeout=120 \<br>
>> > meta migration-threshold=2 target-role=Started<br>
>> > primitive fence-e1b13 stonith:fence_ilo \<br>
>> > params ipaddr=e1b13-ilo login=fence_agent passwd=XXX ssl_insecure=1 \<br>
>> > op monitor interval=300 timeout=120 \<br>
>> > meta migration-threshold=2 target-role=Started<br>
>> > ..... extra resources ......<br>
>> > location l-f-e1b03 fence-e1b03 \<br>
>> > rule -inf: #uname eq e1b03 \<br>
>> > rule 10000: #uname eq e1b07<br>
>> > location l-f-e1b07 fence-e1b07 \<br>
>> > rule -inf: #uname eq e1b07 \<br>
>> > rule 10000: #uname eq e1b03<br>
>> > location l-f-e1b12 fence-e1b12 \<br>
>> > rule -inf: #uname eq e1b12 \<br>
>> > rule 10000: #uname eq e1b13<br>
>> > location l-f-e1b13 fence-e1b13 \<br>
>> > rule -inf: #uname eq e1b13 \<br>
>> > rule 10000: #uname eq e1b12<br>
>> > property cib-bootstrap-options: \<br>
>> > have-watchdog=false \<br>
>> > dc-version=1.1.15-e174ec8 \<br>
>> > cluster-infrastructure=<wbr>corosync \<br>
>> > stonith-enabled=true \<br>
>> > cluster-name=test-cluster \<br>
>> > no-quorum-policy=freeze \<br>
>> > last-lrm-refresh=1483125286<br>
>> > ------------------------------<wbr>------------------------------<br>
>> ----------------<br>
>> > ------------<br>
>> ><br>
>> > Regards,<br>
>> > Ali<br>
>><br>
<br>
<br>
<br>
</div></div></blockquote></div><br></div></div></div>