<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#3333FF" bgcolor="#FFFFFF">
<tt>Hello again Ken et all.<br>
<br>
I realized about many things investigating this issue but I feel I need a bit more help from you guys.<br>
<br>
It's clear the monitoring process is reporting a timeout. Although I've increased this timeout to 30c using pcmk_monitoring_timeout,<br>
and during this last 2 hours the process did not fail, I'd like to understand more in detail how this process works and if I'm
<br>
getting a timeout after 20 secs, it looks to me something else could be happening in my systems.<br>
<br>
I tried enabling debug again and, as before, the 'debug' option creates the file but does not update anything unless I enable 'verbose'.<br>
Funny thing because when I enable it, I hit a bug and the fencing does not start:<br>
<br>
<a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/show_bug.cgi?id=1549366">https://bugzilla.redhat.com/show_bug.cgi?id=1549366</a><br>
<br>
I enabled debug at corosync layer and I got some more information that was nice to better understand this issue but still, not enough<br>
information to narrow down where the issue comes from.<br>
<br>
Said this, I'd like to know, if there is a way to review more in detail what the monitoring process is doing like ping, status, etc
<br>
and it that time is dedicated to the same action all those secs. <br>
<br>
Any idea will be more than welcome.<br>
<br>
As always, appreciate your help.<br>
<br>
Regards<br>
Javier<br>
<br>
<br>
</tt><br>
<div class="moz-cite-prefix">
<div style="mso-line-height-rule:exactly;-webkit-text-size-adjust:100%;">
<table cellpadding="0" cellspacing="0" border="0" style="width:100%;">
<tbody>
<tr style="font-size:0;">
<td align="left" style="vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;">
<tbody>
<tr style="font-size:0;">
<td align="left" style="padding:15px 0 30px;vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;">
<tbody>
<tr style="font-size:0;">
<td align="left" style="padding:0;vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;">
<tbody>
<tr style="font-size:0;">
<td align="left" style="vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;color:#3B3739;font-style:normal;font-weight:700;white-space:nowrap;">
<tbody>
<tr style="font-size:14.67px;">
<td align="left" style="vertical-align:top;font-family:Arial;">Francisco Javier<span style="font-family:remialcxesans;font-size:1px;color:#FFFFFF;line-height:1px;"></span></td>
<td align="left" style="vertical-align:top;font-family:Arial;font-weight:400;"> </td>
<td align="left" style="vertical-align:top;font-family:Arial;">Lopez</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr style="font-size:0;">
<td align="left" style="vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;color:#3B3739;font-style:normal;font-weight:400;white-space:nowrap;">
<tbody>
<tr style="font-size:13.33px;">
<td align="left" style="vertical-align:top;font-family:Arial;">IT System Engineer</td>
<td align="left" style="vertical-align:top;color:#EA7200;font-family:Arial;"> | </td>
<td align="left" style="vertical-align:top;font-family:Arial;">Global IT</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr style="font-size:0;">
<td align="left" style="vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;color:#EA7200;font-style:normal;font-weight:400;white-space:nowrap;">
<tbody>
<tr style="font-size:13.33px;">
<td align="left" style="vertical-align:top;color:#3B3739;font-family:Arial;">O: <a href="tel:+34%20619%20728%20249" target="_blank" id="LPlnk689713" style="text-decoration:none;color:#3B3739;"><strong style="font-weight:400;">+34 619 728 249</strong></a></td>
<td align="left" style="vertical-align:top;font-family:Arial;"> | </td>
<td align="left" style="vertical-align:top;color:#3B3739;font-family:Arial;">M: <a href="tel:+34%20619%20728%20249" target="_blank" id="LPlnk689713" style="text-decoration:none;color:#3B3739;"><strong style="font-weight:400;">+34 619 728 249</strong></a></td>
<td align="left" style="vertical-align:top;font-family:Arial;"> | <br>
</td>
<td align="left" style="vertical-align:top;color:#0B4CB4;font-family:Arial;"><a href="mailto:franciscojavier.lopez@solera.com" target="_blank" id="LPlnk689713" style="text-decoration:none;color:#0B4CB4;"><strong style="font-weight:400;">franciscojavier.lopez@solera.com</strong></a></td>
<td align="left" style="vertical-align:top;font-family:Arial;"> | </td>
<td align="left" style="vertical-align:top;color:#0B4CB4;font-family:Arial;"><a href="https://www.solera.com/" target="_blank" id="LPlnk689713" title="https://www.solera.com/" style="text-decoration:none;color:#0B4CB4;"><strong style="font-weight:400;">Solera.com</strong></a></td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr style="font-size:0;">
<td align="left" style="vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;color:#3B3739;font-style:normal;font-weight:400;white-space:nowrap;">
<tbody>
<tr style="font-size:13.33px;">
<td align="left" style="vertical-align:top;font-family:Arial;">Audatex Datos, S.A.</td>
<td align="left" style="vertical-align:top;color:#EA7200;font-family:Arial;"> | </td>
<td align="left" style="vertical-align:top;font-family:Arial;">Avda. de Bruselas, 36, Salida 16, A‑1 (Diversia)</td>
<td align="left" style="vertical-align:top;font-family:Arial;">, </td>
<td align="left" style="vertical-align:top;font-family:Arial;">Alcobendas</td>
<td align="left" style="vertical-align:top;font-family:Arial;">, </td>
<td align="left" style="vertical-align:top;font-family:Arial;">Madrid</td>
<td align="left" style="vertical-align:top;font-family:Arial;">, </td>
<td align="left" style="vertical-align:top;font-family:Arial;">28108</td>
<td align="left" style="vertical-align:top;font-family:Arial;">, </td>
<td align="left" style="vertical-align:top;font-family:Arial;">Spain</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr style="font-size:0;line-height:normal;">
<td align="left" style="padding:10px 0;vertical-align:top;"><img src="cid:image790996.png@A70D2A26.F4AADDCB" border="0" alt="" style="font-size:0;"></td>
</tr>
<tr style="font-size:0;">
<td style="padding:0;"> </td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
On 5/21/2019 6:19 PM, Ken Gaillot wrote:<br>
</div>
<blockquote type="cite" cite="mid:f051f197f70e0a6c668f1e761d343bf4af9e75cc.camel@redhat.com">
<pre class="moz-quote-pre" wrap="">On Tue, 2019-05-21 at 11:10 +0000, Lopez, Francisco Javier [Global IT]
wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Hello guys !
Need your help to try to understand and debug what I'm facing in one
of my clusters.
I set up fencing with this detail:
# pcs -f stonith_cfg stonith create fence_ao_pg01 fence_vmware_soap
ipaddr=<IP> ssl_insecure=1 login="<User>" passwd="<Passwd>"
pcmk_reboot_action=reboot pcmk_host_list="ao-pg01-p.axadmin.net"
power_wait=3 op monitor interval=60s
# pcs -f stonith_cfg stonith create fence_ao_pg02 fence_vmware_soap
ipaddr=<IP> ssl_insecure=1 login="<User>" passwd="<Passwd>"
pcmk_reboot_action=reboot pcmk_host_list="ao-pg02-p.axadmin.net"
power_wait=3 op monitor interval=60s
# pcs -f stonith_cfg constraint location fence_ao_pg01 avoids ao-
pg01-p.axadmin.net=INFINITY
# pcs -f stonith_cfg constraint location fence_ao_pg02 avoids ao-
pg02-p.axadmin.net=INFINITY
# pcs cluster cib-push stonith_cfg
The pcs status shows all ok during some time and then it turns to:
[root@ao-pg01-p ~]# pcs status --full
Cluster name: ao_cl_p_01
Stack: corosync
Current DC: ao-pg01-p.axadmin.net (1) (version 1.1.19-8.el7_6.4-
c3c624ea3d) - partition with quorum
Last updated: Tue May 21 12:18:46 2019
Last change: Fri May 17 18:54:32 2019 by hacluster via crmd on ao-
pg01-p.axadmin.net
2 nodes configured
3 resources configured
Online: [ ao-pg01-p.axadmin.net (1) ao-pg02-p.axadmin.net (2) ]
Full list of resources:
ao-cl-p-01-vip01 (ocf::heartbeat:IPaddr2): Started ao-pg01-
p.axadmin.net
fence_ao_pg01 (stonith:fence_vmware_soap): Stopped
fence_ao_pg02 (stonith:fence_vmware_soap): Stopped
Node Attributes:
* Node ao-pg01-p.axadmin.net (1):
* Node ao-pg02-p.axadmin.net (2):
Migration Summary:
* Node ao-pg02-p.axadmin.net (2):
fence_ao_pg01: migration-threshold=1000000 fail-count=1000000
last-failure='Sat May 18 00:22:22 2019'
* Node ao-pg01-p.axadmin.net (1):
fence_ao_pg02: migration-threshold=1000000 fail-count=1000000
last-failure='Fri May 17 20:52:53 2019'
Failed Actions:
* fence_ao_pg01_start_0 on ao-pg02-p.axadmin.net 'unknown error' (1):
call=22, status=Timed Out, exitreason='',
last-rc-change='Sat May 18 00:19:49 2019', queued=0ms,
exec=20022ms
* fence_ao_pg02_start_0 on ao-pg01-p.axadmin.net 'unknown error' (1):
call=84, status=Timed Out, exitreason='',
last-rc-change='Fri May 17 20:52:33 2019', queued=0ms,
exec=20032ms
PCSD Status:
ao-pg02-p.axadmin.net: Online
ao-pg01-p.axadmin.net: Online
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
>From the output I see there seems to be a 'Timed Out' but I'd like to
understand if this is a configuration issue
or something else I'm not aware of.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
When pacemaker starts a fence device, it issues a monitor command to
the fence agent. That command is what's timing out here.
The first thing I'd try is running the monitor command manually using
the parameters in the device configuration. The fence agent likely has
a debug option you could turn on to get more details.
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">
I'm attaching part of the log that shows the problem related to 17-
May.
Regards
Francisco Javier Lopez IT System Engineer |
Global IT O: +34 619 728 249 | M: +34 619 728 249
|
<a class="moz-txt-link-abbreviated" href="mailto:franciscojavier.lopez@solera.com">franciscojavier.lopez@solera.com</a> | Solera.com Aud
atex Datos, S.A. | Avda. de Bruselas, 36, Salida 16, A‑1
(Diversia) , Alcobendas , Madrid , 28108
, Spain
" Este e-mail y sus archivos adjuntos son confidenciales y están
dirigidos exclusivamente a la(s) persona(s) destinataria prevista. Si
ha recibido este mensaje por error, por favor, notifique
inmediatamente al remitente y elimine este mensaje. La empresa no
firma contratos por e-mail y todas las negociaciones están sujetas a
la firma de un contrato por escrito.
This e-mail and any attached files are confidential and intended for
the named addressee(s) only. If you have received this message in
error, please notify the sender and delete the email immediately. The
company does not conclude contracts by email and all negotiations are
subject to written contract. "
_______________________________________________
Manage your subscription:
<a class="moz-txt-link-freetext" href="https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.clusterlabs.org%2Fmailman%2Flistinfo%2Fusers&data=01%7C01%7C%7Cf499cca6634445d48c4008d6de082302%7Cc45b48f313bb448b9356ba7b863c2189%7C1&sdata=iPCgwWckXvP91cmB9NiZD6hYcPujBe6asBDwjG7avG8%3D&reserved=0">https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.clusterlabs.org%2Fmailman%2Flistinfo%2Fusers&data=01%7C01%7C%7Cf499cca6634445d48c4008d6de082302%7Cc45b48f313bb448b9356ba7b863c2189%7C1&sdata=iPCgwWckXvP91cmB9NiZD6hYcPujBe6asBDwjG7avG8%3D&reserved=0</a>
ClusterLabs home: <a class="moz-txt-link-freetext" href="https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.clusterlabs.org%2F&data=01%7C01%7C%7Cf499cca6634445d48c4008d6de082302%7Cc45b48f313bb448b9356ba7b863c2189%7C1&sdata=6C%2BVkrMHkAXJK%2FhCXbUbI94zdAwtM4EC4R8tvKdHim8%3D&reserved=0">https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.clusterlabs.org%2F&data=01%7C01%7C%7Cf499cca6634445d48c4008d6de082302%7Cc45b48f313bb448b9356ba7b863c2189%7C1&sdata=6C%2BVkrMHkAXJK%2FhCXbUbI94zdAwtM4EC4R8tvKdHim8%3D&reserved=0</a>
</pre>
</blockquote>
</blockquote>
<br>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
" Este e-mail y sus archivos adjuntos son confidenciales y están dirigidos exclusivamente a la(s) persona(s) destinataria prevista. Si ha recibido este mensaje por error, por favor, notifique inmediatamente al remitente y elimine este mensaje. La empresa no
firma contratos por e-mail y todas las negociaciones están sujetas a la firma de un contrato por escrito.
<br>
<br>
This e-mail and any attached files are confidential and intended for the named addressee(s) only. If you have received this message in error, please notify the sender and delete the email immediately. The company does not conclude contracts by email and all
negotiations are subject to written contract. "<br>
</font>
</body>
</html>