[ClusterLabs] Antw: [EXT] Failed fencing monitor process (fence_vmware_soap) RHEL 8
Ken Gaillot
kgaillot at redhat.com
Thu Jun 18 11:24:48 EDT 2020
Note that a failed start of a stonith device will not prevent the
cluster from using that device for fencing. It just prevents the
cluster from monitoring the device.
On Thu, 2020-06-18 at 08:20 +0000, Strahil Nikolov wrote:
> What about second fencing mechanism ?
> You can add a shared (independent) vmdk as an sbd device. The
> reconfiguration will require cluster downtime, but this is only
> necessary once.
> Once 2 fencing mechanisms are available - you can configure the order
> easily.
> Best Regards,
> Strahil Nikolov
>
>
>
>
>
>
> В четвъртък, 18 юни 2020 г., 10:29:22 Гринуич+3, Ulrich Windl <
> ulrich.windl at rz.uni-regensburg.de> написа:
>
>
>
>
>
> Hi!
>
> I can't give much detailed advice, but I think any network service
> should have a timeout of at least 30 Sekonds (you have
> timeout=20000ms).
>
> And "after 1000000 failures" is symbolic, not literal: It means it
> failed too often, so I won't retry.
>
> Regards,
> Ulrich
>
> > > > Howard <hmoneta at gmail.com> schrieb am 17.06.2020 um 21:05 in
> > > > Nachricht
>
> <2817_1592420740_5EEA6983_2817_3_1_CAO51vj6oXjfvhGQz7oOu=Pi+D_cKh5M1g
> fDL_2tAbKmw
> mQYeA at mail.gmail.com>:
> > Hello, recently I received some really great advice from this
> > community
> > regarding changing the token timeout value in corosync. Thank you!
> > Since
> > then the cluster has been working perfectly with no errors in the
> > log for
> > more than a week.
> >
> > This morning I logged in to find a stopped stonith device. If I'm
> > reading
> > the log right, it looks like it failed 1 million times in ~20
> > seconds then
> > gave up. If you wouldn't mind looking at the logs below, is there
> > some way
> > that I can make this more robust so that it can recover? I'll be
> > investigating the reason for the timeout but would like to help the
> > system
> > recover on its own.
> >
> > Servers: RHEL 8.2
> >
> > Cluster name: cluster_pgperf2
> > Stack: corosync
> > Current DC: srv1 (version 2.0.2-3.el8_1.2-744a30d655) - partition
> > with
> > quorum
> > Last updated: Wed Jun 17 11:47:42 2020
> > Last change: Tue Jun 16 22:00:29 2020 by root via crm_attribute on
> > srv1
> >
> > 2 nodes configured
> > 4 resources configured
> >
> > Online: [ srv1 srv2 ]
> >
> > Full list of resources:
> >
> > Clone Set: pgsqld-clone [pgsqld] (promotable)
> > Masters: [ srv1 ]
> > Slaves: [ srv2 ]
> > pgsql-master-ip (ocf::heartbeat:IPaddr2): Started
> > srv1
> > vmfence (stonith:fence_vmware_soap): Stopped
> >
> > Failed Resource Actions:
> > * vmfence_start_0 on srv2 'OCF_TIMEOUT' (198): call=19,
> > status=Timed Out,
> > exitreason='',
> > last-rc-change='Wed Jun 17 08:34:16 2020', queued=7ms,
> > exec=20184ms
> > * vmfence_start_0 on srv1 'OCF_TIMEOUT' (198): call=44,
> > status=Timed Out,
> > exitreason='',
> > last-rc-change='Wed Jun 17 08:33:55 2020', queued=0ms,
> > exec=20008ms
> >
> > Daemon Status:
> > corosync: active/disabled
> > pacemaker: active/disabled
> > pcsd: active/enabled
> >
> > pcs resource config
> > Clone: pgsqld-clone
> > Meta Attrs: notify=true promotable=true
> > Resource: pgsqld (class=ocf provider=heartbeat type=pgsqlms)
> > Attributes: bindir=/usr/bin pgdata=/var/lib/pgsql/data
> > Operations: demote interval=0s timeout=120s (pgsqld-demote-
> > interval-0s)
> > methods interval=0s timeout=5 (pgsqld-methods-
> > interval-0s)
> > monitor interval=15s role=Master timeout=60s
> > (pgsqld-monitor-interval-15s)
> > monitor interval=16s role=Slave timeout=60s
> > (pgsqld-monitor-interval-16s)
> > notify interval=0s timeout=60s (pgsqld-notify-
> > interval-0s)
> > promote interval=0s timeout=30s (pgsqld-promote-
> > interval-0s)
> > reload interval=0s timeout=20 (pgsqld-reload-
> > interval-0s)
> > start interval=0s timeout=60s (pgsqld-start-
> > interval-0s)
> > stop interval=0s timeout=60s (pgsqld-stop-interval-
> > 0s)
> > monitor interval=60s timeout=60s
> > (pgsqld-monitor-interval-60s)
> > Resource: pgsql-master-ip (class=ocf provider=heartbeat
> > type=IPaddr2)
> > Attributes: cidr_netmask=24 ip=xxx.xxx.xxx.xxx
> > Operations: monitor interval=10s (pgsql-master-ip-monitor-
> > interval-10s)
> > start interval=0s timeout=20s
> > (pgsql-master-ip-start-interval-0s)
> > stop interval=0s timeout=20s
> > (pgsql-master-ip-stop-interval-0s)
> >
> > pcs stonith config
> > Resource: vmfence (class=stonith type=fence_vmware_soap)
> > Attributes: ipaddr=xxx.xxx.xxx.xxx login=xxxx\xxxxxxxx
> > passwd_script=xxxxxxxx pcmk_host_map=srv1:xxxxxxxxx;srv2:yyyyyyyyy
> > ssl=1
> > ssl_insecure=1
> > Operations: monitor interval=60s (vmfence-monitor-interval-60s)
> >
> > pcs resource failcount show
> > Failcounts for resource 'vmfence'
> > srv1: INFINITY
> > srv2: INFINITY
> >
> > Here are the versions installed:
> > [postgres at srv1 cluster]$ rpm -qa|grep
> > "pacemaker\|pcs\|corosync\|fence-agents-vmware-soap\|paf"
> > corosync-3.0.2-3.el8_1.1.x86_64
> > corosync-qdevice-3.0.0-2.el8.x86_64
> > corosync-qnetd-3.0.0-2.el8.x86_64
> > corosynclib-3.0.2-3.el8_1.1.x86_64
> > fence-agents-vmware-soap-4.2.1-41.el8.noarch
> > pacemaker-2.0.2-3.el8_1.2.x86_64
> > pacemaker-cli-2.0.2-3.el8_1.2.x86_64
> > pacemaker-cluster-libs-2.0.2-3.el8_1.2.x86_64
> > pacemaker-libs-2.0.2-3.el8_1.2.x86_64
> > pacemaker-schemas-2.0.2-3.el8_1.2.noarch
> > pcs-0.10.2-4.el8.x86_64
> > resource-agents-paf-2.3.0-1.noarch
> >
> > Here are the errors and warnings from the pacemaker.log from the
> > first
> > warning until it gave up.
> >
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:33:55 srv1 pacemaker-
> > fenced
> > [26722] (child_timeout_callback) warning:
> > fence_vmware_soap_monitor_1 process (PID 43095) timed out
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:33:55 srv1 pacemaker-
> > fenced
> > [26722] (operation_finished) warning:
> > fence_vmware_soap_monitor_1:43095 - timed out after 20000ms
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:33:55 srv1 pacemaker-
> > controld
> > [26726] (process_lrm_event) error: Result of monitor
> > operation for
> > vmfence on srv1: Timed Out | call=39 key=vmfence_monitor_60000
> > timeout=20000ms
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:33:55 srv1
> > pacemaker-schedulerd[26725] (unpack_rsc_op_failure) warning:
> > Processing
> > failed monitor of vmfence on srv1: OCF_TIMEOUT | rc=198
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:15 srv1 pacemaker-
> > fenced
> > [26722] (child_timeout_callback) warning:
> > fence_vmware_soap_monitor_1 process (PID 43215) timed out
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:15 srv1 pacemaker-
> > fenced
> > [26722] (operation_finished) warning:
> > fence_vmware_soap_monitor_1:43215 - timed out after 20000ms
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:15 srv1 pacemaker-
> > controld
> > [26726] (process_lrm_event) error: Result of start operation
> > for
> > vmfence on srv1: Timed Out | call=44 key=vmfence_start_0
> > timeout=20000ms
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:15 srv1 pacemaker-
> > controld
> > [26726] (status_from_rc) warning: Action 39
> > (vmfence_start_0) on
> > srv1 failed (target: 0 vs. rc: 198): Error
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:15 srv1
> > pacemaker-schedulerd[26725] (unpack_rsc_op_failure) warning:
> > Processing
> > failed start of vmfence on srv1: OCF_TIMEOUT | rc=198
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:15 srv1
> > pacemaker-schedulerd[26725] (unpack_rsc_op_failure) warning:
> > Processing
> > failed start of vmfence on srv1: OCF_TIMEOUT | rc=198
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:15 srv1
> > pacemaker-schedulerd[26725] (unpack_rsc_op_failure) warning:
> > Processing
> > failed start of vmfence on srv1: OCF_TIMEOUT | rc=198
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:15 srv1
> > pacemaker-schedulerd[26725] (unpack_rsc_op_failure) warning:
> > Processing
> > failed start of vmfence on srv1: OCF_TIMEOUT | rc=198
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:15 srv1
> > pacemaker-schedulerd[26725] (check_migration_threshold)
> > warning:
> > Forcing vmfence away from srv1 after 1000000 failures (max=5)
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:15 srv1
> > pacemaker-schedulerd[26725] (unpack_rsc_op_failure) warning:
> > Processing
> > failed start of vmfence on srv1: OCF_TIMEOUT | rc=198
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:15 srv1
> > pacemaker-schedulerd[26725] (unpack_rsc_op_failure) warning:
> > Processing
> > failed start of vmfence on srv1: OCF_TIMEOUT | rc=198
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:15 srv1
> > pacemaker-schedulerd[26725] (check_migration_threshold)
> > warning:
> > Forcing vmfence away from srv1 after 1000000 failures (max=5)
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:36 srv1 pacemaker-
> > controld
> > [26726] (status_from_rc) warning: Action 38
> > (vmfence_start_0) on
> > srv2 failed (target: 0 vs. rc: 198): Error
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:36 srv1
> > pacemaker-schedulerd[26725] (unpack_rsc_op_failure) warning:
> > Processing
> > failed start of vmfence on srv2: OCF_TIMEOUT | rc=198
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:36 srv1
> > pacemaker-schedulerd[26725] (unpack_rsc_op_failure) warning:
> > Processing
> > failed start of vmfence on srv2: OCF_TIMEOUT | rc=198
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:36 srv1
> > pacemaker-schedulerd[26725] (unpack_rsc_op_failure) warning:
> > Processing
> > failed start of vmfence on srv1: OCF_TIMEOUT | rc=198
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:36 srv1
> > pacemaker-schedulerd[26725] (check_migration_threshold)
> > warning:
> > Forcing vmfence away from srv1 after 1000000 failures (max=5)
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:36 srv1
> > pacemaker-schedulerd[26725] (unpack_rsc_op_failure) warning:
> > Processing
> > failed start of vmfence on srv2: OCF_TIMEOUT | rc=198
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:36 srv1
> > pacemaker-schedulerd[26725] (unpack_rsc_op_failure) warning:
> > Processing
> > failed start of vmfence on srv2: OCF_TIMEOUT | rc=198
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:36 srv1
> > pacemaker-schedulerd[26725] (unpack_rsc_op_failure) warning:
> > Processing
> > failed start of vmfence on srv1: OCF_TIMEOUT | rc=198
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:36 srv1
> > pacemaker-schedulerd[26725] (check_migration_threshold)
> > warning:
> > Forcing vmfence away from srv1 after 1000000 failures (max=5)
> > /var/log/pacemaker/pacemaker.log:Jun 17 08:34:36 srv1
> > pacemaker-schedulerd[26725] (check_migration_threshold)
> > warning:
> > Forcing vmfence away from srv2 after 1000000 failures (max=5)
>
>
>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
--
Ken Gaillot <kgaillot at redhat.com>
More information about the Users
mailing list