[ClusterLabs] fence_azure_arm flooding syslog

Mon Feb 11 08:58:22 EST 2019

You can try adding verbose=1 and see if that gives you some more
details on where it's failing.

On 11/02/19 14:47 +0100, Thomas Berreis wrote:
>We need this feature because shutdown / reboot takes too much time ( > 5
>min) and network fencing fences the virtual machine much faster ( < 5 sec).
>We finished all the required steps and network fencing works as expected but
>I'm still confused about these errors in the log and the failure counts
>showed by pcs ...
>
>fence_azure_arm: Keyring cache token has failed: No recommended backend was
>available. Install the keyrings.alt package if you want to use the
>non-recommended backends. See README.rst for details.
>
>stonith-ng[7789]: warning: fence_azure_arm[7896] stderr: [ 2019-02-11
>13:41:19,178 WARNING: Keyring cache token has failed: No recommended backend
>was available. Install the keyrings.alt package if you want to use the
>non-recommended backends. See README.rst for details. ]
>
>stonith-net_monitor_60000 on node02 'unknown error' (1): call=369,
>status=Timed Out, exitreason='', last-rc-change='Mon Feb 11 10:38:02 2019',
>queued=0ms, exec=20007ms
>
>Thomas
>
>-----Original Message-----
>From: Users [mailto:users-bounces at clusterlabs.org] On Behalf Of Oyvind
>Albrigtsen
>Sent: Montag, 11. Februar 2019 13:58
>To: Cluster Labs - All topics related to open-source clustering welcomed
><users at clusterlabs.org>
>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
>
>Oh. Thanks for clarifying.
>
>Network-fencing requires extra setup as explained if you run "pcs stonith
>resource describe fence_azure_arm".
>
>You probably dont need network-fencing, so you can just skip that part.
>
>On 11/02/19 13:50 +0100, Thomas Berreis wrote:
>>> It seems like the issue might be the space in "pcmk_host_list= node01".
>>Sorry, this typo was only in my mail because I anonymized in- and output.
>>
>>The issue only occurs when I add the parameter "network-fencing=on".
>>
>>WARNING:msrestazure.azure_active_directory:Keyring cache token has failed:
>>No recommended backend was available. Install the keyrings.alt package
>>if you want to use the non-recommended backends. See README.rst for
>details.
>>2019-02-11 12:29:37,079 WARNING: Keyring cache token has failed: No
>>recommended backend was available. Install the keyrings.alt package if
>>you want to use the non-recommended backends. See README.rst for details.
>>
>>
>>Thomas
>>
>>-----Original Message-----
>>From: Users [mailto:users-bounces at clusterlabs.org] On Behalf Of Oyvind
>>Albrigtsen
>>Sent: Montag, 11. Februar 2019 10:38
>>To: Cluster Labs - All topics related to open-source clustering
>>welcomed <users at clusterlabs.org>
>>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
>>
>>pcmk_host_list is used by Pacemaker to limit which hosts can be fenced
>>by the STONITH device, so you'll have to use the plug parameter for
>testing.
>>Pacemaker uses it internally when it fences a host, so you wouldnt use
>>plug= as part of the pcs stonith create command.
>>
>>It seems like the issue might be the space in "pcmk_host_list= node01".
>>
>>Oyvind
>>
>>On 10/02/19 14:43 +0100, Thomas Berreis wrote:
>>>Hi Oyvind,
>>>
>>>Seems, option "pcmk_host_list" is not supported by fence_azure_arm.
>>>Instead fence_azure_arm requires "plug" and however this is not
>>>supported
>>by pcs.
>>>What's the correct syntax to add such a stonith device with
>>fence_azure_arm?
>>>
>>>[root ~]# fence_azure_arm
>>>login=***
>>>network-fencing=on
>>>passwd=***
>>>pcmk_host_list= node01
>>>resourceGroup=***
>>>retry_on=0
>>>subscriptionId=***
>>>tenantId=***
>>>WARNING:root:Parse error: Ignoring unknown option 'pcmk_host_list= node01'
>>>
>>>ERROR:root:Failed: You have to enter plug number or machine
>>>identification
>>>
>>>2019-02-10 12:54:51,862 ERROR: Failed: You have to enter plug number
>>>or machine identification
>>>
>>>------------------------------
>>>
>>>[root ~]# fence_azure_arm
>>>login=***
>>>network-fencing=on
>>>passwd=***
>>>pcmk_host_list= node01
>>>resourceGroup=***
>>>retry_on=0
>>>subscriptionId=***
>>>tenantId=***
>>>plug=node01
>>>WARNING:root:Parse error: Ignoring unknown option 'pcmk_host_list= node01'
>>>
>>>WARNING:msrestazure.azure_active_directory:Keyring cache token has failed:
>>>No recommended backend was available.
>>>2019-02-10 12:56:30,377 WARNING: Keyring cache token has failed: No
>>>recommended backend was available. Install th
>>>WARNING:msrestazure.azure_active_directory:Keyring cache token has failed:
>>>No recommended backend was available.
>>>2019-02-10 12:56:30,729 WARNING: Keyring cache token has failed: No
>>>recommended backend was available. Install th
>>>Success: Powered OFF
>>>
>>>------------------------------
>>>
>>>[root ~]# fence_azure_arm
>>>login=***
>>>network-fencing=on
>>>passwd=***
>>>plug=node01
>>>resourceGroup=***
>>>retry_on=0
>>>subscriptionId=***
>>>tenantId=***
>>>
>>>Success: Powered OFF
>>>
>>>------------------------------
>>>
>>>[root ~]# pcs stonith create stonith-net fence_azure_arm [...]
>>>plug=node01 network-fencing=on
>>>Error: invalid stonith option 'plug'
>>>
>>>[root ~]# fence_azure_arm -h
>>>Usage:
>>>        fence_azure_arm [options]
>>>Options:
>>>[...]
>>>   -n, --plug=[id]                Physical plug number on device, UUID or
>>>identification of machine
>>>
>>>
>>>Thanks!
>>>
>>>Thomas
>>>
>>>-----Original Message-----
>>>From: Users [mailto:users-bounces at clusterlabs.org] On Behalf Of Oyvind
>>>Albrigtsen
>>>Sent: Freitag, 8. Februar 2019 14:16
>>>To: Cluster Labs - All topics related to open-source clustering
>>>welcomed <users at clusterlabs.org>
>>>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
>>>
>>>Hi Thomas,
>>>
>>>It looks like it's trying to use credentials from keyring, which
>>>generates these error messages.
>>>
>>>Maybe there's a typo in one of the authentitcation parameters in your
>>>config. Can you post the output of "pcs stonith show --full"?
>>>
>>>You can try running fence_azure_arm without parameters on the command
>>>line and copy the parameters (one per line) to it, and add e.g.
>>>action=status and end with Ctrl+D to see .
>>>
>>>
>>>Oyvind
>>>
>>>On 08/02/19 14:02 +0100, Thomas Berreis wrote:
>>>>Hi,
>>>>
>>>>
>>>>
>>>>I'm having a two-node cluster in Azure with fencing enabled via
>>>>module fence_azure_arm.
>>>>
>>>>The cluster management is flooding my syslog on both nodes with these
>>>>messages:
>>>>
>>>>
>>>>
>>>>** fence_azure_arm: Keyring cache token has failed: No recommended
>>>>backend was available. Install the keyrings.alt package if you want
>>>>to use the non-recommended backends. See README.rst for details.
>>>>
>>>>
>>>>
>>>>** stonith-ng[5220]: warning: fence_azure_arm[14847] stderr: [
>>>>2019-02-08
>>>>12:52:02,703 WARNING: Keyring cache token has failed: No recommended
>>>>backend was available. Install the keyrings.alt package if you want
>>>>to use the non-recommended backends. See README.rst for details. ]
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>Can anyone give me a hint how to fix these issue?
>>>>
>>>>
>>>>
>>>>pcs version: 0.9.165
>>>>
>>>>pacemaker version: 1.1.19-8.el7_6.1
>>>>
>>>>fence_azure_arm version: 4.2.1
>>>>
>>>>corosync 2.4.3
>>>>
>>>>
>>>>
>>>>Thanks!
>>>>
>>>>
>>>>
>>>>Thomas
>>>>
>>>
>>>>_______________________________________________
>>>>Users mailing list: Users at clusterlabs.org
>>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>>
>>>>Project Home: http://www.clusterlabs.org Getting started:
>>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>Bugs: http://bugs.clusterlabs.org
>>>
>>>_______________________________________________
>>>Users mailing list: Users at clusterlabs.org
>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>>Project Home: http://www.clusterlabs.org Getting started:
>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>Bugs: http://bugs.clusterlabs.org
>>>
>>>_______________________________________________
>>>Users mailing list: Users at clusterlabs.org
>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>>Project Home: http://www.clusterlabs.org Getting started:
>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>Bugs: http://bugs.clusterlabs.org
>>_______________________________________________
>>Users mailing list: Users at clusterlabs.org
>>https://lists.clusterlabs.org/mailman/listinfo/users
>>
>>Project Home: http://www.clusterlabs.org Getting started:
>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>Bugs: http://bugs.clusterlabs.org
>>
>>_______________________________________________
>>Users mailing list: Users at clusterlabs.org
>>https://lists.clusterlabs.org/mailman/listinfo/users
>>
>>Project Home: http://www.clusterlabs.org Getting started:
>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>Bugs: http://bugs.clusterlabs.org
>_______________________________________________
>Users mailing list: Users at clusterlabs.org
>https://lists.clusterlabs.org/mailman/listinfo/users
>
>Project Home: http://www.clusterlabs.org Getting started:
>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>Bugs: http://bugs.clusterlabs.org
>
>_______________________________________________
>Users mailing list: Users at clusterlabs.org
>https://lists.clusterlabs.org/mailman/listinfo/users
>
>Project Home: http://www.clusterlabs.org
>Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>Bugs: http://bugs.clusterlabs.org