[ClusterLabs] fence_azure_arm flooding syslog

Tue Feb 12 08:51:29 UTC 2019

At least that limits what the issue could be.

I'd check if the credentials allows getting and setting
network-settings:
https://github.com/ClusterLabs/fence-agents/blob/master/agents/azure_arm/fence_azure_arm.py#L204

On 11/02/19 16:46 +0100, Thomas Berreis wrote:
>Hi Oyvind,
>
>Debug mode doesn't help me. The error occurs after successfully getting
>Bearer token. No other issues are shown:
>
>DEBUG:requests_oauthlib.oauth2_session:Invoking 0 token response hooks.
>2019-02-11 15:27:12,687 DEBUG: Invoking 0 token response hooks.
>DEBUG:requests_oauthlib.oauth2_session:Obtained token {u'resource':
>u'https://management.core.windows.net/', u'access_token': ***',
>u'ext_expires_in': u'3599', u'expires_in': u'3599', u'expires_at':
>1549902431.687557, u'token_type': u'Bearer', u'not_before': u'1549898532',
>u'expires_on': u'1549902432'}.
>2019-02-11 15:27:12,687 DEBUG: Obtained token {u'resource':
>u'https://management.core.windows.net/', u'access_token': u'***',
>u'ext_expires_in': u'3599', u'expires_in': u'3599', u'expires_at':
>1549902431.687557, u'token_type': u'Bearer', u'not_before': u'1549898532',
>u'expires_on': u'1549902432'}.
>WARNING:msrestazure.azure_active_directory:Keyring cache token has failed:
>No recommended backend was available. Install the keyrings.alt package if
>you want to use the non-recommended backends. See README.rst for details.
>2019-02-11 15:27:12,687 WARNING: Keyring cache token has failed: No
>recommended backend was available. Install the keyrings.alt package if you
>want to use the non-recommended backends. See README.rst for details.
>DEBUG:requests_oauthlib.oauth2_session:Encoding `client_id` "***" with
>`client_secret` as Basic auth credentials.
>2019-02-11 15:27:12,727 DEBUG: Encoding `client_id` "***" with
>`client_secret` as Basic auth credentials.
>DEBUG:requests_oauthlib.oauth2_session:Requesting url
>https://login.microsoftonline.com/***/oauth2/token using method POST.
>2019-02-11 15:27:12,727 DEBUG: Requesting url
>https://login.microsoftonline.com/***/oauth2/token using method POST.
>
>2019-02-11 15:27:13,072 DEBUG: Obtained token {u'resource':
>u'https://management.core.windows.net/', u'access_token': u'e***w',
>u'ext_expires_in': u'3600', u'expires_in': u'3600', u'expires_at':
>1549902433.072768, u'token_type': u'Bearer', u'not_before': u'1549898533',
>u'expires_on': u'1549902433'}.
>WARNING:msrestazure.azure_active_directory:Keyring cache token has failed:
>No recommended backend was available. Install the keyrings.alt package if
>you want to use the non-recommended backends. See README.rst for details.
>2019-02-11 15:27:13,073 WARNING: Keyring cache token has failed: No
>recommended backend was available. Install the keyrings.alt package if you
>want to use the non-recommended backends. See README.rst for details.
>INFO:root:getting power status for VM node01
>
>Thomas
>
>-----Original Message-----
>From: Users [mailto:users-bounces at clusterlabs.org] On Behalf Of Oyvind
>Albrigtsen
>Sent: Montag, 11. Februar 2019 14:58
>To: Cluster Labs - All topics related to open-source clustering welcomed
><users at clusterlabs.org>
>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
>
>You can try adding verbose=1 and see if that gives you some more details on
>where it's failing.
>
>On 11/02/19 14:47 +0100, Thomas Berreis wrote:
>>We need this feature because shutdown / reboot takes too much time ( >
>>5
>>min) and network fencing fences the virtual machine much faster ( < 5 sec).
>>We finished all the required steps and network fencing works as
>>expected but I'm still confused about these errors in the log and the
>>failure counts showed by pcs ...
>>
>>fence_azure_arm: Keyring cache token has failed: No recommended backend
>>was available. Install the keyrings.alt package if you want to use the
>>non-recommended backends. See README.rst for details.
>>
>>stonith-ng[7789]: warning: fence_azure_arm[7896] stderr: [ 2019-02-11
>>13:41:19,178 WARNING: Keyring cache token has failed: No recommended
>>backend was available. Install the keyrings.alt package if you want to
>>use the non-recommended backends. See README.rst for details. ]
>>
>>stonith-net_monitor_60000 on node02 'unknown error' (1): call=369,
>>status=Timed Out, exitreason='', last-rc-change='Mon Feb 11 10:38:02
>>2019', queued=0ms, exec=20007ms
>>
>>Thomas
>>
>>-----Original Message-----
>>From: Users [mailto:users-bounces at clusterlabs.org] On Behalf Of Oyvind
>>Albrigtsen
>>Sent: Montag, 11. Februar 2019 13:58
>>To: Cluster Labs - All topics related to open-source clustering
>>welcomed <users at clusterlabs.org>
>>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
>>
>>Oh. Thanks for clarifying.
>>
>>Network-fencing requires extra setup as explained if you run "pcs
>>stonith resource describe fence_azure_arm".
>>
>>You probably dont need network-fencing, so you can just skip that part.
>>
>>On 11/02/19 13:50 +0100, Thomas Berreis wrote:
>>>> It seems like the issue might be the space in "pcmk_host_list= node01".
>>>Sorry, this typo was only in my mail because I anonymized in- and output.
>>>
>>>The issue only occurs when I add the parameter "network-fencing=on".
>>>
>>>WARNING:msrestazure.azure_active_directory:Keyring cache token has failed:
>>>No recommended backend was available. Install the keyrings.alt package
>>>if you want to use the non-recommended backends. See README.rst for
>>details.
>>>2019-02-11 12:29:37,079 WARNING: Keyring cache token has failed: No
>>>recommended backend was available. Install the keyrings.alt package if
>>>you want to use the non-recommended backends. See README.rst for details.
>>>
>>>
>>>Thomas
>>>
>>>-----Original Message-----
>>>From: Users [mailto:users-bounces at clusterlabs.org] On Behalf Of Oyvind
>>>Albrigtsen
>>>Sent: Montag, 11. Februar 2019 10:38
>>>To: Cluster Labs - All topics related to open-source clustering
>>>welcomed <users at clusterlabs.org>
>>>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
>>>
>>>pcmk_host_list is used by Pacemaker to limit which hosts can be fenced
>>>by the STONITH device, so you'll have to use the plug parameter for
>>testing.
>>>Pacemaker uses it internally when it fences a host, so you wouldnt use
>>>plug= as part of the pcs stonith create command.
>>>
>>>It seems like the issue might be the space in "pcmk_host_list= node01".
>>>
>>>Oyvind
>>>
>>>On 10/02/19 14:43 +0100, Thomas Berreis wrote:
>>>>Hi Oyvind,
>>>>
>>>>Seems, option "pcmk_host_list" is not supported by fence_azure_arm.
>>>>Instead fence_azure_arm requires "plug" and however this is not
>>>>supported
>>>by pcs.
>>>>What's the correct syntax to add such a stonith device with
>>>fence_azure_arm?
>>>>
>>>>[root ~]# fence_azure_arm
>>>>login=***
>>>>network-fencing=on
>>>>passwd=***
>>>>pcmk_host_list= node01
>>>>resourceGroup=***
>>>>retry_on=0
>>>>subscriptionId=***
>>>>tenantId=***
>>>>WARNING:root:Parse error: Ignoring unknown option 'pcmk_host_list=
>node01'
>>>>
>>>>ERROR:root:Failed: You have to enter plug number or machine
>>>>identification
>>>>
>>>>2019-02-10 12:54:51,862 ERROR: Failed: You have to enter plug number
>>>>or machine identification
>>>>
>>>>------------------------------
>>>>
>>>>[root ~]# fence_azure_arm
>>>>login=***
>>>>network-fencing=on
>>>>passwd=***
>>>>pcmk_host_list= node01
>>>>resourceGroup=***
>>>>retry_on=0
>>>>subscriptionId=***
>>>>tenantId=***
>>>>plug=node01
>>>>WARNING:root:Parse error: Ignoring unknown option 'pcmk_host_list=
>node01'
>>>>
>>>>WARNING:msrestazure.azure_active_directory:Keyring cache token has
>failed:
>>>>No recommended backend was available.
>>>>2019-02-10 12:56:30,377 WARNING: Keyring cache token has failed: No
>>>>recommended backend was available. Install th
>>>>WARNING:msrestazure.azure_active_directory:Keyring cache token has
>failed:
>>>>No recommended backend was available.
>>>>2019-02-10 12:56:30,729 WARNING: Keyring cache token has failed: No
>>>>recommended backend was available. Install th
>>>>Success: Powered OFF
>>>>
>>>>------------------------------
>>>>
>>>>[root ~]# fence_azure_arm
>>>>login=***
>>>>network-fencing=on
>>>>passwd=***
>>>>plug=node01
>>>>resourceGroup=***
>>>>retry_on=0
>>>>subscriptionId=***
>>>>tenantId=***
>>>>
>>>>Success: Powered OFF
>>>>
>>>>------------------------------
>>>>
>>>>[root ~]# pcs stonith create stonith-net fence_azure_arm [...]
>>>>plug=node01 network-fencing=on
>>>>Error: invalid stonith option 'plug'
>>>>
>>>>[root ~]# fence_azure_arm -h
>>>>Usage:
>>>>        fence_azure_arm [options]
>>>>Options:
>>>>[...]
>>>>   -n, --plug=[id]                Physical plug number on device, UUID or
>>>>identification of machine
>>>>
>>>>
>>>>Thanks!
>>>>
>>>>Thomas
>>>>
>>>>-----Original Message-----
>>>>From: Users [mailto:users-bounces at clusterlabs.org] On Behalf Of
>>>>Oyvind Albrigtsen
>>>>Sent: Freitag, 8. Februar 2019 14:16
>>>>To: Cluster Labs - All topics related to open-source clustering
>>>>welcomed <users at clusterlabs.org>
>>>>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
>>>>
>>>>Hi Thomas,
>>>>
>>>>It looks like it's trying to use credentials from keyring, which
>>>>generates these error messages.
>>>>
>>>>Maybe there's a typo in one of the authentitcation parameters in your
>>>>config. Can you post the output of "pcs stonith show --full"?
>>>>
>>>>You can try running fence_azure_arm without parameters on the command
>>>>line and copy the parameters (one per line) to it, and add e.g.
>>>>action=status and end with Ctrl+D to see .
>>>>
>>>>
>>>>Oyvind
>>>>
>>>>On 08/02/19 14:02 +0100, Thomas Berreis wrote:
>>>>>Hi,
>>>>>
>>>>>
>>>>>
>>>>>I'm having a two-node cluster in Azure with fencing enabled via
>>>>>module fence_azure_arm.
>>>>>
>>>>>The cluster management is flooding my syslog on both nodes with
>>>>>these
>>>>>messages:
>>>>>
>>>>>
>>>>>
>>>>>** fence_azure_arm: Keyring cache token has failed: No recommended
>>>>>backend was available. Install the keyrings.alt package if you want
>>>>>to use the non-recommended backends. See README.rst for details.
>>>>>
>>>>>
>>>>>
>>>>>** stonith-ng[5220]: warning: fence_azure_arm[14847] stderr: [
>>>>>2019-02-08
>>>>>12:52:02,703 WARNING: Keyring cache token has failed: No recommended
>>>>>backend was available. Install the keyrings.alt package if you want
>>>>>to use the non-recommended backends. See README.rst for details. ]
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>Can anyone give me a hint how to fix these issue?
>>>>>
>>>>>
>>>>>
>>>>>pcs version: 0.9.165
>>>>>
>>>>>pacemaker version: 1.1.19-8.el7_6.1
>>>>>
>>>>>fence_azure_arm version: 4.2.1
>>>>>
>>>>>corosync 2.4.3
>>>>>
>>>>>
>>>>>
>>>>>Thanks!
>>>>>
>>>>>
>>>>>
>>>>>Thomas
>>>>>
>>>>
>>>>>_______________________________________________
>>>>>Users mailing list: Users at clusterlabs.org
>>>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>
>>>>>Project Home: http://www.clusterlabs.org Getting started:
>>>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>Bugs: http://bugs.clusterlabs.org
>>>>
>>>>_______________________________________________
>>>>Users mailing list: Users at clusterlabs.org
>>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>>
>>>>Project Home: http://www.clusterlabs.org Getting started:
>>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>Bugs: http://bugs.clusterlabs.org
>>>>
>>>>_______________________________________________
>>>>Users mailing list: Users at clusterlabs.org
>>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>>
>>>>Project Home: http://www.clusterlabs.org Getting started:
>>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>Bugs: http://bugs.clusterlabs.org
>>>_______________________________________________
>>>Users mailing list: Users at clusterlabs.org
>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>>Project Home: http://www.clusterlabs.org Getting started:
>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>Bugs: http://bugs.clusterlabs.org
>>>
>>>_______________________________________________
>>>Users mailing list: Users at clusterlabs.org
>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>>Project Home: http://www.clusterlabs.org Getting started:
>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>Bugs: http://bugs.clusterlabs.org
>>_______________________________________________
>>Users mailing list: Users at clusterlabs.org
>>https://lists.clusterlabs.org/mailman/listinfo/users
>>
>>Project Home: http://www.clusterlabs.org Getting started:
>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>Bugs: http://bugs.clusterlabs.org
>>
>>_______________________________________________
>>Users mailing list: Users at clusterlabs.org
>>https://lists.clusterlabs.org/mailman/listinfo/users
>>
>>Project Home: http://www.clusterlabs.org Getting started:
>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>Bugs: http://bugs.clusterlabs.org
>_______________________________________________
>Users mailing list: Users at clusterlabs.org
>https://lists.clusterlabs.org/mailman/listinfo/users
>
>Project Home: http://www.clusterlabs.org Getting started:
>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>Bugs: http://bugs.clusterlabs.org
>
>_______________________________________________
>Users mailing list: Users at clusterlabs.org
>https://lists.clusterlabs.org/mailman/listinfo/users
>
>Project Home: http://www.clusterlabs.org
>Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>Bugs: http://bugs.clusterlabs.org