[ClusterLabs] fence_azure_arm flooding syslog

Wed Feb 13 04:02:31 EST 2019

That's weird. I initially tested it in US East 2.

I'd check that the packages/versions for python-azure-sdk,
python-msrest, python-msrestazure, python-keyring, etc are the same
on your nodes in Europe vs US locations.

If they are the same you could create a support ticket on Azure to
ask if there's been any changes to the API in the US, or if they can
see what's causing it to fail in the US location and not in Europe.

On 12/02/19 17:37 +0100, Thomas Berreis wrote:
>Network fencing is working correctly therefore credentials should be fine.
>Maybe there are differences between Azure Cloud in Europe and US because I
>get these errors only with our clusters in US Central and US East 2 but not
>in Europe West and North. Any hint?
>
>Thomas
>
>-----Original Message-----
>From: Users [mailto:users-bounces at clusterlabs.org] On Behalf Of Oyvind
>Albrigtsen
>Sent: Dienstag, 12. Februar 2019 09:51
>To: Cluster Labs - All topics related to open-source clustering welcomed
><users at clusterlabs.org>
>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
>
>At least that limits what the issue could be.
>
>I'd check if the credentials allows getting and setting
>network-settings:
>https://github.com/ClusterLabs/fence-agents/blob/master/agents/azure_arm/fen
>ce_azure_arm.py#L204
>
>On 11/02/19 16:46 +0100, Thomas Berreis wrote:
>>Hi Oyvind,
>>
>>Debug mode doesn't help me. The error occurs after successfully getting
>>Bearer token. No other issues are shown:
>>
>>DEBUG:requests_oauthlib.oauth2_session:Invoking 0 token response hooks.
>>2019-02-11 15:27:12,687 DEBUG: Invoking 0 token response hooks.
>>DEBUG:requests_oauthlib.oauth2_session:Obtained token {u'resource':
>>u'https://management.core.windows.net/', u'access_token': ***',
>>u'ext_expires_in': u'3599', u'expires_in': u'3599', u'expires_at':
>>1549902431.687557, u'token_type': u'Bearer', u'not_before':
>>u'1549898532',
>>u'expires_on': u'1549902432'}.
>>2019-02-11 15:27:12,687 DEBUG: Obtained token {u'resource':
>>u'https://management.core.windows.net/', u'access_token': u'***',
>>u'ext_expires_in': u'3599', u'expires_in': u'3599', u'expires_at':
>>1549902431.687557, u'token_type': u'Bearer', u'not_before':
>>u'1549898532',
>>u'expires_on': u'1549902432'}.
>>WARNING:msrestazure.azure_active_directory:Keyring cache token has failed:
>>No recommended backend was available. Install the keyrings.alt package
>>if you want to use the non-recommended backends. See README.rst for
>details.
>>2019-02-11 15:27:12,687 WARNING: Keyring cache token has failed: No
>>recommended backend was available. Install the keyrings.alt package if
>>you want to use the non-recommended backends. See README.rst for details.
>>DEBUG:requests_oauthlib.oauth2_session:Encoding `client_id` "***" with
>>`client_secret` as Basic auth credentials.
>>2019-02-11 15:27:12,727 DEBUG: Encoding `client_id` "***" with
>>`client_secret` as Basic auth credentials.
>>DEBUG:requests_oauthlib.oauth2_session:Requesting url
>>https://login.microsoftonline.com/***/oauth2/token using method POST.
>>2019-02-11 15:27:12,727 DEBUG: Requesting url
>>https://login.microsoftonline.com/***/oauth2/token using method POST.
>>
>>2019-02-11 15:27:13,072 DEBUG: Obtained token {u'resource':
>>u'https://management.core.windows.net/', u'access_token': u'e***w',
>>u'ext_expires_in': u'3600', u'expires_in': u'3600', u'expires_at':
>>1549902433.072768, u'token_type': u'Bearer', u'not_before':
>>u'1549898533',
>>u'expires_on': u'1549902433'}.
>>WARNING:msrestazure.azure_active_directory:Keyring cache token has failed:
>>No recommended backend was available. Install the keyrings.alt package
>>if you want to use the non-recommended backends. See README.rst for
>details.
>>2019-02-11 15:27:13,073 WARNING: Keyring cache token has failed: No
>>recommended backend was available. Install the keyrings.alt package if
>>you want to use the non-recommended backends. See README.rst for details.
>>INFO:root:getting power status for VM node01
>>
>>Thomas
>>
>>-----Original Message-----
>>From: Users [mailto:users-bounces at clusterlabs.org] On Behalf Of Oyvind
>>Albrigtsen
>>Sent: Montag, 11. Februar 2019 14:58
>>To: Cluster Labs - All topics related to open-source clustering
>>welcomed <users at clusterlabs.org>
>>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
>>
>>You can try adding verbose=1 and see if that gives you some more
>>details on where it's failing.
>>
>>On 11/02/19 14:47 +0100, Thomas Berreis wrote:
>>>We need this feature because shutdown / reboot takes too much time ( >
>>>5
>>>min) and network fencing fences the virtual machine much faster ( < 5
>sec).
>>>We finished all the required steps and network fencing works as
>>>expected but I'm still confused about these errors in the log and the
>>>failure counts showed by pcs ...
>>>
>>>fence_azure_arm: Keyring cache token has failed: No recommended
>>>backend was available. Install the keyrings.alt package if you want to
>>>use the non-recommended backends. See README.rst for details.
>>>
>>>stonith-ng[7789]: warning: fence_azure_arm[7896] stderr: [ 2019-02-11
>>>13:41:19,178 WARNING: Keyring cache token has failed: No recommended
>>>backend was available. Install the keyrings.alt package if you want to
>>>use the non-recommended backends. See README.rst for details. ]
>>>
>>>stonith-net_monitor_60000 on node02 'unknown error' (1): call=369,
>>>status=Timed Out, exitreason='', last-rc-change='Mon Feb 11 10:38:02
>>>2019', queued=0ms, exec=20007ms
>>>
>>>Thomas
>>>
>>>-----Original Message-----
>>>From: Users [mailto:users-bounces at clusterlabs.org] On Behalf Of Oyvind
>>>Albrigtsen
>>>Sent: Montag, 11. Februar 2019 13:58
>>>To: Cluster Labs - All topics related to open-source clustering
>>>welcomed <users at clusterlabs.org>
>>>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
>>>
>>>Oh. Thanks for clarifying.
>>>
>>>Network-fencing requires extra setup as explained if you run "pcs
>>>stonith resource describe fence_azure_arm".
>>>
>>>You probably dont need network-fencing, so you can just skip that part.
>>>
>>>On 11/02/19 13:50 +0100, Thomas Berreis wrote:
>>>>> It seems like the issue might be the space in "pcmk_host_list= node01".
>>>>Sorry, this typo was only in my mail because I anonymized in- and output.
>>>>
>>>>The issue only occurs when I add the parameter "network-fencing=on".
>>>>
>>>>WARNING:msrestazure.azure_active_directory:Keyring cache token has
>failed:
>>>>No recommended backend was available. Install the keyrings.alt
>>>>package if you want to use the non-recommended backends. See
>>>>README.rst for
>>>details.
>>>>2019-02-11 12:29:37,079 WARNING: Keyring cache token has failed: No
>>>>recommended backend was available. Install the keyrings.alt package
>>>>if you want to use the non-recommended backends. See README.rst for
>details.
>>>>
>>>>
>>>>Thomas
>>>>
>>>>-----Original Message-----
>>>>From: Users [mailto:users-bounces at clusterlabs.org] On Behalf Of
>>>>Oyvind Albrigtsen
>>>>Sent: Montag, 11. Februar 2019 10:38
>>>>To: Cluster Labs - All topics related to open-source clustering
>>>>welcomed <users at clusterlabs.org>
>>>>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
>>>>
>>>>pcmk_host_list is used by Pacemaker to limit which hosts can be
>>>>fenced by the STONITH device, so you'll have to use the plug
>>>>parameter for
>>>testing.
>>>>Pacemaker uses it internally when it fences a host, so you wouldnt
>>>>use plug= as part of the pcs stonith create command.
>>>>
>>>>It seems like the issue might be the space in "pcmk_host_list= node01".
>>>>
>>>>Oyvind
>>>>
>>>>On 10/02/19 14:43 +0100, Thomas Berreis wrote:
>>>>>Hi Oyvind,
>>>>>
>>>>>Seems, option "pcmk_host_list" is not supported by fence_azure_arm.
>>>>>Instead fence_azure_arm requires "plug" and however this is not
>>>>>supported
>>>>by pcs.
>>>>>What's the correct syntax to add such a stonith device with
>>>>fence_azure_arm?
>>>>>
>>>>>[root ~]# fence_azure_arm
>>>>>login=***
>>>>>network-fencing=on
>>>>>passwd=***
>>>>>pcmk_host_list= node01
>>>>>resourceGroup=***
>>>>>retry_on=0
>>>>>subscriptionId=***
>>>>>tenantId=***
>>>>>WARNING:root:Parse error: Ignoring unknown option 'pcmk_host_list=
>>node01'
>>>>>
>>>>>ERROR:root:Failed: You have to enter plug number or machine
>>>>>identification
>>>>>
>>>>>2019-02-10 12:54:51,862 ERROR: Failed: You have to enter plug number
>>>>>or machine identification
>>>>>
>>>>>------------------------------
>>>>>
>>>>>[root ~]# fence_azure_arm
>>>>>login=***
>>>>>network-fencing=on
>>>>>passwd=***
>>>>>pcmk_host_list= node01
>>>>>resourceGroup=***
>>>>>retry_on=0
>>>>>subscriptionId=***
>>>>>tenantId=***
>>>>>plug=node01
>>>>>WARNING:root:Parse error: Ignoring unknown option 'pcmk_host_list=
>>node01'
>>>>>
>>>>>WARNING:msrestazure.azure_active_directory:Keyring cache token has
>>failed:
>>>>>No recommended backend was available.
>>>>>2019-02-10 12:56:30,377 WARNING: Keyring cache token has failed: No
>>>>>recommended backend was available. Install th
>>>>>WARNING:msrestazure.azure_active_directory:Keyring cache token has
>>failed:
>>>>>No recommended backend was available.
>>>>>2019-02-10 12:56:30,729 WARNING: Keyring cache token has failed: No
>>>>>recommended backend was available. Install th
>>>>>Success: Powered OFF
>>>>>
>>>>>------------------------------
>>>>>
>>>>>[root ~]# fence_azure_arm
>>>>>login=***
>>>>>network-fencing=on
>>>>>passwd=***
>>>>>plug=node01
>>>>>resourceGroup=***
>>>>>retry_on=0
>>>>>subscriptionId=***
>>>>>tenantId=***
>>>>>
>>>>>Success: Powered OFF
>>>>>
>>>>>------------------------------
>>>>>
>>>>>[root ~]# pcs stonith create stonith-net fence_azure_arm [...]
>>>>>plug=node01 network-fencing=on
>>>>>Error: invalid stonith option 'plug'
>>>>>
>>>>>[root ~]# fence_azure_arm -h
>>>>>Usage:
>>>>>        fence_azure_arm [options]
>>>>>Options:
>>>>>[...]
>>>>>   -n, --plug=[id]                Physical plug number on device, UUID
>or
>>>>>identification of machine
>>>>>
>>>>>
>>>>>Thanks!
>>>>>
>>>>>Thomas
>>>>>
>>>>>-----Original Message-----
>>>>>From: Users [mailto:users-bounces at clusterlabs.org] On Behalf Of
>>>>>Oyvind Albrigtsen
>>>>>Sent: Freitag, 8. Februar 2019 14:16
>>>>>To: Cluster Labs - All topics related to open-source clustering
>>>>>welcomed <users at clusterlabs.org>
>>>>>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
>>>>>
>>>>>Hi Thomas,
>>>>>
>>>>>It looks like it's trying to use credentials from keyring, which
>>>>>generates these error messages.
>>>>>
>>>>>Maybe there's a typo in one of the authentitcation parameters in
>>>>>your config. Can you post the output of "pcs stonith show --full"?
>>>>>
>>>>>You can try running fence_azure_arm without parameters on the
>>>>>command line and copy the parameters (one per line) to it, and add e.g.
>>>>>action=status and end with Ctrl+D to see .
>>>>>
>>>>>
>>>>>Oyvind
>>>>>
>>>>>On 08/02/19 14:02 +0100, Thomas Berreis wrote:
>>>>>>Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>>I'm having a two-node cluster in Azure with fencing enabled via
>>>>>>module fence_azure_arm.
>>>>>>
>>>>>>The cluster management is flooding my syslog on both nodes with
>>>>>>these
>>>>>>messages:
>>>>>>
>>>>>>
>>>>>>
>>>>>>** fence_azure_arm: Keyring cache token has failed: No recommended
>>>>>>backend was available. Install the keyrings.alt package if you want
>>>>>>to use the non-recommended backends. See README.rst for details.
>>>>>>
>>>>>>
>>>>>>
>>>>>>** stonith-ng[5220]: warning: fence_azure_arm[14847] stderr: [
>>>>>>2019-02-08
>>>>>>12:52:02,703 WARNING: Keyring cache token has failed: No
>>>>>>recommended backend was available. Install the keyrings.alt package
>>>>>>if you want to use the non-recommended backends. See README.rst for
>>>>>>details. ]
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>Can anyone give me a hint how to fix these issue?
>>>>>>
>>>>>>
>>>>>>
>>>>>>pcs version: 0.9.165
>>>>>>
>>>>>>pacemaker version: 1.1.19-8.el7_6.1
>>>>>>
>>>>>>fence_azure_arm version: 4.2.1
>>>>>>
>>>>>>corosync 2.4.3
>>>>>>
>>>>>>
>>>>>>
>>>>>>Thanks!
>>>>>>
>>>>>>
>>>>>>
>>>>>>Thomas
>>>>>>
>>>>>
>>>>>>_______________________________________________
>>>>>>Users mailing list: Users at clusterlabs.org
>>>>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>
>>>>>>Project Home: http://www.clusterlabs.org Getting started:
>>>>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>Bugs: http://bugs.clusterlabs.org
>>>>>
>>>>>_______________________________________________
>>>>>Users mailing list: Users at clusterlabs.org
>>>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>
>>>>>Project Home: http://www.clusterlabs.org Getting started:
>>>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>Bugs: http://bugs.clusterlabs.org
>>>>>
>>>>>_______________________________________________
>>>>>Users mailing list: Users at clusterlabs.org
>>>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>
>>>>>Project Home: http://www.clusterlabs.org Getting started:
>>>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>Bugs: http://bugs.clusterlabs.org
>>>>_______________________________________________
>>>>Users mailing list: Users at clusterlabs.org
>>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>>
>>>>Project Home: http://www.clusterlabs.org Getting started:
>>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>Bugs: http://bugs.clusterlabs.org
>>>>
>>>>_______________________________________________
>>>>Users mailing list: Users at clusterlabs.org
>>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>>
>>>>Project Home: http://www.clusterlabs.org Getting started:
>>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>Bugs: http://bugs.clusterlabs.org
>>>_______________________________________________
>>>Users mailing list: Users at clusterlabs.org
>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>>Project Home: http://www.clusterlabs.org Getting started:
>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>Bugs: http://bugs.clusterlabs.org
>>>
>>>_______________________________________________
>>>Users mailing list: Users at clusterlabs.org
>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>>Project Home: http://www.clusterlabs.org Getting started:
>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>Bugs: http://bugs.clusterlabs.org
>>_______________________________________________
>>Users mailing list: Users at clusterlabs.org
>>https://lists.clusterlabs.org/mailman/listinfo/users
>>
>>Project Home: http://www.clusterlabs.org Getting started:
>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>Bugs: http://bugs.clusterlabs.org
>>
>>_______________________________________________
>>Users mailing list: Users at clusterlabs.org
>>https://lists.clusterlabs.org/mailman/listinfo/users
>>
>>Project Home: http://www.clusterlabs.org Getting started:
>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>Bugs: http://bugs.clusterlabs.org
>_______________________________________________
>Users mailing list: Users at clusterlabs.org
>https://lists.clusterlabs.org/mailman/listinfo/users
>
>Project Home: http://www.clusterlabs.org Getting started:
>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>Bugs: http://bugs.clusterlabs.org
>
>_______________________________________________
>Users mailing list: Users at clusterlabs.org
>https://lists.clusterlabs.org/mailman/listinfo/users
>
>Project Home: http://www.clusterlabs.org
>Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>Bugs: http://bugs.clusterlabs.org