[ClusterLabs] fence_azure_arm flooding syslog

Mon Feb 11 10:46:42 EST 2019

Hi Oyvind,

Debug mode doesn't help me. The error occurs after successfully getting
Bearer token. No other issues are shown:

DEBUG:requests_oauthlib.oauth2_session:Invoking 0 token response hooks.
2019-02-11 15:27:12,687 DEBUG: Invoking 0 token response hooks.
DEBUG:requests_oauthlib.oauth2_session:Obtained token {u'resource':
u'https://management.core.windows.net/', u'access_token': ***',
u'ext_expires_in': u'3599', u'expires_in': u'3599', u'expires_at':
1549902431.687557, u'token_type': u'Bearer', u'not_before': u'1549898532',
u'expires_on': u'1549902432'}.
2019-02-11 15:27:12,687 DEBUG: Obtained token {u'resource':
u'https://management.core.windows.net/', u'access_token': u'***',
u'ext_expires_in': u'3599', u'expires_in': u'3599', u'expires_at':
1549902431.687557, u'token_type': u'Bearer', u'not_before': u'1549898532',
u'expires_on': u'1549902432'}.
WARNING:msrestazure.azure_active_directory:Keyring cache token has failed:
No recommended backend was available. Install the keyrings.alt package if
you want to use the non-recommended backends. See README.rst for details.
2019-02-11 15:27:12,687 WARNING: Keyring cache token has failed: No
recommended backend was available. Install the keyrings.alt package if you
want to use the non-recommended backends. See README.rst for details.
DEBUG:requests_oauthlib.oauth2_session:Encoding `client_id` "***" with
`client_secret` as Basic auth credentials.
2019-02-11 15:27:12,727 DEBUG: Encoding `client_id` "***" with
`client_secret` as Basic auth credentials.
DEBUG:requests_oauthlib.oauth2_session:Requesting url
https://login.microsoftonline.com/***/oauth2/token using method POST.
2019-02-11 15:27:12,727 DEBUG: Requesting url
https://login.microsoftonline.com/***/oauth2/token using method POST.

2019-02-11 15:27:13,072 DEBUG: Obtained token {u'resource':
u'https://management.core.windows.net/', u'access_token': u'e***w',
u'ext_expires_in': u'3600', u'expires_in': u'3600', u'expires_at':
1549902433.072768, u'token_type': u'Bearer', u'not_before': u'1549898533',
u'expires_on': u'1549902433'}.
WARNING:msrestazure.azure_active_directory:Keyring cache token has failed:
No recommended backend was available. Install the keyrings.alt package if
you want to use the non-recommended backends. See README.rst for details.
2019-02-11 15:27:13,073 WARNING: Keyring cache token has failed: No
recommended backend was available. Install the keyrings.alt package if you
want to use the non-recommended backends. See README.rst for details.
INFO:root:getting power status for VM node01

Thomas

-----Original Message-----
From: Users [mailto:users-bounces at clusterlabs.org] On Behalf Of Oyvind
Albrigtsen
Sent: Montag, 11. Februar 2019 14:58
To: Cluster Labs - All topics related to open-source clustering welcomed
<users at clusterlabs.org>
Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog

You can try adding verbose=1 and see if that gives you some more details on
where it's failing.

On 11/02/19 14:47 +0100, Thomas Berreis wrote:
>We need this feature because shutdown / reboot takes too much time ( > 
>5
>min) and network fencing fences the virtual machine much faster ( < 5 sec).
>We finished all the required steps and network fencing works as 
>expected but I'm still confused about these errors in the log and the 
>failure counts showed by pcs ...
>
>fence_azure_arm: Keyring cache token has failed: No recommended backend 
>was available. Install the keyrings.alt package if you want to use the 
>non-recommended backends. See README.rst for details.
>
>stonith-ng[7789]: warning: fence_azure_arm[7896] stderr: [ 2019-02-11
>13:41:19,178 WARNING: Keyring cache token has failed: No recommended 
>backend was available. Install the keyrings.alt package if you want to 
>use the non-recommended backends. See README.rst for details. ]
>
>stonith-net_monitor_60000 on node02 'unknown error' (1): call=369, 
>status=Timed Out, exitreason='', last-rc-change='Mon Feb 11 10:38:02 
>2019', queued=0ms, exec=20007ms
>
>Thomas
>
>-----Original Message-----
>From: Users [mailto:users-bounces at clusterlabs.org] On Behalf Of Oyvind 
>Albrigtsen
>Sent: Montag, 11. Februar 2019 13:58
>To: Cluster Labs - All topics related to open-source clustering 
>welcomed <users at clusterlabs.org>
>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
>
>Oh. Thanks for clarifying.
>
>Network-fencing requires extra setup as explained if you run "pcs 
>stonith resource describe fence_azure_arm".
>
>You probably dont need network-fencing, so you can just skip that part.
>
>On 11/02/19 13:50 +0100, Thomas Berreis wrote:
>>> It seems like the issue might be the space in "pcmk_host_list= node01".
>>Sorry, this typo was only in my mail because I anonymized in- and output.
>>
>>The issue only occurs when I add the parameter "network-fencing=on".
>>
>>WARNING:msrestazure.azure_active_directory:Keyring cache token has failed:
>>No recommended backend was available. Install the keyrings.alt package 
>>if you want to use the non-recommended backends. See README.rst for
>details.
>>2019-02-11 12:29:37,079 WARNING: Keyring cache token has failed: No 
>>recommended backend was available. Install the keyrings.alt package if 
>>you want to use the non-recommended backends. See README.rst for details.
>>
>>
>>Thomas
>>
>>-----Original Message-----
>>From: Users [mailto:users-bounces at clusterlabs.org] On Behalf Of Oyvind 
>>Albrigtsen
>>Sent: Montag, 11. Februar 2019 10:38
>>To: Cluster Labs - All topics related to open-source clustering 
>>welcomed <users at clusterlabs.org>
>>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
>>
>>pcmk_host_list is used by Pacemaker to limit which hosts can be fenced 
>>by the STONITH device, so you'll have to use the plug parameter for
>testing.
>>Pacemaker uses it internally when it fences a host, so you wouldnt use 
>>plug= as part of the pcs stonith create command.
>>
>>It seems like the issue might be the space in "pcmk_host_list= node01".
>>
>>Oyvind
>>
>>On 10/02/19 14:43 +0100, Thomas Berreis wrote:
>>>Hi Oyvind,
>>>
>>>Seems, option "pcmk_host_list" is not supported by fence_azure_arm.
>>>Instead fence_azure_arm requires "plug" and however this is not 
>>>supported
>>by pcs.
>>>What's the correct syntax to add such a stonith device with
>>fence_azure_arm?
>>>
>>>[root ~]# fence_azure_arm
>>>login=***
>>>network-fencing=on
>>>passwd=***
>>>pcmk_host_list= node01
>>>resourceGroup=***
>>>retry_on=0
>>>subscriptionId=***
>>>tenantId=***
>>>WARNING:root:Parse error: Ignoring unknown option 'pcmk_host_list=
node01'
>>>
>>>ERROR:root:Failed: You have to enter plug number or machine 
>>>identification
>>>
>>>2019-02-10 12:54:51,862 ERROR: Failed: You have to enter plug number 
>>>or machine identification
>>>
>>>------------------------------
>>>
>>>[root ~]# fence_azure_arm
>>>login=***
>>>network-fencing=on
>>>passwd=***
>>>pcmk_host_list= node01
>>>resourceGroup=***
>>>retry_on=0
>>>subscriptionId=***
>>>tenantId=***
>>>plug=node01
>>>WARNING:root:Parse error: Ignoring unknown option 'pcmk_host_list=
node01'
>>>
>>>WARNING:msrestazure.azure_active_directory:Keyring cache token has
failed:
>>>No recommended backend was available.
>>>2019-02-10 12:56:30,377 WARNING: Keyring cache token has failed: No 
>>>recommended backend was available. Install th 
>>>WARNING:msrestazure.azure_active_directory:Keyring cache token has
failed:
>>>No recommended backend was available.
>>>2019-02-10 12:56:30,729 WARNING: Keyring cache token has failed: No 
>>>recommended backend was available. Install th
>>>Success: Powered OFF
>>>
>>>------------------------------
>>>
>>>[root ~]# fence_azure_arm
>>>login=***
>>>network-fencing=on
>>>passwd=***
>>>plug=node01
>>>resourceGroup=***
>>>retry_on=0
>>>subscriptionId=***
>>>tenantId=***
>>>
>>>Success: Powered OFF
>>>
>>>------------------------------
>>>
>>>[root ~]# pcs stonith create stonith-net fence_azure_arm [...]
>>>plug=node01 network-fencing=on
>>>Error: invalid stonith option 'plug'
>>>
>>>[root ~]# fence_azure_arm -h
>>>Usage:
>>>        fence_azure_arm [options]
>>>Options:
>>>[...]
>>>   -n, --plug=[id]                Physical plug number on device, UUID or
>>>identification of machine
>>>
>>>
>>>Thanks!
>>>
>>>Thomas
>>>
>>>-----Original Message-----
>>>From: Users [mailto:users-bounces at clusterlabs.org] On Behalf Of 
>>>Oyvind Albrigtsen
>>>Sent: Freitag, 8. Februar 2019 14:16
>>>To: Cluster Labs - All topics related to open-source clustering 
>>>welcomed <users at clusterlabs.org>
>>>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
>>>
>>>Hi Thomas,
>>>
>>>It looks like it's trying to use credentials from keyring, which 
>>>generates these error messages.
>>>
>>>Maybe there's a typo in one of the authentitcation parameters in your 
>>>config. Can you post the output of "pcs stonith show --full"?
>>>
>>>You can try running fence_azure_arm without parameters on the command 
>>>line and copy the parameters (one per line) to it, and add e.g.
>>>action=status and end with Ctrl+D to see .
>>>
>>>
>>>Oyvind
>>>
>>>On 08/02/19 14:02 +0100, Thomas Berreis wrote:
>>>>Hi,
>>>>
>>>>
>>>>
>>>>I'm having a two-node cluster in Azure with fencing enabled via 
>>>>module fence_azure_arm.
>>>>
>>>>The cluster management is flooding my syslog on both nodes with 
>>>>these
>>>>messages:
>>>>
>>>>
>>>>
>>>>** fence_azure_arm: Keyring cache token has failed: No recommended 
>>>>backend was available. Install the keyrings.alt package if you want 
>>>>to use the non-recommended backends. See README.rst for details.
>>>>
>>>>
>>>>
>>>>** stonith-ng[5220]: warning: fence_azure_arm[14847] stderr: [
>>>>2019-02-08
>>>>12:52:02,703 WARNING: Keyring cache token has failed: No recommended 
>>>>backend was available. Install the keyrings.alt package if you want 
>>>>to use the non-recommended backends. See README.rst for details. ]
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>Can anyone give me a hint how to fix these issue?
>>>>
>>>>
>>>>
>>>>pcs version: 0.9.165
>>>>
>>>>pacemaker version: 1.1.19-8.el7_6.1
>>>>
>>>>fence_azure_arm version: 4.2.1
>>>>
>>>>corosync 2.4.3
>>>>
>>>>
>>>>
>>>>Thanks!
>>>>
>>>>
>>>>
>>>>Thomas
>>>>
>>>
>>>>_______________________________________________
>>>>Users mailing list: Users at clusterlabs.org 
>>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>>
>>>>Project Home: http://www.clusterlabs.org Getting started:
>>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>Bugs: http://bugs.clusterlabs.org
>>>
>>>_______________________________________________
>>>Users mailing list: Users at clusterlabs.org 
>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>>Project Home: http://www.clusterlabs.org Getting started:
>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>Bugs: http://bugs.clusterlabs.org
>>>
>>>_______________________________________________
>>>Users mailing list: Users at clusterlabs.org 
>>>https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>>Project Home: http://www.clusterlabs.org Getting started:
>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>Bugs: http://bugs.clusterlabs.org
>>_______________________________________________
>>Users mailing list: Users at clusterlabs.org 
>>https://lists.clusterlabs.org/mailman/listinfo/users
>>
>>Project Home: http://www.clusterlabs.org Getting started:
>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>Bugs: http://bugs.clusterlabs.org
>>
>>_______________________________________________
>>Users mailing list: Users at clusterlabs.org 
>>https://lists.clusterlabs.org/mailman/listinfo/users
>>
>>Project Home: http://www.clusterlabs.org Getting started:
>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>Bugs: http://bugs.clusterlabs.org
>_______________________________________________
>Users mailing list: Users at clusterlabs.org 
>https://lists.clusterlabs.org/mailman/listinfo/users
>
>Project Home: http://www.clusterlabs.org Getting started:
>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>Bugs: http://bugs.clusterlabs.org
>
>_______________________________________________
>Users mailing list: Users at clusterlabs.org 
>https://lists.clusterlabs.org/mailman/listinfo/users
>
>Project Home: http://www.clusterlabs.org Getting started: 
>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: Users at clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org