[ClusterLabs] pcs cluster auth succeeds but pcs cluster setup failed with"error checking node availability: Unable to authenticate ..."
Ken
ninja_ken at 163.com
Mon Mar 9 21:02:53 EDT 2020
Hi Strahil,
Thanks for the quick and kind response. I tried your suggestions today, including checking the time synchronization, firewall and pcsd daemon status, and everything looks fine. There's another email from Tomas Jelinek yesterday which lead me to some other places where I learned that this is actually a confirmed bug, and code fix has been commited by contributors in pcs's github repository. You can check his email for more details.
>You have to check:
>1. DNS resolvability
>I would rexommend you to use FQDN for the node names.
>2. Time is in sync (either chrony/ntp is running and in sync)
>3. Firewall ports are open
>For RHEL, firewalld is the default and it has a 'high-availability' service
>4. Check that pcsd is enabled and running
>5. Doublecheck your command , if possible copy/paste from 'pcs cluster --help'
At 2020-03-09 13:28:22, "Strahil Nikolov" <hunter86_bg at yahoo.com> wrote:
>On March 9, 2020 6:35:32 AM GMT+02:00, Ken <ninja_ken at 163.com> wrote:
>>Hi Pacemaker users,
>>
>>
>>I'm new to pacemaker, and recently I started from Pacemaker 2.0 doc --
>>Clusters from Scratch (en-US), but got bogged down in cluster setting
>>up step "pcs cluster auth ... and pcs cluster setup ..."
>>While running pcs cluster auth, it says Authorized but the console
>>debug output has errors.
>>
>>
>>root at mirror1:/home/m1# pcs cluster auth mirror1 mirror2 --debug
>>Running: /usr/bin/ruby -I/usr/share/pcsd/ /usr/share/pcsd/pcsd-cli.rb
>>read_tokens
>>--Debug Input Start--
>>{}
>>--Debug Input End--
>>Return Value: 0
>>--Debug Output Start--
>>{
>> "status": "ok",
>> "data": {
>> },
>> "log": [
>>"I, [2020-03-08T21:24:11.121997 #14049] INFO -- : PCSD Debugging
>>enabled\n",
>>"D, [2020-03-08T21:24:11.122027 #14049] DEBUG -- : Did not detect RHEL
>>6\n",
>>"I, [2020-03-08T21:24:11.122059 #14049] INFO -- : Running:
>>/usr/sbin/corosync-cmapctl totem.cluster_name\n",
>>"I, [2020-03-08T21:24:11.122073 #14049] INFO -- : CIB USER: hacluster,
>>groups: \n",
>>"D, [2020-03-08T21:24:11.127385 #14049] DEBUG -- :
>>[\"totem.cluster_name (str) = debian\\n\"]\n",
>> "D, [2020-03-08T21:24:11.127432 #14049] DEBUG -- : []\n",
>>"D, [2020-03-08T21:24:11.127451 #14049] DEBUG -- : Duration:
>>0.005305199s\n",
>> "I, [2020-03-08T21:24:11.127513 #14049] INFO -- : Return Value: 0\n",
>>"W, [2020-03-08T21:24:11.127655 #14049] WARN -- : Cannot read config
>>'tokens' from '/var/lib/pcsd/tokens': No such file or directory @
>>rb_sysopen - /var/lib/pcsd/tokens\n",
>>"E, [2020-03-08T21:24:11.127694 #14049] ERROR -- : Unable to parse
>>tokens file: A JSON text must at least contain two octets!\n"
>> ]
>>}
>>--Debug Output End--
>>
>>Sending HTTP Request to: https://mirror1:2224/remote/check_auth
>>Data: None
>>Response Code: 401
>>Username: hacluster
>>Password:
>>Running: /usr/bin/ruby -I/usr/share/pcsd/ /usr/share/pcsd/pcsd-cli.rb
>>auth
>>--Debug Input Start--
>>{"username": "hacluster", "local": false, "nodes": ["mirror1",
>>"mirror2"], "password": "123456", "force": false}
>>--Debug Input End--
>>Return Value: 0
>>--Debug Output Start--
>>{
>> "status": "ok",
>> "data": {
>> "auth_responses": {
>> "mirror1": {
>> "status": "ok",
>> "token": "2873a435-339b-402f-854f-542379b480ec"
>> },
>> "mirror2": {
>> "status": "ok",
>> "token": "16f6346d-742c-454d-ac11-55137189baac"
>> }
>> },
>> "sync_successful": true,
>> "sync_nodes_err": [
>>
>> ],
>> "sync_responses": {
>> }
>> },
>> "log": [
>>"I, [2020-03-08T21:24:16.598892 #14060] INFO -- : PCSD Debugging
>>enabled\n",
>>"D, [2020-03-08T21:24:16.598922 #14060] DEBUG -- : Did not detect RHEL
>>6\n",
>>"I, [2020-03-08T21:24:16.598956 #14060] INFO -- : Running:
>>/usr/sbin/corosync-cmapctl totem.cluster_name\n",
>>"I, [2020-03-08T21:24:16.598969 #14060] INFO -- : CIB USER: hacluster,
>>groups: \n",
>>"D, [2020-03-08T21:24:16.604361 #14060] DEBUG -- :
>>[\"totem.cluster_name (str) = debian\\n\"]\n",
>> "D, [2020-03-08T21:24:16.604405 #14060] DEBUG -- : []\n",
>>"D, [2020-03-08T21:24:16.604423 #14060] DEBUG -- : Duration:
>>0.005386404s\n",
>> "I, [2020-03-08T21:24:16.604459 #14060] INFO -- : Return Value: 0\n",
>>"W, [2020-03-08T21:24:16.605121 #14060] WARN -- : Cannot read config
>>'tokens' from '/var/lib/pcsd/tokens': No such file or directory @
>>rb_sysopen - /var/lib/pcsd/tokens\n",
>>"E, [2020-03-08T21:24:16.605196 #14060] ERROR -- : Unable to parse
>>tokens file: A JSON text must at least contain two octets!\n",
>>"I, [2020-03-08T21:24:16.605227 #14060] INFO -- : SRWT Node: mirror1
>>Request: check_auth\n",
>>"E, [2020-03-08T21:24:16.605238 #14060] ERROR -- : Unable to connect to
>>node mirror1, no token available\n",
>>"W, [2020-03-08T21:24:16.605361 #14060] WARN -- : Cannot read config
>>'tokens' from '/var/lib/pcsd/tokens': No such file or directory @
>>rb_sysopen - /var/lib/pcsd/tokens\n",
>>"E, [2020-03-08T21:24:16.605406 #14060] ERROR -- : Unable to parse
>>tokens file: A JSON text must at least contain two octets!\n",
>>"I, [2020-03-08T21:24:16.605419 #14060] INFO -- : SRWT Node: mirror2
>>Request: check_auth\n",
>>"E, [2020-03-08T21:24:16.605430 #14060] ERROR -- : Unable to connect to
>>node mirror2, no token available\n",
>>"I, [2020-03-08T21:24:16.690075 #14060] INFO -- : Running:
>>/usr/sbin/pcs status nodes corosync\n",
>>"I, [2020-03-08T21:24:16.690132 #14060] INFO -- : CIB USER: hacluster,
>>groups: \n",
>>"D, [2020-03-08T21:24:16.816941 #14060] DEBUG -- : [\"Corosync
>>Nodes:\\n\", \" Online:\\n\", \" Offline:\\n\"]\n",
>> "D, [2020-03-08T21:24:16.816995 #14060] DEBUG -- : []\n",
>>"D, [2020-03-08T21:24:16.817015 #14060] DEBUG -- : Duration:
>>0.126806344s\n",
>> "I, [2020-03-08T21:24:16.817064 #14060] INFO -- : Return Value: 0\n",
>>"W, [2020-03-08T21:24:16.817151 #14060] WARN -- : Cannot read config
>>'tokens' from '/var/lib/pcsd/tokens': No such file or directory @
>>rb_sysopen - /var/lib/pcsd/tokens\n",
>>"E, [2020-03-08T21:24:16.817191 #14060] ERROR -- : Unable to parse
>>tokens file: A JSON text must at least contain two octets!\n",
>>"I, [2020-03-08T21:24:16.817381 #14060] INFO -- : Sending config
>>'tokens' version 1 c23384dc06c9d778394a42869c9fdbf5683b2832 to nodes:
>>\n"
>> ]
>>}
>>--Debug Output End--
>>
>>mirror1: Authorized
>>mirror2: Authorized
>>root at mirror1:/home/m1#
>>
>>
>>
>>Then I tried to do next step, pcs cluster setup, and it goes
>>
>>
>>root at mirror1:/home/m1# pcs cluster setup --name aCluster mirror1
>>mirror2 --debug
>>Running: /usr/sbin/corosync -v
>>Return Value: 0
>>--Debug Output Start--
>>Corosync Cluster Engine, version '2.3.5'
>>Copyright (c) 2006-2009 Red Hat, Inc.
>>--Debug Output End--
>>
>>Running: /usr/bin/ruby -I/usr/share/pcsd/ /usr/share/pcsd/pcsd-cli.rb
>>read_tokens
>>--Debug Input Start--
>>{}
>>--Debug Input End--
>>Return Value: 0
>>--Debug Output Start--
>>{
>> "status": "ok",
>> "data": {
>> },
>> "log": [
>>"I, [2020-03-08T21:27:11.229802 #14136] INFO -- : PCSD Debugging
>>enabled\n",
>>"D, [2020-03-08T21:27:11.229864 #14136] DEBUG -- : Did not detect RHEL
>>6\n",
>>"I, [2020-03-08T21:27:11.229901 #14136] INFO -- : Running:
>>/usr/sbin/corosync-cmapctl totem.cluster_name\n",
>>"I, [2020-03-08T21:27:11.229914 #14136] INFO -- : CIB USER: hacluster,
>>groups: \n",
>>"D, [2020-03-08T21:27:11.234800 #14136] DEBUG -- :
>>[\"totem.cluster_name (str) = debian\\n\"]\n",
>> "D, [2020-03-08T21:27:11.234847 #14136] DEBUG -- : []\n",
>>"D, [2020-03-08T21:27:11.234865 #14136] DEBUG -- : Duration:
>>0.004878528s\n",
>> "I, [2020-03-08T21:27:11.234900 #14136] INFO -- : Return Value: 0\n",
>>"W, [2020-03-08T21:27:11.235015 #14136] WARN -- : Cannot read config
>>'tokens' from '/var/lib/pcsd/tokens': No such file or directory @
>>rb_sysopen - /var/lib/pcsd/tokens\n",
>>"E, [2020-03-08T21:27:11.235051 #14136] ERROR -- : Unable to parse
>>tokens file: A JSON text must at least contain two octets!\n"
>> ]
>>}
>>--Debug Output End--
>>
>>Sending HTTP Request to: https://mirror1:2224/remote/node_available
>>Data: None
>>Response Code: 401
>>Error: mirror1: error checking node availability: Unable to
>>authenticate to mirror1 - (HTTP error: 401), try running 'pcs cluster
>>auth'
>>Running: /usr/bin/ruby -I/usr/share/pcsd/ /usr/share/pcsd/pcsd-cli.rb
>>read_tokens
>>--Debug Input Start--
>>{}
>>--Debug Input End--
>>Return Value: 0
>>--Debug Output Start--
>>{
>> "status": "ok",
>> "data": {
>> },
>> "log": [
>>"I, [2020-03-08T21:27:11.518706 #14147] INFO -- : PCSD Debugging
>>enabled\n",
>>"D, [2020-03-08T21:27:11.518736 #14147] DEBUG -- : Did not detect RHEL
>>6\n",
>>"I, [2020-03-08T21:27:11.518755 #14147] INFO -- : Running:
>>/usr/sbin/corosync-cmapctl totem.cluster_name\n",
>>"I, [2020-03-08T21:27:11.518768 #14147] INFO -- : CIB USER: hacluster,
>>groups: \n",
>>"D, [2020-03-08T21:27:11.523738 #14147] DEBUG -- :
>>[\"totem.cluster_name (str) = debian\\n\"]\n",
>> "D, [2020-03-08T21:27:11.523820 #14147] DEBUG -- : []\n",
>>"D, [2020-03-08T21:27:11.523841 #14147] DEBUG -- : Duration:
>>0.004964793s\n",
>> "I, [2020-03-08T21:27:11.523876 #14147] INFO -- : Return Value: 0\n",
>>"W, [2020-03-08T21:27:11.523986 #14147] WARN -- : Cannot read config
>>'tokens' from '/var/lib/pcsd/tokens': No such file or directory @
>>rb_sysopen - /var/lib/pcsd/tokens\n",
>>"E, [2020-03-08T21:27:11.524024 #14147] ERROR -- : Unable to parse
>>tokens file: A JSON text must at least contain two octets!\n"
>> ]
>>}
>>--Debug Output End--
>>
>>Sending HTTP Request to: https://mirror2:2224/remote/node_available
>>Data: None
>>Response Code: 401
>>Error: mirror2: error checking node availability: Unable to
>>authenticate to mirror2 - (HTTP error: 401), try running 'pcs cluster
>>auth'
>>Error: nodes availability check failed, use --force to override.
>>WARNING: This will destroy existing cluster on the nodes.
>>root at mirror1:/home/m1#
>>
>>
>>
>>I believed that I have been sticking to the instructions in the
>>documentation, but right now I'm out of clues for what's wrong. Please
>>advise if you know what might be the problem.
>>
>>The two Linux OS's I'm using are Ubuntu 16.04.1, and they are virtual
>>machines on VMWare Workstation Pro.
>
>I have noticed that the errors from pcs are not so clear.
>
>You have to check:
>1. DNS resolvability
>I would rexommend you to use FQDN for the node names.
>2. Time is in sync (either chrony/ntp is running and in sync)
>3. Firewall ports are open
>For RHEL, firewalld is the default and it has a 'high-availability' service
>4. Check that pcsd is enabled and running
>5. Doublecheck your command , if possible copy/paste from 'pcs cluster --help'
>
>
>Best Regards,
>Strahil Nikolov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200310/7dd7f67a/attachment-0001.htm>
More information about the Users
mailing list