[ClusterLabs] pcs cluster auth succeeds but pcs cluster setup failed with"error checking node availability: Unable to authenticate ..."
Strahil Nikolov
hunter86_bg at yahoo.com
Mon Mar 9 01:28:22 EDT 2020
On March 9, 2020 6:35:32 AM GMT+02:00, Ken <ninja_ken at 163.com> wrote:
>Hi Pacemaker users,
>
>
>I'm new to pacemaker, and recently I started from Pacemaker 2.0 doc --
>Clusters from Scratch (en-US), but got bogged down in cluster setting
>up step "pcs cluster auth ... and pcs cluster setup ..."
>While running pcs cluster auth, it says Authorized but the console
>debug output has errors.
>
>
>root at mirror1:/home/m1# pcs cluster auth mirror1 mirror2 --debug
>Running: /usr/bin/ruby -I/usr/share/pcsd/ /usr/share/pcsd/pcsd-cli.rb
>read_tokens
>--Debug Input Start--
>{}
>--Debug Input End--
>Return Value: 0
>--Debug Output Start--
>{
> "status": "ok",
> "data": {
> },
> "log": [
>"I, [2020-03-08T21:24:11.121997 #14049] INFO -- : PCSD Debugging
>enabled\n",
>"D, [2020-03-08T21:24:11.122027 #14049] DEBUG -- : Did not detect RHEL
>6\n",
>"I, [2020-03-08T21:24:11.122059 #14049] INFO -- : Running:
>/usr/sbin/corosync-cmapctl totem.cluster_name\n",
>"I, [2020-03-08T21:24:11.122073 #14049] INFO -- : CIB USER: hacluster,
>groups: \n",
>"D, [2020-03-08T21:24:11.127385 #14049] DEBUG -- :
>[\"totem.cluster_name (str) = debian\\n\"]\n",
> "D, [2020-03-08T21:24:11.127432 #14049] DEBUG -- : []\n",
>"D, [2020-03-08T21:24:11.127451 #14049] DEBUG -- : Duration:
>0.005305199s\n",
> "I, [2020-03-08T21:24:11.127513 #14049] INFO -- : Return Value: 0\n",
>"W, [2020-03-08T21:24:11.127655 #14049] WARN -- : Cannot read config
>'tokens' from '/var/lib/pcsd/tokens': No such file or directory @
>rb_sysopen - /var/lib/pcsd/tokens\n",
>"E, [2020-03-08T21:24:11.127694 #14049] ERROR -- : Unable to parse
>tokens file: A JSON text must at least contain two octets!\n"
> ]
>}
>--Debug Output End--
>
>Sending HTTP Request to: https://mirror1:2224/remote/check_auth
>Data: None
>Response Code: 401
>Username: hacluster
>Password:
>Running: /usr/bin/ruby -I/usr/share/pcsd/ /usr/share/pcsd/pcsd-cli.rb
>auth
>--Debug Input Start--
>{"username": "hacluster", "local": false, "nodes": ["mirror1",
>"mirror2"], "password": "123456", "force": false}
>--Debug Input End--
>Return Value: 0
>--Debug Output Start--
>{
> "status": "ok",
> "data": {
> "auth_responses": {
> "mirror1": {
> "status": "ok",
> "token": "2873a435-339b-402f-854f-542379b480ec"
> },
> "mirror2": {
> "status": "ok",
> "token": "16f6346d-742c-454d-ac11-55137189baac"
> }
> },
> "sync_successful": true,
> "sync_nodes_err": [
>
> ],
> "sync_responses": {
> }
> },
> "log": [
>"I, [2020-03-08T21:24:16.598892 #14060] INFO -- : PCSD Debugging
>enabled\n",
>"D, [2020-03-08T21:24:16.598922 #14060] DEBUG -- : Did not detect RHEL
>6\n",
>"I, [2020-03-08T21:24:16.598956 #14060] INFO -- : Running:
>/usr/sbin/corosync-cmapctl totem.cluster_name\n",
>"I, [2020-03-08T21:24:16.598969 #14060] INFO -- : CIB USER: hacluster,
>groups: \n",
>"D, [2020-03-08T21:24:16.604361 #14060] DEBUG -- :
>[\"totem.cluster_name (str) = debian\\n\"]\n",
> "D, [2020-03-08T21:24:16.604405 #14060] DEBUG -- : []\n",
>"D, [2020-03-08T21:24:16.604423 #14060] DEBUG -- : Duration:
>0.005386404s\n",
> "I, [2020-03-08T21:24:16.604459 #14060] INFO -- : Return Value: 0\n",
>"W, [2020-03-08T21:24:16.605121 #14060] WARN -- : Cannot read config
>'tokens' from '/var/lib/pcsd/tokens': No such file or directory @
>rb_sysopen - /var/lib/pcsd/tokens\n",
>"E, [2020-03-08T21:24:16.605196 #14060] ERROR -- : Unable to parse
>tokens file: A JSON text must at least contain two octets!\n",
>"I, [2020-03-08T21:24:16.605227 #14060] INFO -- : SRWT Node: mirror1
>Request: check_auth\n",
>"E, [2020-03-08T21:24:16.605238 #14060] ERROR -- : Unable to connect to
>node mirror1, no token available\n",
>"W, [2020-03-08T21:24:16.605361 #14060] WARN -- : Cannot read config
>'tokens' from '/var/lib/pcsd/tokens': No such file or directory @
>rb_sysopen - /var/lib/pcsd/tokens\n",
>"E, [2020-03-08T21:24:16.605406 #14060] ERROR -- : Unable to parse
>tokens file: A JSON text must at least contain two octets!\n",
>"I, [2020-03-08T21:24:16.605419 #14060] INFO -- : SRWT Node: mirror2
>Request: check_auth\n",
>"E, [2020-03-08T21:24:16.605430 #14060] ERROR -- : Unable to connect to
>node mirror2, no token available\n",
>"I, [2020-03-08T21:24:16.690075 #14060] INFO -- : Running:
>/usr/sbin/pcs status nodes corosync\n",
>"I, [2020-03-08T21:24:16.690132 #14060] INFO -- : CIB USER: hacluster,
>groups: \n",
>"D, [2020-03-08T21:24:16.816941 #14060] DEBUG -- : [\"Corosync
>Nodes:\\n\", \" Online:\\n\", \" Offline:\\n\"]\n",
> "D, [2020-03-08T21:24:16.816995 #14060] DEBUG -- : []\n",
>"D, [2020-03-08T21:24:16.817015 #14060] DEBUG -- : Duration:
>0.126806344s\n",
> "I, [2020-03-08T21:24:16.817064 #14060] INFO -- : Return Value: 0\n",
>"W, [2020-03-08T21:24:16.817151 #14060] WARN -- : Cannot read config
>'tokens' from '/var/lib/pcsd/tokens': No such file or directory @
>rb_sysopen - /var/lib/pcsd/tokens\n",
>"E, [2020-03-08T21:24:16.817191 #14060] ERROR -- : Unable to parse
>tokens file: A JSON text must at least contain two octets!\n",
>"I, [2020-03-08T21:24:16.817381 #14060] INFO -- : Sending config
>'tokens' version 1 c23384dc06c9d778394a42869c9fdbf5683b2832 to nodes:
>\n"
> ]
>}
>--Debug Output End--
>
>mirror1: Authorized
>mirror2: Authorized
>root at mirror1:/home/m1#
>
>
>
>Then I tried to do next step, pcs cluster setup, and it goes
>
>
>root at mirror1:/home/m1# pcs cluster setup --name aCluster mirror1
>mirror2 --debug
>Running: /usr/sbin/corosync -v
>Return Value: 0
>--Debug Output Start--
>Corosync Cluster Engine, version '2.3.5'
>Copyright (c) 2006-2009 Red Hat, Inc.
>--Debug Output End--
>
>Running: /usr/bin/ruby -I/usr/share/pcsd/ /usr/share/pcsd/pcsd-cli.rb
>read_tokens
>--Debug Input Start--
>{}
>--Debug Input End--
>Return Value: 0
>--Debug Output Start--
>{
> "status": "ok",
> "data": {
> },
> "log": [
>"I, [2020-03-08T21:27:11.229802 #14136] INFO -- : PCSD Debugging
>enabled\n",
>"D, [2020-03-08T21:27:11.229864 #14136] DEBUG -- : Did not detect RHEL
>6\n",
>"I, [2020-03-08T21:27:11.229901 #14136] INFO -- : Running:
>/usr/sbin/corosync-cmapctl totem.cluster_name\n",
>"I, [2020-03-08T21:27:11.229914 #14136] INFO -- : CIB USER: hacluster,
>groups: \n",
>"D, [2020-03-08T21:27:11.234800 #14136] DEBUG -- :
>[\"totem.cluster_name (str) = debian\\n\"]\n",
> "D, [2020-03-08T21:27:11.234847 #14136] DEBUG -- : []\n",
>"D, [2020-03-08T21:27:11.234865 #14136] DEBUG -- : Duration:
>0.004878528s\n",
> "I, [2020-03-08T21:27:11.234900 #14136] INFO -- : Return Value: 0\n",
>"W, [2020-03-08T21:27:11.235015 #14136] WARN -- : Cannot read config
>'tokens' from '/var/lib/pcsd/tokens': No such file or directory @
>rb_sysopen - /var/lib/pcsd/tokens\n",
>"E, [2020-03-08T21:27:11.235051 #14136] ERROR -- : Unable to parse
>tokens file: A JSON text must at least contain two octets!\n"
> ]
>}
>--Debug Output End--
>
>Sending HTTP Request to: https://mirror1:2224/remote/node_available
>Data: None
>Response Code: 401
>Error: mirror1: error checking node availability: Unable to
>authenticate to mirror1 - (HTTP error: 401), try running 'pcs cluster
>auth'
>Running: /usr/bin/ruby -I/usr/share/pcsd/ /usr/share/pcsd/pcsd-cli.rb
>read_tokens
>--Debug Input Start--
>{}
>--Debug Input End--
>Return Value: 0
>--Debug Output Start--
>{
> "status": "ok",
> "data": {
> },
> "log": [
>"I, [2020-03-08T21:27:11.518706 #14147] INFO -- : PCSD Debugging
>enabled\n",
>"D, [2020-03-08T21:27:11.518736 #14147] DEBUG -- : Did not detect RHEL
>6\n",
>"I, [2020-03-08T21:27:11.518755 #14147] INFO -- : Running:
>/usr/sbin/corosync-cmapctl totem.cluster_name\n",
>"I, [2020-03-08T21:27:11.518768 #14147] INFO -- : CIB USER: hacluster,
>groups: \n",
>"D, [2020-03-08T21:27:11.523738 #14147] DEBUG -- :
>[\"totem.cluster_name (str) = debian\\n\"]\n",
> "D, [2020-03-08T21:27:11.523820 #14147] DEBUG -- : []\n",
>"D, [2020-03-08T21:27:11.523841 #14147] DEBUG -- : Duration:
>0.004964793s\n",
> "I, [2020-03-08T21:27:11.523876 #14147] INFO -- : Return Value: 0\n",
>"W, [2020-03-08T21:27:11.523986 #14147] WARN -- : Cannot read config
>'tokens' from '/var/lib/pcsd/tokens': No such file or directory @
>rb_sysopen - /var/lib/pcsd/tokens\n",
>"E, [2020-03-08T21:27:11.524024 #14147] ERROR -- : Unable to parse
>tokens file: A JSON text must at least contain two octets!\n"
> ]
>}
>--Debug Output End--
>
>Sending HTTP Request to: https://mirror2:2224/remote/node_available
>Data: None
>Response Code: 401
>Error: mirror2: error checking node availability: Unable to
>authenticate to mirror2 - (HTTP error: 401), try running 'pcs cluster
>auth'
>Error: nodes availability check failed, use --force to override.
>WARNING: This will destroy existing cluster on the nodes.
>root at mirror1:/home/m1#
>
>
>
>I believed that I have been sticking to the instructions in the
>documentation, but right now I'm out of clues for what's wrong. Please
>advise if you know what might be the problem.
>
>The two Linux OS's I'm using are Ubuntu 16.04.1, and they are virtual
>machines on VMWare Workstation Pro.
I have noticed that the errors from pcs are not so clear.
You have to check:
1. DNS resolvability
I would rexommend you to use FQDN for the node names.
2. Time is in sync (either chrony/ntp is running and in sync)
3. Firewall ports are open
For RHEL, firewalld is the default and it has a 'high-availability' service
4. Check that pcsd is enabled and running
5. Doublecheck your command , if possible copy/paste from 'pcs cluster --help'
Best Regards,
Strahil Nikolov
More information about the Users
mailing list