[ClusterLabs] PCS cluster auth fails

Fri Feb 7 01:39:58 EST 2020

Hi ,

I am using a two node corosync cluster(node1 and node2). When I do pcs cluster auth during corosync-qdevice configuration to the qdevice node(qnode), I am getting the below error,

$ sudo pcs cluster auth qnode -u hacluster -p **************** --debug
Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb auth
Environment:
  GEM_HOME=/usr/lib/pcsd/vendor/bundle/ruby
  HISTSIZE=1000
  HOME=/root
  HOSTNAME=node1
  LANG=en_US.UTF-8
  LC_ALL=C
  LOGNAME=root
  LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
  MAIL=/var/spool/mail/admin
  PATH=/sbin:/bin:/usr/sbin:/usr/bin
  PCSD_DEBUG=true
  PCSD_NETWORK_TIMEOUT=60
  PS1=[\u@\h-$node_name \W]\$
  SHELL=/sbin/nologin
  SUDO_COMMAND=/sbin/pcs cluster auth qnode -u hacluster -p *********** --debug
  SUDO_GID=5007
  SUDO_UID=5008
  SUDO_USER=admin
  TERM=xterm
  USER=root
  USERNAME=root
--Debug Input Start--
{"username": "hacluster", "local": false, "nodes": {"qnode": null}, "password": "***********", "force": false}
--Debug Input End--

Finished running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb auth
Return value: 0
--Debug Stdout Start--
{
  "status": "ok",
  "data": {
    "auth_responses": {
      "qnode": {
        "status": "ok",
        "token": "66c1020d-1089-4d8b-beab-a8f76e2c8b89"
      }
    },
    "sync_successful": true,
    "sync_nodes_err": [
      "node2"
    ],
    "sync_responses": {
      "node2": {
        "status": "error"
      },
      "node1": {
        "status": "ok",
        "result": {
          "tokens": "accepted"
        }
      }
    }
  },
  "log": [
    "I, [2020-02-07T17:11:34.011890 #30323]  INFO -- : PCSD Debugging enabled\n",
    "D, [2020-02-07T17:11:34.012401 #30323] DEBUG -- : Did not detect RHEL 6\n",
    "D, [2020-02-07T17:11:34.012446 #30323] DEBUG -- : Detected systemd is in use\n",
    "I, [2020-02-07T17:11:34.160540 #30323]  INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
    "I, [2020-02-07T17:11:34.160707 #30323]  INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2020-02-07T17:11:34.176123 #30323] DEBUG -- : [\"totem.cluster_name (str) = HBASE\\n\"]\n",
    "D, [2020-02-07T17:11:34.176307 #30323] DEBUG -- : []\n",
    "D, [2020-02-07T17:11:34.176357 #30323] DEBUG -- : Duration: 0.015370265s\n",
    "I, [2020-02-07T17:11:34.176443 #30323]  INFO -- : Return Value: 0\n",
    "I, [2020-02-07T17:11:34.407886 #30323]  INFO -- : Running: /usr/sbin/pcs status nodes corosync\n",
    "I, [2020-02-07T17:11:34.407989 #30323]  INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2020-02-07T17:11:34.775751 #30323] DEBUG -- : [\"Corosync Nodes:\\n\", \" Online: node2 node1\\n\", \" Offline:\\n\"]\n",
    "D, [2020-02-07T17:11:34.775909 #30323] DEBUG -- : []\n",
    "D, [2020-02-07T17:11:34.775960 #30323] DEBUG -- : Duration: 0.367740571s\n",
    "I, [2020-02-07T17:11:34.776056 #30323]  INFO -- : Return Value: 0\n",
    "I, [2020-02-07T17:11:34.776517 #30323]  INFO -- : Sending config 'tokens' version 5 40712570bb7e5718afce943a151b259f04e7c080 to nodes: node2, node1\n",
    "I, [2020-02-07T17:11:34.777079 #30323]  INFO -- : SRWT Node: node2 Request: set_configs\n",
    "I, [2020-02-07T17:11:34.777695 #30323]  INFO -- : SRWT Node: node1 Request: set_configs\n",
    "I, [2020-02-07T17:11:34.865021 #30323]  INFO -- : Sending config response from node2: {\"status\"=>\"error\"}\n",
    "I, [2020-02-07T17:11:34.865141 #30323]  INFO -- : Sending config response from node1: {\"status\"=>\"ok\", \"result\"=>{\"tokens\"=>\"accepted\"}}\n"
  ]
}

--Debug Stdout End--
--Debug Stderr Start--

--Debug Stderr End--

qnode: Authorized
Error: Unable to synchronize and save tokens on nodes: node2. Are they authorized?

Initially I got synchronize error for node1 and node2. After pcsd restart issue got resolved in node1 and now issue occurs only in node2.  Even in debug logs I am not able to get the exact error message. During the auth , I am getting the below error in pcsd log,

2020-02-07 17:11:34] ERROR LoadError: cannot load such file -- rack/multipart/parser
              /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.6.10/lib/rack/multipart.rb:25:in `parse_multipart'
              /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.6.10/lib/rack/request.rb:375:in `parse_multipart'
              /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.6.10/lib/rack/request.rb:207:in `POST'
              /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.6.10/lib/rack/methodoverride.rb:43:in `method_override_param'
              /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.6.10/lib/rack/methodoverride.rb:27:in `method_override'
              /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.6.10/lib/rack/methodoverride.rb:15:in `call'
              /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.8/lib/sinatra/base.rb:182:in `call'
              /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.8/lib/sinatra/base.rb:2013:in `call'
              /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.8/lib/sinatra/base.rb:1487:in `block in call'
              /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.8/lib/sinatra/base.rb:1787:in `synchronize'
              /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.8/lib/sinatra/base.rb:1487:in `call'
              /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.6.10/lib/rack/handler/webrick.rb:88:in `service'
              /usr/share/ruby/webrick/httpserver.rb:138:in `service'
              /usr/share/ruby/webrick/httpserver.rb:94:in `run'
              /usr/share/ruby/webrick/server.rb:295:in `block in start_thread'

Is there any way to know what the error is here.

With Regards
Somanath Thilak J

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20200207/a3a74057/attachment-0001.html>