[ClusterLabs] QDevice not found after reboot but appears after cluster restart

[john tillman]

Hi again,

I was hoping someone here could help me with an investigation.  I have a
two cluster setup with a qdevice. 'pcs quorum status' from a cluster node
shows the qdevice casting a vote.  On the qdevice node
'corosync-qnetd-tool -s' says I have 2 connected clients and 1 cluster. 
The vote count looks correct when I shutdown either one of the cluster
nodes or the qdevice.  So the voting seems to be working at this point.

>From this state, if I reboot both my cluster nodes at the same time but
leave the qdevice node running, the cluster will not see the qdevice when
the nodes come back up: 'pcs quorum status' show 3 votes expected but only
2 votes cast (from the cluster nodes).

If I now reboot my qdevice, it is still not seen by the cluster and
corosync-qnetd-tool on the qdevice shows no connections.

Next, if I restart the cluster from the commandline with 'pcs cluster
stop/start --all' the qdevice is found and casts its vote and it looks
like it is working again.

The only anomaly I have found in the logs so far is that when I restart
the cluster, the corosync.log has this lines
[VOTEQ] waiting for quorum device.  Qdevice poll (but maximum for 30000 ms)
This line doesn't appear in the log when I reboot the cluster nodes.

So when the cluster restarts it is doing this "VOTEQ" step but does not
when the node reboots.

I am not sure where to look next.  Any advice for something to look for in
the log files or perhaps there's a configuration value that I overlooked?

-John