[ClusterLabs] corosync - CS_ERR_BAD_HANDLE when multiple nodes are starting up

[Thomas Lamprecht]

Hello,

we are using corosync version needle (2.3.5) for our cluster filesystem 
(pmxcfs).
The situation is the following. First we start up the pmxcfs, which is 
an fuse fs. And if there is an cluster configuration, we start also 
corosync.
This allows the filesystem to exist on one node 'cluster's or forcing it 
in an local mode. We use CPG to send our messages to all members,
the filesystem is in the RAM and all fs operations are sent 'over the wire'.

The problem is now the following:
When we're restarting all (in my test case 3) nodes at the same time, I 
get in 1 from 10 cases only CS_ERR_BAD_HANDLE back when calling 
cpg_mcast_joined to send out the data, but only one node. 
corosyn-quorumtool shows that we have quorum, and the logs are also 
showing a healthy connect to the corosync cluster. The failing handle is 
initialized once at the initialization of our filesystem. Should it be 
reinitialized on every reconnect?
Restarting the filesystem solves this problem, the strange thing is that 
isn't clearly reproduce-able and often works just fine.

Are there some known problems or steps we should look for?