[Pacemaker] Serious issue with booth for site failover

[Yves Trudeau]

Hi,
    working on a geo-redundant setup, I uncovered a problem with booth. 
  In order to simplify, I did an experiment with only booth, no 
pacemaker.  The behavior is the same with pacemaker.

Version used
------------

git log
commit 55ab027233407fd44850f0c4905b085205d55f64
Author: Xia Li <xli at suse.com>
Date:   Thu Jan 10 13:48:20 2013 +0800

Config file
-----------

transport="UDP"
port="6666"
arbitrator="10.3.3.1"
site="10.3.1.10"
site="10.3.2.10"
ticket="ticketMaster;120"

*same on all nodes.

Invocations
-----------

root at 10.3.3.1:~# /usr/sbin/boothd arbitrator -D

root at 10.3.1.10:~# /usr/sbin/boothd site -D

root at 10.3.2.10:~# /usr/sbin/boothd site -D

Initial state
-------------

root at 10.3.3.1:~# booth client list
ticket: ticketMaster, owner: None, expires: INF

* same on all 3 nodes

Granting the ticket
-------------------

root at 10.3.3.1:~# booth client grant -t ticketMaster -s 10.3.2.10
cluster[25103]: 2013/01/18_11:16:35 info: grant command sent, result 
will be returned asynchronously, you can get the result from the log files

Status after grant
------------------

root at 10.3.3.1:~# booth client list
ticket: ticketMaster, owner: 10.3.2.10, expires: 2013/01/18 11:20:11

* same on all 3 nodes, so far so good

Simulation a network outage on 10.3.2.10
----------------------------------------

root at 10.3.2.10:~# iptables -I INPUT -s 10.3.1.0/24 -j DROP; iptables -I 
INPUT -s 10.3.3.0/24 -j DROP; iptables -I OUTPUT -d 10.3.1.0/24 -j DROP; 
iptables -I OUTPUT -d 10.3.2.0/24 -j DROP

after the outage, here the last lines of the arbitrator:

booth-arbitrator[25055]: 2013/01/18_11:26:47 debug: exit start_lease_promise
booth-arbitrator[25055]: 2013/01/18_11:26:47 debug: exit lease_promise
booth-arbitrator[25055]: 2013/01/18_11:26:47 debug: acceptor promise ...
booth-arbitrator[25055]: 2013/01/18_11:26:47 debug: ballot number: 4, 
highest promised: 5
booth-arbitrator[25055]: 2013/01/18_11:28:11 debug: lease expires ...
booth-arbitrator[25055]: 2013/01/18_11:28:11 info: command: 'crm_ticket 
-t ticketMaster -S owner -v -1' was executed
Error signing on to the CIB service: connection failed
booth-arbitrator[25055]: 2013/01/18_11:28:11 info: command: 'crm_ticket 
-t ticketMaster -S expires -v 0' was executed
Error signing on to the CIB service: connection failed
booth-arbitrator[25055]: 2013/01/18_11:28:11 info: command: 'crm_ticket 
-t ticketMaster -S ballot -v 2' was executed
Error signing on to the CIB service: connection failed
booth-arbitrator[25055]: 2013/01/18_11:28:11 info: command: 'crm_ticket 
-t ticketMaster -r --force' was executed
Error signing on to the CIB service: connection failed
booth-arbitrator[25055]: 2013/01/18_11:28:11 debug: only proposer can do 
this
booth-arbitrator[25055]: 2013/01/18_11:28:23 debug: lease_retry ...
booth-arbitrator[25055]: 2013/01/18_11:28:23 debug: only proposer can do 
this

and of course:

root at 10.3.1.10:~# booth client list
ticket: ticketMaster, owner: None, expires: INF

The debug message "only proposer can do this" comes from the 
paxos_round_request functions in paxos.c with the condition:

if (!(pi->ps->role[myid] & PROPOSER)) {
	log_debug("only proposer can do this");
	return -EOPNOTSUPP;
}

So my pick is the the PROPOSER bit is not set correctly in the structure 
in the lease_expires and lease_retry functions.  I am not very familiar 
with that code base but I'll try to figure out the issue and submit a 
patch on git hub.

Regards,

Yves