Mar 22 12:29:28 debug [TOTEM ] The token was lost in the OPERATIONAL state. Mar 22 12:29:28 notice [TOTEM ] A processor failed, forming new configuration. Mar 22 12:29:28 debug [TOTEM ] entering GATHER state from 2(The token was lost in the OPERATIONAL state.). Mar 22 12:29:31 debug [TOTEM ] entering GATHER state from 0(consensus timeout). Mar 22 12:29:31 debug [TOTEM ] got commit token Mar 22 12:29:31 debug [TOTEM ] Saving state aru 4a high seq received 4a Mar 22 12:29:31 debug [TOTEM ] Storing new sequence id for ring d94 Mar 22 12:29:31 debug [TOTEM ] entering COMMIT state. Mar 22 12:29:31 debug [TOTEM ] got commit token Mar 22 12:29:31 debug [TOTEM ] entering RECOVERY state. Mar 22 12:29:31 debug [TOTEM ] TRANS [0] member 172.17.0.34: Mar 22 12:29:31 debug [TOTEM ] TRANS [1] member 172.17.0.42: Mar 22 12:29:31 debug [TOTEM ] position [0] member 172.17.0.34: Mar 22 12:29:31 debug [TOTEM ] previous ring seq d90 rep 172.17.0.34 Mar 22 12:29:31 debug [TOTEM ] aru 4a high delivered 4a received flag 1 Mar 22 12:29:31 debug [TOTEM ] position [1] member 172.17.0.42: Mar 22 12:29:31 debug [TOTEM ] previous ring seq d90 rep 172.17.0.34 Mar 22 12:29:31 debug [TOTEM ] aru 4a high delivered 4a received flag 1 Mar 22 12:29:31 debug [TOTEM ] Did not need to originate any messages in recovery. Mar 22 12:29:31 debug [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru ffffffff Mar 22 12:29:31 debug [TOTEM ] install seq 0 aru 0 high seq received 0 Mar 22 12:29:31 debug [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0 Mar 22 12:29:31 debug [TOTEM ] install seq 0 aru 0 high seq received 0 Mar 22 12:29:31 debug [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0 Mar 22 12:29:31 debug [TOTEM ] install seq 0 aru 0 high seq received 0 Mar 22 12:29:31 debug [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0 Mar 22 12:29:31 debug [TOTEM ] install seq 0 aru 0 high seq received 0 Mar 22 12:29:31 debug [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0 Mar 22 12:29:31 debug [TOTEM ] Resetting old ring state Mar 22 12:29:31 debug [TOTEM ] recovery to regular 1-0 Mar 22 12:29:31 debug [MAIN ] Member left: r(0) ip(172.17.0.35) Mar 22 12:29:31 debug [TOTEM ] waiting_trans_ack changed to 1 Mar 22 12:29:31 debug [TOTEM ] entering OPERATIONAL state. Mar 22 12:29:31 notice [TOTEM ] A new membership (172.17.0.34:3476) was formed. Members left: 2016002 Mar 22 12:29:31 debug [SYNC ] Committing synchronization for corosync configuration map access Mar 22 12:29:31 debug [QB ] Not first sync -> no action Mar 22 12:29:31 debug [CPG ] got joinlist message from node 1ec303 Mar 22 12:29:31 debug [CPG ] comparing: sender r(0) ip(172.17.0.34) ; members(old:3 left:1) Mar 22 12:29:31 debug [CPG ] comparing: sender r(0) ip(172.17.0.42) ; members(old:3 left:1) Mar 22 12:29:31 debug [CPG ] chosen downlist: sender r(0) ip(172.17.0.34) ; members(old:3 left:1) Mar 22 12:29:31 debug [CPG ] left_list_entries:1 Mar 22 12:29:31 debug [CPG ] left_list[0] group:dlm:controld\x00, ip:r(0) ip(172.17.0.35) , pid:4725 Mar 22 12:29:31 debug [CPG ] left_list_entries:1 Mar 22 12:29:31 debug [CPG ] left_list[0] group:dlm:ls:GFS\x00, ip:r(0) ip(172.17.0.35) , pid:4725 Mar 22 12:29:31 debug [CPG ] got joinlist message from node 1ec301 Mar 22 12:29:31 debug [SYNC ] Committing synchronization for corosync cluster closed process group service v1.01 Mar 22 12:29:31 debug [CPG ] joinlist_messages[0] group:dlm:ls:GFS\x00, ip:r(0) ip(172.17.0.34) , pid:6312 Mar 22 12:29:31 debug [CPG ] joinlist_messages[1] group:dlm:controld\x00, ip:r(0) ip(172.17.0.34) , pid:6312 Mar 22 12:29:31 debug [CPG ] joinlist_messages[2] group:dlm:ls:GFS\x00, ip:r(0) ip(172.17.0.42) , pid:6302 Mar 22 12:29:31 debug [CPG ] joinlist_messages[3] group:dlm:controld\x00, ip:r(0) ip(172.17.0.42) , pid:6302 Mar 22 12:29:31 debug [QUORUM] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No Mar 22 12:29:31 debug [QUORUM] got nodeinfo message from cluster node 2016001 Mar 22 12:29:31 debug [QUORUM] nodeinfo message[2016001]: votes: 1, expected: 3 flags: 1 Mar 22 12:29:31 debug [QUORUM] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No Mar 22 12:29:31 debug [QUORUM] got nodeinfo message from cluster node 2016001 Mar 22 12:29:31 debug [QUORUM] nodeinfo message[0]: votes: 0, expected: 0 flags: 0 Mar 22 12:29:31 debug [QUORUM] got nodeinfo message from cluster node 2016003 Mar 22 12:29:31 debug [QUORUM] nodeinfo message[2016003]: votes: 1, expected: 3 flags: 1 Mar 22 12:29:31 debug [QUORUM] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No Mar 22 12:29:31 debug [QUORUM] total_votes=2, expected_votes=3 Mar 22 12:29:31 debug [QUORUM] node 2016001 state=1, votes=1, expected=3 Mar 22 12:29:31 debug [QUORUM] node 2016002 state=2, votes=1, expected=3 Mar 22 12:29:31 debug [QUORUM] node 2016003 state=1, votes=1, expected=3 Mar 22 12:29:31 debug [QUORUM] lowest node id: 2016001 us: 2016003 Mar 22 12:29:31 debug [QUORUM] got nodeinfo message from cluster node 2016003 Mar 22 12:29:31 debug [QUORUM] nodeinfo message[0]: votes: 0, expected: 0 flags: 0 Mar 22 12:29:31 debug [SYNC ] Committing synchronization for corosync vote quorum service v1.0 Mar 22 12:29:31 debug [QUORUM] total_votes=2, expected_votes=3 Mar 22 12:29:31 debug [QUORUM] node 2016001 state=1, votes=1, expected=3 Mar 22 12:29:31 debug [QUORUM] node 2016002 state=2, votes=1, expected=3 Mar 22 12:29:31 debug [QUORUM] node 2016003 state=1, votes=1, expected=3 Mar 22 12:29:31 debug [QUORUM] lowest node id: 2016001 us: 2016003 Mar 22 12:29:31 notice [QUORUM] Members[2]: 2016001 2016003 Mar 22 12:29:31 debug [QUORUM] sending quorum notification to (nil), length = 56 Mar 22 12:29:31 notice [MAIN ] Completed service synchronization, ready to provide service. Mar 22 12:29:31 debug [TOTEM ] waiting_trans_ack changed to 0