<div dir="ltr"> We have three node cluster that is setup to stop resources on lost quorum.<br>Failure (network going down) handling is done properly, but recovery doesn't seem to work.<br><br>What happens is, services crash when we re-enable network connection.<br><br>From journal:<br><br>```<br>...<br>Jul 12 00:27:32 <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a> corosync[9069]: corosync: totemsrp.c:1328: memb_consensus_agreed: Assertion `token_memb_entries >= 1' failed.<br>Jul 12 00:27:33 <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a> attrd[9104]:    error: Connection to the CPG API failed: Library error (2)<br>Jul 12 00:27:33 <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a> stonith-ng[9100]:    error: Connection to the CPG API failed: Library error (2)<br>Jul 12 00:27:33 <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a> systemd[1]: corosync.service: Main process exited, code=dumped, status=6/ABRT<br>Jul 12 00:27:33 <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a> cib[9098]:    error: Connection to the CPG API failed: Library error (2)<br>Jul 12 00:27:33 <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a> systemd[1]: corosync.service: Failed with result 'core-dump'.<br>Jul 12 00:27:33 <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a> pacemakerd[9087]:    error: Connection to the CPG API failed: Library error (2)<br>Jul 12 00:27:33 <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a> systemd[1]: pacemaker.service: Main process exited, code=exited, status=107/n/a<br>Jul 12 00:27:33 <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a> systemd[1]: pacemaker.service: Failed with result 'exit-code'.<br>Jul 12 00:27:33 <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a> systemd[1]: Stopped Pacemaker High Availability Cluster Manager.<br>Jul 12 00:27:33 <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a> lrmd[9102]:  warning: new_event_notification (9102-9107-7): Bad file descriptor (9)<br>...<br>```<br>Pacemaker's log shows no relevant info.<br><br>This is from corosync's log:<br><br>```<br>Jul 12 00:27:33 [9107] <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a>       crmd:     info: qb_ipcs_us_withdraw:    withdrawing server sockets<br>Jul 12 00:27:33 [9104] <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a>      attrd:    error: pcmk_cpg_dispatch:      Connection to the CPG API failed: Library error (2)<br>Jul 12 00:27:33 [9100] <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a> stonith-ng:    error: pcmk_cpg_dispatch:      Connection to the CPG API failed: Library error (2)<br>Jul 12 00:27:33 [9098] <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a>        cib:    error: pcmk_cpg_dispatch:      Connection to the CPG API failed: Library error (2)<br>Jul 12 00:27:33 [9087] <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a> pacemakerd:    error: pcmk_cpg_dispatch:      Connection to the CPG API failed: Library error (2)<br>Jul 12 00:27:33 [9104] <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a>      attrd:     info: qb_ipcs_us_withdraw:    withdrawing server sockets<br>Jul 12 00:27:33 [9087] <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a> pacemakerd:     info: crm_xml_cleanup:        Cleaning up memory from libxml2<br>Jul 12 00:27:33 [9107] <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a>       crmd:     info: crm_xml_cleanup:        Cleaning up memory from libxml2<br>Jul 12 00:27:33 [9100] <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a> stonith-ng:     info: qb_ipcs_us_withdraw:    withdrawing server sockets<br>Jul 12 00:27:33 [9104] <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a>      attrd:     info: crm_xml_cleanup:        Cleaning up memory from libxml2<br>Jul 12 00:27:33 [9098] <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a>        cib:     info: qb_ipcs_us_withdraw:    withdrawing server sockets<br>Jul 12 00:27:33 [9100] <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a> stonith-ng:     info: crm_xml_cleanup:        Cleaning up memory from libxml2<br>Jul 12 00:27:33 [9098] <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a>        cib:     info: qb_ipcs_us_withdraw:    withdrawing server sockets<br>Jul 12 00:27:33 [9098] <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a>        cib:     info: qb_ipcs_us_withdraw:    withdrawing server sockets<br>Jul 12 00:27:33 [9098] <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a>        cib:     info: crm_xml_cleanup:        Cleaning up memory from libxml2<br>Jul 12 00:27:33 [9102] <a href="http://itaftestkvmls02.dc.itaf.eu">itaftestkvmls02.dc.itaf.eu</a>       lrmd:  warning: qb_ipcs_event_sendv:    new_event_notification (9102-9107-7): Bad file descriptor (9)<br>```<br><br>Please let me know if you need any further info, I'll be more than happy to provide it.<br><br>This is always reproducible in our environment:<br>Ubuntu 18.04.2<br>corosync 2.4.3-0ubuntu1.1<br>pcs 0.9.164-1<br><div>pacemaker 1.1.18-0ubuntu1.1</div><div><br></div><div>Kind regards,</div><div>Momo.<br></div></div>