[Pacemaker] pacemaker shutdown under high load

[Andrew Beekhof]

On 09/10/2013, at 10:53 PM, Alessandro Bono <alessandro.bono at gmail.com> wrote:

> Hi
> 
> 
> this week end my pacemaker shutdown on primary node during machine backup
> attached compressed log of primary node, logs of secondary node is too big, if needed I can provide as external link
> inspecting logs I found these errors

looks like corosync went away from underneath pacemaker, hence "Corosync connection lost!  Exiting."

> 
> Oct 05 22:26:46 [31338] ga1-ext        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=ga2-ext/crmd/17, version=0.155.87)
> Oct 05 22:26:46 [31341] ga1-ext      attrd:    error: pcmk_cpg_dispatch:        Connection to the CPG API failed: Library error (2)
> Oct 05 22:26:46 [31343] ga1-ext       crmd:    error: pcmk_cpg_dispatch:        Connection to the CPG API failed: Library error (2)
> Oct 05 22:26:46 [31343] ga1-ext       crmd:    error: crmd_cs_destroy:  connection terminated
> Oct 05 22:26:46 [31343] ga1-ext       crmd:    debug: qb_ipcs_unref:    qb_ipcs_unref() - destroying
> Oct 05 22:26:47 [31343] ga1-ext       crmd:     info: qb_ipcs_us_withdraw:      withdrawing server sockets
> Oct 05 22:26:47 [31343] ga1-ext       crmd:    debug: qb_ipcc_disconnect:       qb_ipcc_disconnect()
> Oct 05 22:26:47 [31343] ga1-ext       crmd:    debug: qb_rb_close:      Closing ringbuffer: /dev/shm/qb-attrd-request-31341-31343-9-header
> Oct 05 22:26:46 [31332] ga1-ext pacemakerd:    error: pcmk_cpg_dispatch:        Connection to the CPG API failed: Library error (2)
> Oct 05 22:26:46 [31339] ga1-ext stonith-ng:    error: pcmk_cpg_dispatch:        Connection to the CPG API failed: Library error (2)
> Oct 05 22:26:46 [31338] ga1-ext        cib:    error: pcmk_cpg_dispatch:        Connection to the CPG API failed: Library error (2)
> Oct 05 22:26:47 [31343] ga1-ext       crmd:    debug: qb_rb_close:      Closing ringbuffer: /dev/shm/qb-attrd-response-31341-31343-9-header
> Oct 05 22:26:47 [31332] ga1-ext pacemakerd:    error: mcp_cpg_destroy:  Connection destroyed
> Oct 05 22:26:47 [31339] ga1-ext stonith-ng:    error: stonith_peer_cs_destroy:  Corosync connection terminated
> Oct 05 22:26:47 [31339] ga1-ext stonith-ng:     info: stonith_shutdown:         Terminating with  1 clients
> Oct 05 22:26:47 [31339] ga1-ext stonith-ng:    debug: cib_native_signoff:       Signing out of the CIB Service
> Oct 05 22:26:47 [31339] ga1-ext stonith-ng:    debug: qb_ipcc_disconnect:       qb_ipcc_disconnect()
> Oct 05 22:26:47 [31343] ga1-ext       crmd:    debug: qb_rb_close:      Closing ringbuffer: /dev/shm/qb-attrd-event-31341-31343-9-header
> Oct 05 22:26:47 [31341] ga1-ext      attrd:     crit: attrd_cs_destroy:         Lost connection to Corosync service!
> Oct 05 22:26:47 [31341] ga1-ext      attrd:   notice: main:     Exiting...
> Oct 05 22:26:47 [31341] ga1-ext      attrd:   notice: main:     Disconnecting client 0x1b03990, pid=31343...
> Oct 05 22:26:47 [31341] ga1-ext      attrd:    debug: qb_ipcs_disconnect:       qb_ipcs_disconnect(31341-31343-9) state:2
> Oct 05 22:26:47 [31341] ga1-ext      attrd:     info: crm_client_destroy:       Destroying 0 events
> Oct 05 22:26:47 [31338] ga1-ext        cib:    error: cib_cs_destroy:   Corosync connection lost!  Exiting.
> 
> ps this is a resend to open a new thread, sorry for double mail
> 
> -- 
> Cordiali Saluti
> Alessandro Bono
> 
> <ga1-ext.corosync.log-20131006.gz>_______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20131016/ad2eecf8/attachment-0003.sig>