[ClusterLabs] libqb 0.17.1 - segfault at 1b8

Radoslaw Garbacz radoslaw.garbacz at xtremedatainc.com
Mon May 2 16:47:39 UTC 2016


Hi,

Firstly thank you for such a great tool.

When testing pacemaker I encountered a start error, which seems to be
related to reported libqb segmentation fault.
- cluster started and acquired quorum
- some nodes failed to connect to CIB, and lost membership as a result
- restart solved the problem

Segmentation fault reports libqb library in version 0.17.1, a standard
package provided for CentOS.6.

Please let me know if the problem is known, and if  there is a remedy (e.g.
using the latest libqb).
Logs are below.


Thank you in advance,




Logs from /var/log/messages:

Apr 22 15:46:41 (...) pacemakerd[111190]:   notice: Additional logging
available in /var/log/pacemaker.log
Apr 22 15:46:41 (...) pacemakerd[111190]:   notice: Configured corosync to
accept connections from group 498: Library error (2)
Apr 22 15:46:41 (...) pacemakerd[111190]:   notice: Starting Pacemaker
1.1.13-1.el6 (Build: 577898d):  generated-manpages agent-manpages ncurses
libqb-logging libqb-ipc upstart nagios  corosync-native atomic-attrd acls
Apr 22 15:46:41 (...) pacemakerd[111190]:   notice: Quorum acquired
Apr 22 15:46:41 (...) pacemakerd[111190]:   notice:
pcmk_quorum_notification: Node (...)[3] - state is now member (was (null))
Apr 22 15:46:41 (...) pacemakerd[111190]:   notice:
pcmk_quorum_notification: Node (...)[4] - state is now member (was (null))
Apr 22 15:46:41 (...) pacemakerd[111190]:   notice:
pcmk_quorum_notification: Node (...)[2] - state is now member (was (null))
Apr 22 15:46:41 (...) pacemakerd[111190]:   notice:
pcmk_quorum_notification: Node (...)[1] - state is now member (was (null))
Apr 22 15:46:41 (...) lrmd[111194]:   notice: Additional logging available
in /var/log/pacemaker.log
Apr 22 15:46:41 (...) stonith-ng[111193]:   notice: Additional logging
available in /var/log/pacemaker.log
Apr 22 15:46:41 (...) cib[111192]:   notice: Additional logging available
in /var/log/pacemaker.log
Apr 22 15:46:41 (...) attrd[111195]:   notice: Additional logging available
in /var/log/pacemaker.log
Apr 22 15:46:41 (...) stonith-ng[111193]:   notice: Connecting to cluster
infrastructure: corosync
Apr 22 15:46:41 (...) pengine[111196]:   notice: Additional logging
available in /var/log/pacemaker.log
Apr 22 15:46:41 (...) attrd[111195]:   notice: Connecting to cluster
infrastructure: corosync
Apr 22 15:46:41 (...) crmd[111197]:   notice: Additional logging available
in /var/log/pacemaker.log
Apr 22 15:46:41 (...) crmd[111197]:   notice: CRM Git Version: 1.1.13-1.el6
(577898d)
Apr 22 15:46:41 (...) attrd[111195]:    error: Could not connect to the
Cluster Process Group API: 11
Apr 22 15:46:41 (...) attrd[111195]:    error: Cluster connection failed
Apr 22 15:46:41 (...) attrd[111195]:   notice: Cleaning up before exit
Apr 22 15:46:41 (...) stonith-ng[111193]:   notice: crm_update_peer_proc:
Node (...)[3] - state is now member (was (null))
Apr 22 15:46:41 (...) pacemakerd[111190]:    error: Managed process 111195
(attrd) dumped core
Apr 22 15:46:41 (...) pacemakerd[111190]:    error: The attrd process
(111195) terminated with signal 11 (core=1)
Apr 22 15:46:41 (...) pacemakerd[111190]:   notice: Respawning failed child
process: attrd
Apr 22 15:46:41 (...) cib[111192]:   notice: Connecting to cluster
infrastructure: corosync
Apr 22 15:46:41 (...) cib[111192]:    error: Could not connect to the
Cluster Process Group API: 11
Apr 22 15:46:41 (...) cib[111192]:     crit: Cannot sign in to the
cluster... terminating
Apr 22 15:46:41 (...) kernel: [17169.112132] attrd[111195]: segfault at 1b8
ip 00007f6fc9dc3181 sp 00007ffd7cf668f0 error 4 in
libqb.so.0.17.1[7f6fc9db4000+21000]
Apr 22 15:46:41 (...) pacemakerd[111190]:  warning: The cib process
(111192) can no longer be respawned, shutting the cluster down.
Apr 22 15:46:41 (...) pacemakerd[111190]:   notice: Shutting down Pacemaker
Apr 22 15:46:41 (...) pacemakerd[111190]:   notice: Stopping crmd: Sent -15
to process 111197
Apr 22 15:46:41 (...) attrd[111198]:   notice: Additional logging available
in /var/log/pacemaker.log
Apr 22 15:46:41 (...) crmd[111197]:  warning: Couldn't complete CIB
registration 1 times... pause and retry
Apr 22 15:46:41 (...) crmd[111197]:   notice: Invoking handler for signal
15: Terminated
Apr 22 15:46:41 (...) crmd[111197]:   notice: Requesting shutdown, upper
limit is 1200000ms
Apr 22 15:46:41 (...) crmd[111197]:  warning: FSA: Input I_SHUTDOWN from
crm_shutdown() received in state S_STARTING
Apr 22 15:46:41 (...) crmd[111197]:   notice: State transition S_STARTING
-> S_STOPPING [ input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown ]
Apr 22 15:46:41 (...) crmd[111197]:   notice: Disconnecting from Corosync
Apr 22 15:46:41 (...) attrd[111198]:   notice: Connecting to cluster
infrastructure: corosync
Apr 22 15:46:41 (...) attrd[111198]:    error: Could not connect to the
Cluster Process Group API: 11
Apr 22 15:46:41 (...) attrd[111198]:    error: Cluster connection failed
Apr 22 15:46:41 (...) attrd[111198]:   notice: Cleaning up before exit
Apr 22 15:46:41 (...) pacemakerd[111190]:   notice: Stopping pengine: Sent
-15 to process 111196
Apr 22 15:46:41 (...) pengine[111196]:   notice: Invoking handler for
signal 15: Terminated
Apr 22 15:46:41 (...) pacemakerd[111190]:   notice: Stopping attrd: Sent
-15 to process 111198
Apr 22 15:46:41 (...) pacemakerd[111190]:    error: Managed process 111198
(attrd) dumped core
Apr 22 15:46:41 (...) pacemakerd[111190]:    error: The attrd process
(111198) terminated with signal 11 (core=1)
Apr 22 15:46:41 (...) pacemakerd[111190]:   notice: Stopping lrmd: Sent -15
to process 111194
Apr 22 15:46:41 (...) lrmd[111194]:   notice: Invoking handler for signal
15: Terminated
Apr 22 15:46:41 (...) pacemakerd[111190]:   notice: Stopping stonith-ng:
Sent -15 to process 111193
Apr 22 15:46:41 (...) kernel: [17169.121628] attrd[111198]: segfault at 1b8
ip 00007f3a98f66181 sp 00007ffe33407380 error 4 in
libqb.so.0.17.1[7f3a98f57000+21000]
Apr 22 15:46:50 (...) stonith-ng[111193]:    error: Could not connect to
the CIB service: Transport endpoint is not connected (-107)
Apr 22 15:46:50 (...) stonith-ng[111193]:   notice: Invoking handler for
signal 15: Terminated
Apr 22 15:46:50 (...) pacemakerd[111190]:   notice: Shutdown complete
Apr 22 15:46:50 (...) pacemakerd[111190]:   notice: Attempting to inhibit
respawning after fatal error




Logs from corosync log:

Apr 22 15:46:22 [93582] (...) corosync notice  [MAIN  ] Corosync Cluster
Engine exiting normally
Apr 22 15:46:40 [111147] (...) corosync notice  [MAIN  ] Corosync Cluster
Engine ('2.3.5.12-a71e'): started and ready to provide service.
Apr 22 15:46:40 [111147] (...) corosync info    [MAIN  ] Corosync built-in
features: dbus pie relro bindnow
Apr 22 15:46:40 [111147] (...) corosync notice  [TOTEM ] Initializing
transport (UDP/IP Unicast).
Apr 22 15:46:40 [111147] (...) corosync notice  [TOTEM ] Initializing
transmit/receive security (NSS) crypto: none hash: none
Apr 22 15:46:40 [111147] (...) corosync notice  [TOTEM ] The network
interface [(...)] is now up.
Apr 22 15:46:40 [111147] (...) corosync notice  [SERV  ] Service engine
loaded: corosync configuration map access [0]
Apr 22 15:46:40 [111147] (...) corosync info    [QB    ] server name: cmap
Apr 22 15:46:40 [111147] (...) corosync notice  [SERV  ] Service engine
loaded: corosync configuration service [1]
Apr 22 15:46:40 [111147] (...) corosync info    [QB    ] server name: cfg
Apr 22 15:46:40 [111147] (...) corosync notice  [SERV  ] Service engine
loaded: corosync cluster closed process group service v1.01 [2]
Apr 22 15:46:40 [111147] (...) corosync info    [QB    ] server name: cpg
Apr 22 15:46:40 [111147] (...) corosync notice  [SERV  ] Service engine
loaded: corosync profile loading service [4]
Apr 22 15:46:40 [111147] (...) corosync notice  [QUORUM] Using quorum
provider corosync_votequorum
Apr 22 15:46:40 [111147] (...) corosync notice  [SERV  ] Service engine
loaded: corosync vote quorum service v1.0 [5]
Apr 22 15:46:40 [111147] (...) corosync info    [QB    ] server name:
votequorum
Apr 22 15:46:40 [111147] (...) corosync notice  [SERV  ] Service engine
loaded: corosync cluster quorum service v0.1 [3]
Apr 22 15:46:40 [111147] (...) corosync info    [QB    ] server name: quorum
Apr 22 15:46:40 [111147] (...) corosync notice  [TOTEM ] adding new UDPU
member {(...)}
Apr 22 15:46:40 [111147] (...) corosync notice  [TOTEM ] adding new UDPU
member {(...)}
Apr 22 15:46:40 [111147] (...) corosync notice  [TOTEM ] adding new UDPU
member {(...)}
Apr 22 15:46:40 [111147] (...) corosync notice  [TOTEM ] adding new UDPU
member {(...)}
Apr 22 15:46:40 [111147] (...) corosync notice  [TOTEM ] A new membership
((...):660) was formed. Members joined: 3
Apr 22 15:46:40 [111147] (...) corosync notice  [QUORUM] Members[1]: 3
Apr 22 15:46:40 [111147] (...) corosync notice  [MAIN  ] Completed service
synchronization, ready to provide service.
Apr 22 15:46:40 [111147] (...) corosync notice  [TOTEM ] A new membership
((...):664) was formed. Members joined: 4 2 1
Apr 22 15:46:40 [111147] (...) corosync notice  [QUORUM] This node is
within the primary component and will provide service.
Apr 22 15:46:40 [111147] (...) corosync notice  [QUORUM] Members[4]: 3 4 2 1
Apr 22 15:46:40 [111147] (...) corosync notice  [MAIN  ] Completed service
synchronization, ready to provide service.
Apr 22 15:46:41 [111147] (...) corosync error   [MAIN  ] Denied connection
attempt from 498:498
Apr 22 15:46:41 [111147] (...) corosync error   [QB    ] Invalid IPC
credentials (111148-111195-2).
Apr 22 15:46:41 [111147] (...) corosync error   [MAIN  ] Denied connection
attempt from 498:498
Apr 22 15:46:41 [111147] (...) corosync error   [QB    ] Invalid IPC
credentials (111148-111192-2).
Apr 22 15:46:41 [111147] (...) corosync error   [MAIN  ] Denied connection
attempt from 498:498
Apr 22 15:46:41 [111147] (...) corosync error   [QB    ] Invalid IPC
credentials (111148-111198-2).



-- 
Best Regards,

Radoslaw Garbacz
XtremeData Incorporation
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20160502/9df92cd0/attachment-0003.html>


More information about the Users mailing list