[Pacemaker] solaris problem

Andrei Belov defanator at gmail.com
Mon Mar 25 14:07:31 UTC 2013


Ok, I fixed this issue with the following patch against libqb 0.14.4:

--- lib/unix.c.orig     2013-03-25 12:30:50.445762231 +0000
+++ lib/unix.c  2013-03-25 12:49:59.322276376 +0000
@@ -83,7 +83,7 @@
 #if defined(QB_LINUX) || defined(QB_CYGWIN)
                snprintf(path, PATH_MAX, "/dev/shm/%s", file);
 #else
-               snprintf(path, PATH_MAX, LOCALSTATEDIR "/run/%s", file);
+               snprintf(path, PATH_MAX, "%s/%s", SOCKETDIR, file);
                is_absolute = path;
 #endif
        }
@@ -91,7 +91,7 @@
        if (fd < 0 && !is_absolute) {
                qb_util_perror(LOG_ERR, "couldn't open file %s", path);
 
-               snprintf(path, PATH_MAX, LOCALSTATEDIR "/run/%s", file);
+               snprintf(path, PATH_MAX, "%s/%s", SOCKETDIR, file);
                fd = open_mmap_file(path, file_flags);
                if (fd < 0) {
                        res = -errno;


libqb was configured with --with-socket-dir=/var/run/qb, /var/run/qb owned by
hacluster:haclient - this configuration works fine with both corosync 2.3.0 and
pacemaker 1.1.8.

Though I'm not sure that libqb is the right place to touch - maybe it'd be better
to add some enhancements to pacemaker's lib/common/mainloop.c,
mainloop_add_ipc_server() ?


Cheers.


On Mar 25, 2013, at 16:01 , Andrei Belov <defanator at gmail.com> wrote:

> 
> I've rebuilt libqb using separated SOCKETDIR (/var/run/qb), and set hacluster:haclient ownership to this dir.
> 
> After that pacemakerd has been successfully started with all its childs:
> 
> [root at ha1 /var/run/qb]# pacemakerd -fV
> Could not establish pacemakerd connection: Connection refused (146)
>    info: crm_ipc_connect:      Could not establish pacemakerd connection: Connection refused (146)
>    info: get_cluster_type:     Detected an active 'corosync' cluster
>    info: read_config:  Reading configure for stack: corosync
>  notice: crm_add_logfile:      Additional logging available in /var/log/cluster/corosync.log
>  notice: main:         Starting Pacemaker 1.1.8 (Build: 1f8858c):  ncurses libqb-logging libqb-ipc upstart systemd  corosync-native
>    info: main:         Maximum core file size is: 18446744073709551613
>    info: qb_ipcs_us_publish:   server name: pacemakerd
>  notice: update_node_processes:        48de70 Node 182452614 now known as ha1, was: 
>    info: start_child:  Forked child 60719 for process cib
>    info: start_child:  Forked child 60720 for process stonith-ng
>    info: start_child:  Forked child 60721 for process lrmd
>    info: start_child:  Forked child 60722 for process attrd
>    info: start_child:  Forked child 60723 for process pengine
>    info: start_child:  Forked child 60724 for process crmd
>    info: main:         Starting mainloop
> 
> [root at ha1 /var/run/qb]# ls -l
> total 0
> srwxrwxrwx 1 hacluster root 0 Mar 25 11:50 attrd
> srwxrwxrwx 1 root      root 0 Mar 25 11:43 cfg
> srwxrwxrwx 1 hacluster root 0 Mar 25 11:50 cib_ro
> srwxrwxrwx 1 hacluster root 0 Mar 25 11:50 cib_rw
> srwxrwxrwx 1 hacluster root 0 Mar 25 11:50 cib_shm
> srwxrwxrwx 1 root      root 0 Mar 25 11:43 cmap
> srwxrwxrwx 1 root      root 0 Mar 25 11:43 cpg
> srwxrwxrwx 1 root      root 0 Mar 25 11:50 lrmd
> srwxrwxrwx 1 root      root 0 Mar 25 11:50 pacemakerd
> srwxrwxrwx 1 hacluster root 0 Mar 25 11:50 pengine
> srwxrwxrwx 1 root      root 0 Mar 25 11:43 quorum
> srwxrwxrwx 1 root      root 0 Mar 25 11:50 stonith-ng
> 
> However, libqb still can not create some files in /var/run due to insufficient permissions:
> 
> Mar 25 11:50:45 [60719]        cib:     info: init_cs_connection_once:  Connection to 'corosync': established
> Mar 25 11:50:45 [60719]        cib:     info: crm_get_peer:     Node 182452614 is now known as ha1
> Mar 25 11:50:45 [60719]        cib:     info: crm_get_peer:     Node 182452614 has uuid 182452614
> Mar 25 11:50:45 [60719]        cib:     info: qb_ipcs_us_publish:       server name: cib_ro
> Mar 25 11:50:45 [60719]        cib:     info: qb_ipcs_us_publish:       server name: cib_rw
> Mar 25 11:50:45 [60719]        cib:     info: qb_ipcs_us_publish:       server name: cib_shm
> Mar 25 11:50:45 [60719]        cib:     info: cib_init:         Starting cib mainloop
> Mar 25 11:50:45 [60719]        cib:     info: pcmk_cpg_membership:      Joined[0.0] cib.182452614 
> Mar 25 11:50:45 [60719]        cib:     info: pcmk_cpg_membership:      Member[0.0] cib.182452614 
> Mar 25 11:50:45 [60719]        cib:     info: pcmk_cpg_membership:      Member[0.1] cib.182452614 
> Mar 25 11:50:46 [60719]        cib:    error: qb_sys_mmap_file_open:    couldn't open file /var/run/qb-cib_rw-control-60719-60720-15: Permission denied (13)
> Mar 25 11:50:46 [60719]        cib:    error: qb_ipcs_us_connect:       couldn't create file for mmap (60719-60720-15): Permission denied (13)
> Mar 25 11:50:46 [60719]        cib:    error: handle_new_connection:    Invalid IPC credentials (60719-60720-15).
> Mar 25 11:50:46 [60720] stonith-ng:     info: crm_ipc_connect:  Could not establish cib_rw connection: Permission denied (13)
> Mar 25 11:50:46 [60719]        cib:    error: qb_sys_mmap_file_open:    couldn't open file /var/run/qb-cib_shm-control-60719-60724-16: Permission denied (13)
> Mar 25 11:50:46 [60719]        cib:    error: qb_ipcs_us_connect:       couldn't create file for mmap (60719-60724-16): Permission denied (13)
> Mar 25 11:50:46 [60719]        cib:    error: handle_new_connection:    Invalid IPC credentials (60719-60724-16).
> Mar 25 11:50:46 [60724]       crmd:     info: crm_ipc_connect:  Could not establish cib_shm connection: Permission denied (13)
> Mar 25 11:50:46 [60724]       crmd:     info: do_cib_control:   Could not connect to the CIB service: Transport endpoint is not connected
> Mar 25 11:50:46 [60724]       crmd:  warning: do_cib_control:   Couldn't complete CIB registration 1 times... pause and retry
> 
> 
> If someone has working setup on Linux with corosync 2.x, libqb and pacemaker 1.1.x - I'd be very appreciated for sharing some information about a places which libqb uses for its special socket files.
> 
> Thanks in advance!
> 
> (Can we say now that this problem is libqb-related, not pacemaker?)
> 
> 
> 
> On Mar 25, 2013, at 15:30 , Andrei Belov <defanator at gmail.com> wrote:
> 
>> Andreas,
>> 
>> just tried "PCMK_ipc_type=socket pacemaker -fV" - a bunch of additional "event_send" errors appeared:
>> 
>> Mar 25 11:15:55 [33641] ha1 corosync error   [MAIN  ] event_send retuned -32, expected 256!
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 217!
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 219!
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 223!
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 223!
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 223!
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 223!
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 223!
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 223!
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 256!
>> Mar 25 11:15:55 [53980]    pengine:    error: qb_ipcs_us_publish:       Could not bind AF_UNIX (/var/run/pengine): Permission denied (13)
>> Mar 25 11:15:55 [53980]    pengine:    error: mainloop_add_ipc_server:  Could not start pengine IPC server: Unknown error (-13)
>> Mar 25 11:15:55 [53980]    pengine:    error: main:     Couldn't start IPC server
>> Mar 25 11:15:55 [53975] pacemakerd:    error: pcmk_child_exit:  Child process pengine exited (pid=53980, rc=1)
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 256!
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 223!
>> Mar 25 11:15:55 [53979]      attrd:    error: qb_ipcs_us_publish:       Could not bind AF_UNIX (/var/run/attrd): Permission denied (13)
>> Mar 25 11:15:55 [53979]      attrd:    error: mainloop_add_ipc_server:  Could not start attrd IPC server: Unknown error (-13)
>> Mar 25 11:15:55 [53979]      attrd:    error: main:     Could not start IPC server
>> Mar 25 11:15:55 [53979]      attrd:    error: main:     Aborting startup
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 223!
>> Mar 25 11:15:55 [53975] pacemakerd:    error: pcmk_child_exit:  Child process attrd exited (pid=53979, rc=100)
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 223!
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 223!
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 223!
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 223!
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 256!
>> Mar 25 11:15:55 [53976]        cib:    error: qb_ipcs_us_publish:       Could not bind AF_UNIX (/var/run/cib_ro): Permission denied (13)
>> Mar 25 11:15:55 [53976]        cib:    error: mainloop_add_ipc_server:  Could not start cib_ro IPC server: Unknown error (-13)
>> Mar 25 11:15:55 [53976]        cib:    error: qb_ipcs_us_publish:       Could not bind AF_UNIX (/var/run/cib_rw): Permission denied (13)
>> Mar 25 11:15:55 [53976]        cib:    error: mainloop_add_ipc_server:  Could not start cib_rw IPC server: Unknown error (-13)
>> Mar 25 11:15:55 [53976]        cib:    error: qb_ipcs_us_publish:       Could not bind AF_UNIX (/var/run/cib_shm): Permission denied (13)
>> Mar 25 11:15:55 [53976]        cib:    error: mainloop_add_ipc_server:  Could not start cib_shm IPC server: Unknown error (-13)
>> Mar 25 11:15:55 [53976]        cib:    error: cib_init:         Couldnt start all IPC channels, exiting.
>> Mar 25 11:15:55 [53975] pacemakerd:    error: pcmk_child_exit:  Child process cib exited (pid=53976, rc=255)
>> Mar 25 11:15:55 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 223!
>> Mar 25 11:16:04 [53977] stonith-ng:    error: setup_cib:        Could not connect to the CIB service: -134 fffffd7fc421a0b0
>> Mar 25 11:16:04 [33641] ha1 corosync error   [SERV  ] event_send retuned -32, expected 217!
>> Mar 25 11:16:04 [53975] pacemakerd:   notice: pcmk_shutdown_worker:     Attempting to inhibit respawning after fatal error
>> 
>> 
>> # fgrep 32 /usr/include/sys/errno.h 
>> #define EPIPE   32      /* Broken pipe                          */
>> 
>> 
>> 
>> On Mar 25, 2013, at 13:55 , "Grüninger, Andreas (LGL Extern)" <Andreas.Grueninger at lgl.bwl.de> wrote:
>> 
>>> With solaris/openindiana you should use this setting 
>>> export PCMK_ipc_type=socket 
>>> 
>>> Andreas
>>> 
>>> -----Ursprüngliche Nachricht-----
>>> Von: Andrei Belov [mailto:defanator at gmail.com] 
>>> Gesendet: Montag, 25. März 2013 10:43
>>> An: pacemaker at oss.clusterlabs.org
>>> Betreff: [Pacemaker] solaris problem
>>> 
>>> Hi folks,
>>> 
>>> I'm trying to build test HA cluster on Solaris 5.11 using libqb 0.14.4, corosync 2.3.0 and pacemaker 1.1.8, and I'm facing a strange problem while starting pacemaker.
>>> 
>>> Log shows the following errors:
>>> 
>>> Mar 25 09:21:26 [33720]       lrmd:    error: mainloop_add_ipc_server:  Could not start lrmd IPC server: Unknown error (-48)
>>> Mar 25 09:21:26 [33720]       lrmd:    error: try_server_create:        New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>>> Mar 25 09:21:26 [33720]       lrmd:    error: mainloop_add_ipc_server:  Could not start lrmd IPC server: Unknown error (-48)
>>> Mar 25 09:21:26 [33720]       lrmd:    error: try_server_create:        New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>>> Mar 25 09:21:26 [33720]       lrmd:    error: mainloop_add_ipc_server:  Could not start lrmd IPC server: Unknown error (-48)
>>> Mar 25 09:21:26 [33720]       lrmd:    error: try_server_create:        New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>>> Mar 25 09:21:26 [33720]       lrmd:    error: mainloop_add_ipc_server:  Could not start lrmd IPC server: Unknown error (-48)
>>> Mar 25 09:21:26 [33720]       lrmd:    error: try_server_create:        New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>>> Mar 25 09:21:26 [33720]       lrmd:    error: mainloop_add_ipc_server:  Could not start lrmd IPC server: Unknown error (-48)
>>> Mar 25 09:21:26 [33720]       lrmd:    error: try_server_create:        New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>>> Mar 25 09:21:26 [33720]       lrmd:    error: mainloop_add_ipc_server:  Could not start lrmd IPC server: Unknown error (-48)
>>> Mar 25 09:21:26 [33720]       lrmd:    error: try_server_create:        New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>>> Mar 25 09:21:26 [33720]       lrmd:    error: mainloop_add_ipc_server:  Could not start lrmd IPC server: Unknown error (-48)
>>> Mar 25 09:21:26 [33720]       lrmd:    error: try_server_create:        New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>>> Mar 25 09:21:26 [33720]       lrmd:    error: mainloop_add_ipc_server:  Could not start lrmd IPC server: Unknown error (-48)
>>> Mar 25 09:21:26 [33720]       lrmd:    error: try_server_create:        New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>>> Mar 25 09:21:26 [33720]       lrmd:    error: mainloop_add_ipc_server:  Could not start lrmd IPC server: Unknown error (-48)
>>> Mar 25 09:21:26 [33720]       lrmd:    error: try_server_create:        New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>>> Mar 25 09:21:26 [33720]       lrmd:    error: mainloop_add_ipc_server:  Could not start lrmd IPC server: Unknown error (-48)
>>> Mar 25 09:21:26 [33720]       lrmd:    error: try_server_create:        New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>>> Mar 25 09:21:26 [33720]       lrmd:    error: main:     Failed to allocate lrmd server.  shutting down
>>> Mar 25 09:21:26 [33722]    pengine:    error: mainloop_add_ipc_server:  Could not start pengine IPC server: Unknown error (-48)
>>> Mar 25 09:21:26 [33722]    pengine:    error: main:     Couldn't start IPC server
>>> Mar 25 09:21:26 [33717] pacemakerd:    error: pcmk_child_exit:  Child process lrmd exited (pid=33720, rc=255)
>>> Mar 25 09:21:26 [33721]      attrd:    error: qb_ipcs_us_publish:       Could not bind AF_UNIX (/var/run/attrd): Permission denied (13)
>>> Mar 25 09:21:26 [33721]      attrd:    error: mainloop_add_ipc_server:  Could not start attrd IPC server: Unknown error (-13)
>>> Mar 25 09:21:26 [33721]      attrd:    error: main:     Could not start IPC server
>>> Mar 25 09:21:26 [33721]      attrd:    error: main:     Aborting startup
>>> Mar 25 09:21:26 [33717] pacemakerd:    error: pcmk_child_exit:  Child process pengine exited (pid=33722, rc=1)
>>> Mar 25 09:21:26 [33717] pacemakerd:    error: pcmk_child_exit:  Child process attrd exited (pid=33721, rc=100)
>>> Mar 25 09:21:26 [33718]        cib:    error: qb_ipcs_us_publish:       Could not bind AF_UNIX (/var/run/cib_ro): Permission denied (13)
>>> Mar 25 09:21:26 [33718]        cib:    error: mainloop_add_ipc_server:  Could not start cib_ro IPC server: Unknown error (-13)
>>> Mar 25 09:21:26 [33718]        cib:    error: qb_ipcs_us_publish:       Could not bind AF_UNIX (/var/run/cib_rw): Permission denied (13)
>>> Mar 25 09:21:26 [33718]        cib:    error: mainloop_add_ipc_server:  Could not start cib_rw IPC server: Unknown error (-13)
>>> Mar 25 09:21:26 [33718]        cib:    error: mainloop_add_ipc_server:  Could not start cib_shm IPC server: Unknown error (-48)
>>> Mar 25 09:21:26 [33718]        cib:    error: cib_init:         Couldnt start all IPC channels, exiting.
>>> Mar 25 09:21:26 [33717] pacemakerd:    error: pcmk_child_exit:  Child process cib exited (pid=33718, rc=255)
>>> Mar 25 09:21:35 [33719] stonith-ng:    error: setup_cib:        Could not connect to the CIB service: -134 fffffd7fc421a0b0
>>> Mar 25 09:21:35 [33717] pacemakerd:   notice: pcmk_shutdown_worker:     Attempting to inhibit respawning after fatal error
>>> 
>>> Full log (in case of any things I've probably missed) is attached.
>>> 
>>> I wonder to know the reason of "unknown error (-48)" - on this system 48 in errno.h is "ENOTSUP", but I haven't found the exact place in code where this may happen (so I'm not sure about that).
>>> 
>>> Just for record - I'm able to run corosync on two nodes and see them connected without any visible problems - thus, I suppose there may be something wrong with either pacemaker or libqb.
>>> 
>>> Any help will be greatly appreciated!
>>> 
>>> Thanks,
>>> Andrei.
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>> 
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>> 
> 





More information about the Pacemaker mailing list