<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi,<div class=""><br class=""></div><div class="">Thanks to reply. The problem is opposite to what you are saying.</div><div class=""><br class=""></div><div class="">When I build corosync with old libqb and I verified the new updated node worked properly I updated with new libqb hand-compiled and it works fine.</div><div class="">But in a normale upgrade procedure I first build libqb (removing first the old one) and then corosync, when I follow this order it does not work.</div><div class="">This is what make me crazy.</div><div class="">I do not understand this behavior.<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 6 Jul 2018, at 14:40, Christine Caulfield <<a href="mailto:ccaulfie@redhat.com" class="">ccaulfie@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On 06/07/18 13:24, Salvatore D'angelo wrote:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Hi All,<br class=""><br class="">The option --ulimit memlock=536870912 worked fine.<br class=""><br class="">I have now another strange issue. The upgrade without updating libqb<br class="">(leaving the 0.16.0) worked fine.<br class="">If after the upgrade I stop pacemaker and corosync, I download the<br class="">latest libqb version:<br class=""><a href="https://github.com/ClusterLabs/libqb/releases/download/v1.0.3/libqb-1.0.3.tar.gz" class="">https://github.com/ClusterLabs/libqb/releases/download/v1.0.3/libqb-1.0.3.tar.gz</a><br class="">build and install it everything works fine.<br class=""><br class="">If I try to install in sequence (after the installation of old code):<br class=""><br class="">libqb 1.0.3<br class="">corosync 2.4.4<br class="">pacemaker 1.1.18<br class="">crmsh 3.0.1<br class="">resource agents 4.1.1<br class=""><br class="">when I try to start corosync I got the following error:<br class="">*Starting Corosync Cluster Engine (corosync): /etc/init.d/corosync: line<br class="">99: 8470 Aborted $prog $COROSYNC_OPTIONS > /dev/null 2>&1*<br class="">*[FAILED]*<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Yes. you can't randomly swap in and out hand-compiled libqb versions.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Find one that works and stick to it. It's an annoying 'feature' of newer</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">linkers that we had to workaround in libqb. So if you rebuild libqb</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">1.0.3 then you will, in all likelihood, need to rebuild corosync to</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">match it.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Chrissie</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class=""><br class="">if I launch corosync -f I got:<br class="">*corosync: main.c:143: logsys_qb_init: Assertion `"implicit callsite<br class="">section is populated, otherwise target's build is at fault, preventing<br class="">reliable logging" && __start___verbose != __stop___verbose' failed.*<br class=""><br class="">anything is logged (even in debug mode).<br class=""><br class="">I do not understand why installing libqb during the normal upgrade<br class="">process fails while if I upgrade it after the<br class="">crmsh/pacemaker/corosync/resourceagents upgrade it works fine. <br class=""><br class="">On 3 Jul 2018, at 11:42, Christine Caulfield <<a href="mailto:ccaulfie@redhat.com" class="">ccaulfie@redhat.com</a><br class=""><<a href="mailto:ccaulfie@redhat.com" class="">mailto:ccaulfie@redhat.com</a>>> wrote:<br class=""><blockquote type="cite" class=""><br class="">On 03/07/18 07:53, Jan Pokorný wrote:<br class=""><blockquote type="cite" class="">On 02/07/18 17:19 +0200, Salvatore D'angelo wrote:<br class=""><blockquote type="cite" class="">Today I tested the two suggestions you gave me. Here what I did. <br class="">In the script where I create my 5 machines cluster (I use three<br class="">nodes for pacemaker PostgreSQL cluster and two nodes for glusterfs<br class="">that we use for database backup and WAL files).<br class=""><br class="">FIRST TEST<br class="">——————————<br class="">I added the —shm-size=512m to the “docker create” command. I noticed<br class="">that as soon as I start it the shm size is 512m and I didn’t need to<br class="">add the entry in /etc/fstab. However, I did it anyway:<br class=""><br class="">tmpfs /dev/shm tmpfs defaults,size=512m 0 0<br class=""><br class="">and then<br class="">mount -o remount /dev/shm<br class=""><br class="">Then I uninstalled all pieces of software (crmsh, resource agents,<br class="">corosync and pacemaker) and installed the new one.<br class="">Started corosync and pacemaker but same problem occurred.<br class=""><br class="">SECOND TEST<br class="">———————————<br class="">stopped corosync and pacemaker<br class="">uninstalled corosync<br class="">build corosync with --enable-small-memory-footprint and installed it<br class="">starte corosync and pacemaker<br class=""><br class="">IT WORKED.<br class=""><br class="">I would like to understand now why it didn’t worked in first test<br class="">and why it worked in second. Which kind of memory is used too much<br class="">here? /dev/shm seems not the problem, I allocated 512m on all three<br class="">docker images (obviously on my single Mac) and enabled the container<br class="">option as you suggested. Am I missing something here?<br class=""></blockquote><br class="">My suspicion then fully shifts towards "maximum number of bytes of<br class="">memory that may be locked into RAM" per-process resource limit as<br class="">raised in one of the most recent message ...<br class=""><br class=""><blockquote type="cite" class="">Now I want to use Docker for the moment only for test purpose so it<br class="">could be ok to use the --enable-small-memory-footprint, but there is<br class="">something I can do to have corosync working even without this<br class="">option?<br class=""></blockquote><br class="">... so try running the container the already suggested way:<br class=""><br class=""> docker run ... --ulimit memlock=33554432 ...<br class=""><br class="">or possibly higher (as a rule of thumb, keep doubling the accumulated<br class="">value until some unreasonable amount is reached, like the equivalent<br class="">of already used 512 MiB).<br class=""><br class="">Hope this helps.<br class=""></blockquote><br class="">This makes a lot of sense to me. As Poki pointed out earlier, in<br class="">corosync 2.4.3 (I think) we fixed a regression in that caused corosync<br class="">NOT to be locked in RAM after it forked - which was causing potential<br class="">performance issues. So if you replace an earlier corosync with 2.4.3 or<br class="">later then it will use more locked memory than before.<br class=""><br class="">Chrissie<br class=""><br class=""><br class=""><blockquote type="cite" class=""><br class=""><blockquote type="cite" class="">The reason I am asking this is that, in the future, it could be<br class="">possible we deploy in production our cluster in containerised way<br class="">(for the moment is just an idea). This will save a lot of time in<br class="">developing, maintaining and deploying our patch system. All<br class="">prerequisites and dependencies will be enclosed in container and if<br class="">IT team will do some maintenance on bare metal (i.e. install new<br class="">dependencies) it will not affects our containers. I do not see a lot<br class="">of performance drawbacks in using container. The point is to<br class="">understand if a containerised approach could save us lot of headache<br class="">about maintenance of this cluster without affect performance too<br class="">much. I am notice in Cloud environment this approach in a lot of<br class="">contexts.<br class=""></blockquote><br class=""><br class=""><br class="">_______________________________________________<br class="">Users mailing list: <a href="mailto:Users@clusterlabs.org" class="">Users@clusterlabs.org</a><span class="Apple-converted-space"> </span><<a href="mailto:Users@clusterlabs.org" class="">mailto:Users@clusterlabs.org</a>><br class=""><a href="https://lists.clusterlabs.org/mailman/listinfo/users" class="">https://lists.clusterlabs.org/mailman/listinfo/users</a><br class=""><br class="">Project Home: <a href="http://www.clusterlabs.org/" class="">http://www.clusterlabs.org</a><span class="Apple-converted-space"> </span><<a href="http://www.clusterlabs.org/" class="">http://www.clusterlabs.org/</a>><br class="">Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" class="">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br class="">Bugs: <a href="http://bugs.clusterlabs.org/" class="">http://bugs.clusterlabs.org</a><span class="Apple-converted-space"> </span><<a href="http://bugs.clusterlabs.org/" class="">http://bugs.clusterlabs.org/</a>><br class=""><br class=""></blockquote><br class="">_______________________________________________<br class="">Users mailing list: <a href="mailto:Users@clusterlabs.org" class="">Users@clusterlabs.org</a><span class="Apple-converted-space"> </span><<a href="mailto:Users@clusterlabs.org" class="">mailto:Users@clusterlabs.org</a>><br class=""><a href="https://lists.clusterlabs.org/mailman/listinfo/users" class="">https://lists.clusterlabs.org/mailman/listinfo/users</a><br class=""><br class="">Project Home: <a href="http://www.clusterlabs.org/" class="">http://www.clusterlabs.org</a><span class="Apple-converted-space"> </span><<a href="http://www.clusterlabs.org/" class="">http://www.clusterlabs.org/</a>><br class="">Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" class="">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br class="">Bugs: <a href="http://bugs.clusterlabs.org/" class="">http://bugs.clusterlabs.org</a><span class="Apple-converted-space"> </span><<a href="http://bugs.clusterlabs.org/" class="">http://bugs.clusterlabs.org/</a>><br class=""></blockquote><br class=""><br class=""><br class="">_______________________________________________<br class="">Users mailing list: <a href="mailto:Users@clusterlabs.org" class="">Users@clusterlabs.org</a><br class=""><a href="https://lists.clusterlabs.org/mailman/listinfo/users" class="">https://lists.clusterlabs.org/mailman/listinfo/users</a><br class=""><br class="">Project Home: http://www.clusterlabs.org<br class="">Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf<br class="">Bugs: http://bugs.clusterlabs.org<br class=""><br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Users mailing list: <a href="mailto:Users@clusterlabs.org" class="">Users@clusterlabs.org</a></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""><a href="https://lists.clusterlabs.org/mailman/listinfo/users" class="">https://lists.clusterlabs.org/mailman/listinfo/users</a></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Project Home: <a href="http://www.clusterlabs.org" class="">http://www.clusterlabs.org</a></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" class="">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Bugs: <a href="http://bugs.clusterlabs.org" class="">http://bugs.clusterlabs.org</a></span></div></blockquote></div><br class=""></div></body></html>