<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi,<div class=""><br class=""></div><div class="">Yes,</div><div class=""><br class=""></div><div class="">I am reproducing only the required part for test. I think the original system has a larger shm. The problem is that I do not know exactly how to change it.</div><div class="">I tried the following steps, but I have the impression I didn’t performed the right one:</div><div class=""><br class=""></div><div class="">1. remove everything under /tmp</div><div class="">2. Added the following line to /etc/fstab</div><div class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">tmpfs /tmp tmpfs defaults,nodev,nosuid,mode=1777,size=128M 0 0</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">3. mount /tmp</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">4. df -h</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Filesystem Size Used Avail Use% Mounted on</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">overlay 63G 11G 49G 19% /</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">tmpfs 64M 4.0K 64M 1% /dev</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">tmpfs 1000M 0 1000M 0% /sys/fs/cgroup</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">osxfs 466G 158G 305G 35% /Users</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">/dev/sda1 63G 11G 49G 19% /etc/hosts</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">shm 64M 11M 54M 16% /dev/shm</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">tmpfs 1000M 0 1000M 0% /sys/firmware</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><b class="">tmpfs 128M 0 128M 0% /tmp</b></span></div></span></div><div><br class=""></div><div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">The errors are exactly the same.</span></div><div class="">I have the impression that I changed the wrong parameter. Probably I have to change:</div></div><div><div style="font-family: Menlo; font-size: 11px; margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">shm 64M 11M 54M 16% /dev/shm</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class=""></span></div><div class="">but I do not know how to do that. Any suggestion?</div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class=""></span></div><blockquote type="cite" class=""><div class="">On 26 Jun 2018, at 09:48, Christine Caulfield <<a href="mailto:ccaulfie@redhat.com" class="">ccaulfie@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On 25/06/18 20:41, Salvatore D'angelo wrote:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Hi,<br class=""><br class="">Let me add here one important detail. I use Docker for my test with 5 containers deployed on my Mac.<br class="">Basically the team that worked on this project installed the cluster on soft layer bare metal.<br class="">The PostgreSQL cluster was hard to test and if a misconfiguration occurred recreate the cluster from scratch is not easy.<br class="">Test it was a cumbersome if you consider that we access to the machines with a complex system hard to describe here.<br class="">For this reason I ported the cluster on Docker for test purpose. I am not interested to have it working for months, I just need a proof of concept.<span class="Apple-converted-space"> </span><br class=""><br class="">When the migration works I’ll port everything on bare metal where the size of resources are ambundant. <br class=""><br class="">Now I have enough RAM and disk space on my Mac so if you tell me what should be an acceptable size for several days of running it is ok for me.<br class="">It is ok also have commands to clean the shm when required.<br class="">I know I can find them on Google but if you can suggest me these info I’ll appreciate. I have OS knowledge to do that but I would like to avoid days of guesswork and try and error if possible.<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">I would recommend at least 128MB of space on /dev/shm, 256MB if you can</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">spare it. My 'standard' system uses 75MB under normal running allowing</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">for one command-line query to run.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">If I read this right then you're reproducing a bare-metal system in</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">containers now? so the original systems will have a default /dev/shm</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">size which is probably much larger than your containers?</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">I'm just checking here that we don't have a regression in memory usage</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">as Poki suggested.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Chrissie</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" class="">On 25 Jun 2018, at 21:18, Jan Pokorný <<a href="mailto:jpokorny@redhat.com" class="">jpokorny@redhat.com</a>> wrote:<br class=""><br class="">On 25/06/18 19:06 +0200, Salvatore D'angelo wrote:<br class=""><blockquote type="cite" class="">Thanks for reply. I scratched my cluster and created it again and<br class="">then migrated as before. This time I uninstalled pacemaker,<br class="">corosync, crmsh and resource agents with make uninstall<br class=""><br class="">then I installed new packages. The problem is the same, when<br class="">I launch:<br class="">corosync-quorumtool -ps<br class=""><br class="">I got: Cannot initialize QUORUM service<br class=""><br class="">Here the log with debug enabled:<br class=""><br class=""><br class="">[18019] pg3 corosyncerror [QB ] couldn't create circular mmap on /dev/shm/qb-cfg-event-18020-18028-23-data<br class="">[18019] pg3 corosyncerror [QB ] qb_rb_open:cfg-event-18020-18028-23: Resource temporarily unavailable (11)<br class="">[18019] pg3 corosyncdebug [QB ] Free'ing ringbuffer: /dev/shm/qb-cfg-request-18020-18028-23-header<br class="">[18019] pg3 corosyncdebug [QB ] Free'ing ringbuffer: /dev/shm/qb-cfg-response-18020-18028-23-header<br class="">[18019] pg3 corosyncerror [QB ] shm connection FAILED: Resource temporarily unavailable (11)<br class="">[18019] pg3 corosyncerror [QB ] Error in connection setup (18020-18028-23): Resource temporarily unavailable (11)<br class=""><br class="">I tried to check /dev/shm and I am not sure these are the right<br class="">commands, however:<br class=""><br class="">df -h /dev/shm<br class="">Filesystem Size Used Avail Use% Mounted on<br class="">shm 64M 16M 49M 24% /dev/shm<br class=""><br class="">ls /dev/shm<br class="">qb-cmap-request-18020-18036-25-data qb-corosync-blackbox-data qb-quorum-request-18020-18095-32-data<br class="">qb-cmap-request-18020-18036-25-header qb-corosync-blackbox-header qb-quorum-request-18020-18095-32-header<br class=""><br class="">Is 64 Mb enough for /dev/shm. If no, why it worked with previous<br class="">corosync release?<br class=""></blockquote><br class="">For a start, can you try configuring corosync with<br class="">--enable-small-memory-footprint switch?<br class=""><br class="">Hard to say why the space provisioned to /dev/shm is the direct<br class="">opposite of generous (per today's standards), but may be the result<br class="">of automatic HW adaptation, and if RAM is so scarce in your case,<br class="">the above build-time toggle might help.<br class=""><br class="">If not, then exponentially increasing size of /dev/shm space is<br class="">likely your best bet (I don't recommended fiddling with mlockall()<br class="">and similar measures in corosync).<br class=""><br class="">Of course, feel free to raise a regression if you have a reproducible<br class="">comparison between two corosync (plus possibly different libraries<br class="">like libqb) versions, one that works and one that won't, in<br class="">reproducible conditions (like this small /dev/shm, VM image, etc.).<br class=""><br class="">--<span class="Apple-converted-space"> </span><br class="">Jan (Poki)<br class="">_______________________________________________<br class="">Users mailing list: <a href="mailto:Users@clusterlabs.org" class="">Users@clusterlabs.org</a><br class=""><a href="https://lists.clusterlabs.org/mailman/listinfo/users" class="">https://lists.clusterlabs.org/mailman/listinfo/users</a><br class=""><br class="">Project Home: http://www.clusterlabs.org<br class="">Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf<br class="">Bugs: http://bugs.clusterlabs.org<br class=""></blockquote><br class="">_______________________________________________<br class="">Users mailing list:<span class="Apple-converted-space"> </span><a href="mailto:Users@clusterlabs.org" class="">Users@clusterlabs.org</a><br class=""><a href="https://lists.clusterlabs.org/mailman/listinfo/users" class="">https://lists.clusterlabs.org/mailman/listinfo/users</a><br class=""><br class="">Project Home:<span class="Apple-converted-space"> </span><a href="http://www.clusterlabs.org/" class="">http://www.clusterlabs.org</a><br class="">Getting started:<span class="Apple-converted-space"> </span><a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" class="">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br class="">Bugs:<span class="Apple-converted-space"> </span><a href="http://bugs.clusterlabs.org/" class="">http://bugs.clusterlabs.org</a><br class=""><br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Users mailing list:<span class="Apple-converted-space"> </span></span><a href="mailto:Users@clusterlabs.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Users@clusterlabs.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="https://lists.clusterlabs.org/mailman/listinfo/users" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://lists.clusterlabs.org/mailman/listinfo/users</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Project Home:<span class="Apple-converted-space"> </span></span><a href="http://www.clusterlabs.org/" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://www.clusterlabs.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Getting started:<span class="Apple-converted-space"> </span></span><a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Bugs:<span class="Apple-converted-space"> </span></span><a href="http://bugs.clusterlabs.org/" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://bugs.clusterlabs.org</a></div></blockquote></div><br class=""></div></body></html>