<div dir="ltr">Hi all,<div><br></div><div> I am using the ovndb-servers ocf agent[1] which is a kind of multi-state resource,when I am creating it(please see my previous email), the monitor is called only once, and the start operation is never called, according to below description, the once called monitor operation returned OCF_NOT_RUNNING, should the pacemaker will decide to execute start action based this return code? is there any way to check out what is the next action? Currently in my environment nothing happened and I am almost tried all I known ways to debug, however, no lucky, could anyone help it out? thank you very much.</div><div><br></div><div><table summary="Role implications of OCF return codes" border="1" style="border:1px solid rgb(115,47,47);width:659px;border-collapse:collapse;color:rgb(0,0,0);font-family:"liberation sans","Myriad ","Bitstream Vera Sans","Lucida Grande","Luxi Sans","Trebuchet MS",helvetica,verdana,arial,sans-serif;font-size:12px"><colgroup><col width="50%"><col width="50%"></colgroup><thead><tr><th style="vertical-align:top;border:1px solid rgb(0,0,0);text-align:left;background-color:rgb(44,64,129);padding:0.3em 0.5em;color:white">Monitor Return Code</th><th style="vertical-align:top;border:1px solid rgb(0,0,0);text-align:left;background-color:rgb(44,64,129);padding:0.3em 0.5em;color:white">Description</th></tr></thead><tbody><tr><td style="vertical-align:top;border:1px solid rgb(0,0,0);padding:0.15em 0.5em">OCF_NOT_RUNNING</td><td style="vertical-align:top;border:1px solid rgb(0,0,0);padding:0.15em 0.5em">Stopped</td></tr><tr class="even"><td style="vertical-align:top;border:1px solid rgb(0,0,0);padding:0.15em 0.5em;background-color:rgb(227,224,214)">OCF_SUCCESS</td><td style="vertical-align:top;border:1px solid rgb(0,0,0);padding:0.15em 0.5em;background-color:rgb(227,224,214)">Running (Slave)</td></tr><tr><td style="vertical-align:top;border:1px solid rgb(0,0,0);padding:0.15em 0.5em">OCF_RUNNING_MASTER</td><td style="vertical-align:top;border:1px solid rgb(0,0,0);padding:0.15em 0.5em">Running (Master)</td></tr><tr class="even"><td style="vertical-align:top;border:1px solid rgb(0,0,0);padding:0.15em 0.5em;background-color:rgb(227,224,214)">OCF_FAILED_MASTER</td><td style="vertical-align:top;border:1px solid rgb(0,0,0);padding:0.15em 0.5em;background-color:rgb(227,224,214)">Failed (Master)</td></tr><tr><td style="vertical-align:top;border:1px solid rgb(0,0,0);padding:0.15em 0.5em">Other</td><td style="vertical-align:top;border:1px solid rgb(0,0,0);padding:0.15em 0.5em">Failed (Slave)</td></tr></tbody></table></div><div><br></div><div><br></div><div>[1] <a href="https://github.com/openvswitch/ovs/blob/master/ovn/utilities/ovndb-servers.ocf">https://github.com/openvswitch/ovs/blob/master/ovn/utilities/ovndb-servers.ocf</a></div><div>Hui.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 30, 2017 at 6:39 PM, Hui Xiang <span dir="ltr"><<a href="mailto:xianghuir@gmail.com" target="_blank">xianghuir@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">The really weired thing is that the monitor is only called once other than expected repeatedly, where should I check for it?</div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 30, 2017 at 4:14 PM, Hui Xiang <span dir="ltr"><<a href="mailto:xianghuir@gmail.com" target="_blank">xianghuir@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><span style="font-size:12.8px">Thanks Ken very much for your helpful infomation.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">I am now blocking on I can't see the pacemaker DC do any further start/promote etc action on my resource agents, no helpful logs founded.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">So my first question is that in what kind of situation DC will decide do call start action? does the monitor operation need to be return OCF_SUCCESS? in my case, it will return OCF_NOT_RUNNING, and the monitor operation is not being called any more, which should be wrong as I felt that it should be called intervally. </span></div><div><span style="font-size:12.8px"><br></span></div><div><div style="font-size:12.8px"><span style="font-size:12.8px">The resource agent monitor logistic:</span></div><div style="font-size:12.8px"><span style="font-size:12.8px">In the xx_monitor function it will call xx_update, and there always hit </span><span style="font-size:12.8px"> "</span><span style="font-size:12.8px">$CRM_MASTER -D;;" , what does it usually mean? will it stopped that start operation being called? </span></div><div style="font-size:12.8px"><span style="font-size:12.8px"><br></span></div><div><div><span style="font-size:12.8px">ovsdb_server_master_update() {</span></div><div><span style="font-size:12.8px"> ocf_log info "ovsdb_server_master_update: $1}"</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px"> case $1 in</span></div><div><span style="font-size:12.8px"> $OCF_SUCCESS)</span></div><div><span style="font-size:12.8px"> $CRM_MASTER -v ${slave_score};;</span></div><div><span style="font-size:12.8px"> $OCF_RUNNING_MASTER)</span></div><div><span style="font-size:12.8px"> $CRM_MASTER -v ${master_score};;</span></div><div><span style="font-size:12.8px"> #*) $CRM_MASTER -D;;</span></div><div><span style="font-size:12.8px"> esac</span></div><div><span style="font-size:12.8px"> ocf_log info "ovsdb_server_master_update end}"</span></div><div><span style="font-size:12.8px">}</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">ovsdb_server_monitor() {</span></div><div><span style="font-size:12.8px"> ocf_log info "ovsdb_server_monitor"</span></div><div><span style="font-size:12.8px"> ovsdb_server_check_status</span></div><div><span style="font-size:12.8px"> rc=$?</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px"> ovsdb_server_master_update $rc</span></div><div><span style="font-size:12.8px"> ocf_log info "monitor is going to return $rc"</span></div><div><span style="font-size:12.8px"> return $rc</span></div><div><span style="font-size:12.8px">}</span></div></div></div><div><br></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Below is my cluster configuration:</span></div><div><br></div><div><div style="font-size:12.8px">1. First I have an vip set.</div><div style="font-size:12.8px"><div>[root@node-1 ~]# pcs resource show</div><div> vip__management_old<span style="white-space:pre-wrap"> </span>(ocf::es:ns_IPaddr2):<span style="white-space:pre-wrap"> </span>Started node-1.domain.tld</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">2. Use pcs to create ovndb-servers and constraint </div><div style="font-size:12.8px"><div>[root@node-1 ~]# pcs resource create tst-ovndb ocf:ovn:ovndb-servers manage_northd=yes master_ip=192.168.0.2 nb_master_port=6641 sb_master_port=6642 master</div><div> ([root@node-1 ~]# pcs resource meta tst-ovndb-master notify=true</div><div> Error: unable to find a resource/clone/master/group: tst-ovndb-master) ## returned error, so I changed into below command.</div><div>[root@node-1 ~]# pcs resource master tst-ovndb-master tst-ovndb notify=true</div><div>[root@node-1 ~]# pcs constraint colocation add master tst-ovndb-master with vip__management_old</div><div><br></div><div>3. pcs status</div><div>[root@node-1 ~]# pcs status</div><div> vip__management_old<span style="white-space:pre-wrap"> </span>(ocf::es:ns_IPaddr2):<span style="white-space:pre-wrap"> </span>Started node-1.domain.tld</div><div> Master/Slave Set: tst-ovndb-master [tst-ovndb]</div><div> Stopped: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld ]</div><div><br></div><div>4. pcs resource show XXX</div><div>[root@node-1 ~]# pcs resource show vip__management_old</div><div> Resource: vip__management_old (class=ocf provider=es type=ns_IPaddr2)</div><div> Attributes: nic=br-mgmt base_veth=br-mgmt-hapr ns_veth=hapr-m ip=192.168.0.2 iflabel=ka cidr_netmask=24 ns=haproxy gateway=none gateway_metric=0 iptables_start_rules=false iptables_stop_rules=false iptables_comment=default-comme<wbr>nt </div><div> Meta Attrs: migration-threshold=3 failure-timeout=60 resource-stickiness=1 </div><div> Operations: monitor interval=3 timeout=30 (vip__management_old-monitor-3<wbr>)</div><div> start interval=0 timeout=30 (vip__management_old-start-0)</div><div> stop interval=0 timeout=30 (vip__management_old-stop-0)</div><div>[root@node-1 ~]# pcs resource show tst-ovndb-master</div><div> Master: tst-ovndb-master</div><div> Meta Attrs: notify=true </div><div> Resource: tst-ovndb (class=ocf provider=ovn type=ovndb-servers)</div><div> Attributes: manage_northd=yes master_ip=192.168.0.2 nb_master_port=6641 sb_master_port=6642 </div><div> Operations: start interval=0s timeout=30s (tst-ovndb-start-timeout-30s)</div><div> stop interval=0s timeout=20s (tst-ovndb-stop-timeout-20s)</div><div> promote interval=0s timeout=50s (tst-ovndb-promote-timeout-50s<wbr>)</div><div> demote interval=0s timeout=50s (tst-ovndb-demote-timeout-50s)</div><div> monitor interval=30s timeout=20s (tst-ovndb-monitor-interval-30<wbr>s)</div><div> monitor interval=10s role=Master timeout=20s (tst-ovndb-monitor-interval-10<wbr>s-role-Master)</div><div> monitor interval=30s role=Slave timeout=20s (tst-ovndb-monitor-interval-30<wbr>s-role-Slave)</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><br></div><div><span style="font-size:12.8px">colocation colocation-tst-ovndb-master-vi<wbr>p__management_old-INFINITY inf: tst-ovndb-master:Master vip__management_old:Started</span><br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">5. I have put log in every ovndb-servers op, seems only the monitor op is being called, no promoted by the pacemaker DC:</div><div style="font-size:12.8px"><div><30>Nov 30 15:22:19 node-1 ovndb-servers(tst-ovndb)[29808<wbr>60]: INFO: ovsdb_server_monitor</div><div><30>Nov 30 15:22:19 node-1 ovndb-servers(tst-ovndb)[29808<wbr>60]: INFO: ovsdb_server_check_status</div><div><30>Nov 30 15:22:19 node-1 ovndb-servers(tst-ovndb)[29808<wbr>60]: INFO: return OCFOCF_NOT_RUNNINGG</div><div><30>Nov 30 15:22:20 node-1 ovndb-servers(tst-ovndb)[29808<wbr>60]: INFO: ovsdb_server_master_update: 7}</div><div><30>Nov 30 15:22:20 node-1 ovndb-servers(tst-ovndb)[29808<wbr>60]: INFO: ovsdb_server_master_update end}</div><div><30>Nov 30 15:22:20 node-1 ovndb-servers(tst-ovndb)[29808<wbr>60]: INFO: monitor is going to return 7</div><div><30>Nov 30 15:22:20 node-1 ovndb-servers(undef)[2980970]: INFO: metadata exit OCF_SUCCESS}</div></div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><span style="font-size:12.8px">6. The cluster property:</span></div><div><div><span style="font-size:12.8px">property cib-bootstrap-options: \</span></div><div><span style="font-size:12.8px"> have-watchdog=false \</span></div><div><span style="font-size:12.8px"> dc-version=1.1.12-a14efad \</span></div><div><span style="font-size:12.8px"> cluster-infrastructure=corosyn<wbr>c \</span></div><div><span style="font-size:12.8px"> no-quorum-policy=ignore \</span></div><div><span style="font-size:12.8px"> stonith-enabled=false \</span></div><div><span style="font-size:12.8px"> symmetric-cluster=false \</span></div><div><span style="font-size:12.8px"> last-lrm-refresh=1511802933</span></div></div><div><span style="font-size:12.8px"><br></span></div><div style="font-size:12.8px"><br></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Thank you very much for any help.</span></div><div><span style="font-size:12.8px">Hui.</span></div><div><span style="font-size:12.8px"><br></span></div><div style="font-size:12.8px"><span style="font-size:12.8px"><br></span></div><span style="font-size:12.8px">Date: Mon, 27 Nov 2017 12:07:57 -0600</span><div><span style="font-size:12.8px">From: Ken Gaillot <</span><a href="mailto:kgaillot@redhat.com" style="font-size:12.8px" target="_blank">kgaillot@redhat.com</a><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">To: Cluster Labs - All topics related to open-source clustering</span><br style="font-size:12.8px"><span style="font-size:12.8px"> welcomed <</span><a href="mailto:users@clusterlabs.org" style="font-size:12.8px" target="_blank">users@clusterlabs.org</a><span style="font-size:12.8px">>, </span><a href="mailto:jpokorny@redhat.com" style="font-size:12.8px" target="_blank">jpoko<wbr>rny@redhat.com</a><br style="font-size:12.8px"><span style="font-size:12.8px">Subject: Re: [ClusterLabs] pcs create master/slave resource doesn't</span><br style="font-size:12.8px"><span style="font-size:12.8px"> work</span><br style="font-size:12.8px"><span style="font-size:12.8px">Message-ID: <</span><a href="mailto:1511806077.5194.6.camel@redhat.com" style="font-size:12.8px" target="_blank">1511806077.5194.6.camel@redha<wbr>t.com</a><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">Content-Type: text/plain; charset="UTF-8"</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">On Fri, 2017-11-24 at 18:00 +0800, Hui Xiang wrote:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> Jan,</span><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">> ? Very appreciated on your help, I am getting further more, but still</span><br style="font-size:12.8px"><span style="font-size:12.8px">> it looks very strange.</span><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">> 1. To use "debug-promote", I upgrade pacemaker from 1.12 to 1.16, pcs</span><br style="font-size:12.8px"><span style="font-size:12.8px">> to 0.9.160.</span><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">> 2. Recreate resource with below commands</span><br style="font-size:12.8px"><span style="font-size:12.8px">> pcs resource create ovndb_servers ocf:ovn:ovndb-servers \</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ? master_ip=192.168.0.99 \</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ? op monitor interval="10s" \</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ? op monitor interval="11s" role=Master</span><br style="font-size:12.8px"><span style="font-size:12.8px">> pcs resource master ovndb_servers-master ovndb_servers \</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ? meta notify="true" master-max="1" master-node-max="1" clone-max="3"</span><br style="font-size:12.8px"><span style="font-size:12.8px">> clone-node-max="1"</span><br style="font-size:12.8px"><span style="font-size:12.8px">> pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.99 \</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ? ? op monitor interval=10s</span><br style="font-size:12.8px"><span style="font-size:12.8px">> pcs constraint colocation add VirtualIP with master ovndb_servers-</span><br style="font-size:12.8px"><span style="font-size:12.8px">> master \</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ? score=INFINITY</span><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">> 3. pcs status</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?Master/Slave Set: ovndb_servers-master [ovndb_servers]</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ? ? ?Stopped: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ]</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?VirtualIP (ocf::heartbeat:IPaddr2): Stopped</span><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">> 4. Manually run 'debug-start' on 3 nodes and 'debug-promote' on one</span><br style="font-size:12.8px"><span style="font-size:12.8px">> of nodes</span><br style="font-size:12.8px"><span style="font-size:12.8px">> run below on [ node-1.domain.tld node-2.domain.tld node-3.domain.tld</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ]</span><br style="font-size:12.8px"><span style="font-size:12.8px">> # pcs resource debug-start ovndb_servers --full</span><br style="font-size:12.8px"><span style="font-size:12.8px">> run below on [ node-1.domain.tld ]</span><br style="font-size:12.8px"><span style="font-size:12.8px">> # pcs resource debug-promote ovndb_servers --full</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">Before running debug-* commands, I'd unmanage the resource or put the</span><br style="font-size:12.8px"><span style="font-size:12.8px">cluster in maintenance mode, so Pacemaker doesn't try to "correct" your</span><br style="font-size:12.8px"><span style="font-size:12.8px">actions.</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">> 5. pcs status</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?Master/Slave Set: ovndb_servers-master [ovndb_servers]</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ? ? ?Stopped: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ]</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?VirtualIP (ocf::heartbeat:IPaddr2): Stopped</span><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">> 6. However I have seen that one of ovndb_servers has been indeed</span><br style="font-size:12.8px"><span style="font-size:12.8px">> promoted as master, but pcs status still showed all 'stopped'</span><br style="font-size:12.8px"><span style="font-size:12.8px">> what am I missing?</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">It's hard to tell from these logs. It's possible the resource agent's</span><br style="font-size:12.8px"><span style="font-size:12.8px">monitor command is not exiting with the expected status values:</span><br style="font-size:12.8px"><br style="font-size:12.8px"><a href="http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_requirements_for_multi_state_resource_agents" rel="noreferrer" style="font-size:12.8px" target="_blank">http://clusterlabs.org/doc/en-<wbr>US/Pacemaker/1.1-pcs/html-sing<wbr>le/Pacemake<br>r_Explained/index.html#_requir<wbr>ements_for_multi_state_resourc<wbr>e_agents</a><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">One of the nodes will be elected the DC, meaning it coordinates the</span><br style="font-size:12.8px"><span style="font-size:12.8px">cluster's actions. The DC's logs will have more "pengine:" messages,</span><br style="font-size:12.8px"><span style="font-size:12.8px">with each action that needs to be taken (e.g. "* Start <rsc> <node>").</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">You can look through those actions to see what the cluster decided to</span><br style="font-size:12.8px"><span style="font-size:12.8px">do -- whether the resources were e</span><span class="m_-3331598840412509222m_3471249118468256468gmail-aBn" style="font-size:12.8px"><span class="m_-3331598840412509222m_3471249118468256468gmail-aQJ">ver star</span></span><span style="font-size:12.8px">ted, whether any was</span><br style="font-size:12.8px"><span style="font-size:12.8px">promoted, and whether any were e</span><span class="m_-3331598840412509222m_3471249118468256468gmail-aBn" style="font-size:12.8px"><span class="m_-3331598840412509222m_3471249118468256468gmail-aQJ">xplicitl</span></span><span style="font-size:12.8px">y stopped.</span><br style="font-size:12.8px"><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: ocf_</span><span class="m_-3331598840412509222m_3471249118468256468gmail-aBn" style="font-size:12.8px"><span class="m_-3331598840412509222m_3471249118468256468gmail-aQJ">log:327:</span></span><span style="font-size:12.8px"> __OCF_MSG='ovndb_<wbr>servers:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> Promoting node-1.doma</span><span class="m_-3331598840412509222m_3471249118468256468gmail-aBn" style="font-size:12.8px"><span class="m_-3331598840412509222m_3471249118468256468gmail-aQJ">in.tld a</span></span><span style="font-size:12.8px">s the master'</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: ocf_log:329: case "${__OCF_PRIO}" in</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: ocf_log:333: __OCF_PRIO=INFO</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: ocf_log:338: '[' INFO = DEBUG ']'</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: ocf_log:341: ha_log 'INFO: ovndb_servers:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> Promoting nod</span><span class="m_-3331598840412509222m_3471249118468256468gmail-aBn" style="font-size:12.8px"><span class="m_-3331598840412509222m_3471249118468256468gmail-aQJ">e-1.doma</span></span><span style="font-size:12.8px">in.tld as the master'</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: ha_log:253: __ha_log 'INFO: ovndb_servers:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> Promoting node-1.domain.tld as the master'</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: __ha_log:185: </span><span class="m_-3331598840412509222m_3471249118468256468gmail-aBn" style="font-size:12.8px"><span class="m_-3331598840412509222m_3471249118468256468gmail-aQJ">local ig</span></span><span style="font-size:12.8px">nore_stderr=false</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: __ha</span><span class="m_-3331598840412509222m_3471249118468256468gmail-aBn" style="font-size:12.8px"><span class="m_-3331598840412509222m_3471249118468256468gmail-aQJ">_log:186</span></span><span style="font-size:12.8px">: local loglevel</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 1</span><span class="m_-3331598840412509222m_3471249118468256468gmail-aBn" style="font-size:12.8px"><span class="m_-3331598840412509222m_3471249118468256468gmail-aQJ">7:45:59:</span></span><span style="font-size:12.8px"> __ha_log:188: '[' 'xINFO: ovndb_servers:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> Promoting node-1.domain.tld as the master' = x--ignore-stderr ']'</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? st</span><span class="m_-3331598840412509222m_3471249118468256468gmail-aBn" style="font-size:12.8px"><span class="m_-3331598840412509222m_3471249118468256468gmail-aQJ">derr: + </span></span><span style="font-size:12.8px">17:45:59: __ha_log:190: '[' none = '' ']'</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: __ha_log:192: tty</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: __ha_log:193: '[' x = x0 -a x = xdebug ']'</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: __ha_log:195: '[' false = true ']'</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: __ha_log:199: '[' '' ']'</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: __ha_log:202: echo 'INFO: ovndb_servers:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> Promoting node-1.domain.tld as the master'</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: INFO: ovndb_servers: Promoting node-1.doma</span><span class="m_-3331598840412509222m_3471249118468256468gmail-aBn" style="font-size:12.8px"><span class="m_-3331598840412509222m_3471249118468256468gmail-aQJ">in.tld a</span></span><span style="font-size:12.8px">s the</span><br style="font-size:12.8px"><span style="font-size:12.8px">> master</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: __ha_log:204: return 0</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: ovsdb_server_promote:378:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> /usr/sbin/crm_attr</span><span class="m_-3331598840412509222m_3471249118468256468gmail-aBn" style="font-size:12.8px"><span class="m_-3331598840412509222m_3471249118468256468gmail-aQJ">ibute --</span></span><span style="font-size:12.8px">type crm_config --name OVN_REPL_INFO -s</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ovn_ovsdb_master_server -v node-1.domain.tld</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: ovsdb_server_promote:379:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ovs</span><span class="m_-3331598840412509222m_3471249118468256468gmail-aBn" style="font-size:12.8px"><span class="m_-3331598840412509222m_3471249118468256468gmail-aQJ">db_serve</span></span><span style="font-size:12.8px">r_master_update 8</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? st</span><span class="m_-3331598840412509222m_3471249118468256468gmail-aBn" style="font-size:12.8px"><span class="m_-3331598840412509222m_3471249118468256468gmail-aQJ">derr: + </span></span><span style="font-size:12.8px">17:45:59: ovsdb_server_master_update:</span><span style="font-size:12.8px">214<wbr>: case $1 in</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: ovsdb_server_master_update:</span><span style="font-size:12.8px">218<wbr>:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> /usr/sbin/crm_master -l reboot -v 10</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: ovsdb_server_promote:380: return 0</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: 458: rc=0</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ?>? stderr: + 17:45:59: 459: exit 0</span><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">> On 23/11/17 23:52 +0800, Hui Xiang wrote:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > I am working on HA with 3-nodes, which has below configurations:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> >?</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > """</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > pcs resource create ovndb_servers ocf:ovn:ovndb-servers \</span><br style="font-size:12.8px"><span style="font-size:12.8px">> >???master_ip=168.254.101.2 \</span><br style="font-size:12.8px"><span style="font-size:12.8px">> >???op monitor interval="10s" \</span><br style="font-size:12.8px"><span style="font-size:12.8px">> >???op monitor interval="11s" role=Master</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > pcs resource master ovndb_servers-master ovndb_servers \</span><br style="font-size:12.8px"><span style="font-size:12.8px">> >???meta notify="true" master-max="1" master-node-max="1" clone-</span><br style="font-size:12.8px"><span style="font-size:12.8px">> max="3"</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > clone-node-max="1"</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > pcs resource create VirtualIP ocf:heartbeat:IPaddr2</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ip=168.254.101.2 \</span><br style="font-size:12.8px"><span style="font-size:12.8px">> >?????op monitor interval=10s</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > pcs constraint order promote ovndb_servers-master then VirtualIP</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > pcs constraint colocation add VirtualIP with master ovndb_servers-</span><br style="font-size:12.8px"><span style="font-size:12.8px">> master \</span><br style="font-size:12.8px"><span style="font-size:12.8px">> >???score=INFINITY</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > """</span><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">> (Out of curiosity, this looks like a mix of output from?</span><br style="font-size:12.8px"><span style="font-size:12.8px">> pcs config export pcs-commands [or clufter cib2pcscmd -s]</span><br style="font-size:12.8px"><span style="font-size:12.8px">> and manual editing.??Is this a good guess?)</span><br style="font-size:12.8px"><span style="font-size:12.8px">> It's the output of "pcs status".</span><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">> >???However, after setting it as above, the master is not being</span><br style="font-size:12.8px"><span style="font-size:12.8px">> selected, all</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > are stopped, from pacemaker log, node-1 has been chosen as the</span><br style="font-size:12.8px"><span style="font-size:12.8px">> master, I am</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > confuse where is wrong, can anybody give a help, it would be very</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > appreciated.</span><br style="font-size:12.8px"><span style="font-size:12.8px">> >?</span><br style="font-size:12.8px"><span style="font-size:12.8px">> >?</span><br style="font-size:12.8px"><span style="font-size:12.8px">> >??Master/Slave Set: ovndb_servers-master [ovndb_servers]</span><br style="font-size:12.8px"><span style="font-size:12.8px">> >??????Stopped: [ node-1.domain.tld node-2.domain.tld node-</span><br style="font-size:12.8px"><span style="font-size:12.8px">> 3.domain.tld ]</span><br style="font-size:12.8px"><span style="font-size:12.8px">> >??VirtualIP (ocf::heartbeat:IPaddr2): Stopped</span><br style="font-size:12.8px"><span style="font-size:12.8px">> >?</span><br style="font-size:12.8px"><span style="font-size:12.8px">> >?</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > # pacemaker log</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op: ++ /cib/configuration/resources:?</span><span style="font-size:12.8px"><wbr>?<primitive</span><br style="font-size:12.8px"><span style="font-size:12.8px">> class="ocf"</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > id="ovndb_servers" provider="ovn" type="ovndb-servers"/></span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ++????????????????????????????</span><span style="font-size:12.8px"><wbr>??????<instance_attributes</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > id="ovndb_servers-instance_</span><span style="font-size:12.8px">att<wbr>ributes"></span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op: ++????????????????????????????</span><span style="font-size:12.8px"><wbr>????????<nvpair</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > id="ovndb_servers-instance_</span><span style="font-size:12.8px">att<wbr>ributes-master_ip" name="master_ip"</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > value="168.254.101.2"/></span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ++????????????????????????????</span><span style="font-size:12.8px"><wbr>??????</instance_attributes></span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op: ++????????????????????????????</span><span style="font-size:12.8px"><wbr>??????<operations></span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op: ++????????????????????????????</span><span style="font-size:12.8px"><wbr>????????<op</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > id="ovndb_servers-start-</span><span style="font-size:12.8px">timeou<wbr>t-30s" interval="0s" name="start"</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > timeout="30s"/></span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op: ++????????????????????????????</span><span style="font-size:12.8px"><wbr>????????<op</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > id="ovndb_servers-stop-</span><span style="font-size:12.8px">timeout<wbr>-20s" interval="0s" name="stop"</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > timeout="20s"/></span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op: ++????????????????????????????</span><span style="font-size:12.8px"><wbr>????????<op</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > id="ovndb_servers-promote-</span><span style="font-size:12.8px">time<wbr>out-50s" interval="0s" name="promote"</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > timeout="50s"/></span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op: ++????????????????????????????</span><span style="font-size:12.8px"><wbr>????????<op</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > id="ovndb_servers-demote-</span><span style="font-size:12.8px">timeo<wbr>ut-50s" interval="0s" name="demote"</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > timeout="50s"/></span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op: ++????????????????????????????</span><span style="font-size:12.8px"><wbr>????????<op</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > id="ovndb_servers-monitor-</span><span style="font-size:12.8px">inte<wbr>rval-10s" interval="10s"</span><br style="font-size:12.8px"><span style="font-size:12.8px">> name="monitor"/></span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op: ++????????????????????????????</span><span style="font-size:12.8px"><wbr>????????<op</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > id="ovndb_servers-monitor-</span><span style="font-size:12.8px">inte<wbr>rval-11s-role-Master" interval="11s"</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > name="monitor" role="Master"/></span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op: ++????????????????????????????</span><span style="font-size:12.8px"><wbr>??????</operations></span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op: ++????????????????????????????</span><span style="font-size:12.8px"><wbr>????</primitive></span><br style="font-size:12.8px"><span style="font-size:12.8px">> >?</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665249] node-1.domain.tld??????attrd:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > attrd_peer_update: Setting master-ovndb_servers[node-1.</span><span style="font-size:12.8px">do<wbr>main.tld]:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> (null)</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > -> 5 from node-1.domain.tld</span><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">> If it's probable your ocf:ovn:ovndb-servers agent in master mode can</span><br style="font-size:12.8px"><span style="font-size:12.8px">> run something like "attrd_updater -n master-ovndb_servers -U 5", then</span><br style="font-size:12.8px"><span style="font-size:12.8px">> it was indeed launched OK, and if it does not continue to run as</span><br style="font-size:12.8px"><span style="font-size:12.8px">> expected, there may be a problem with the agent itself.</span><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">> no change.</span><br style="font-size:12.8px"><span style="font-size:12.8px">> You can try running "pcs resource debug-promote ovndb_servers --full"</span><br style="font-size:12.8px"><span style="font-size:12.8px">> to examine the executation details (assuming the agent responds to</span><br style="font-size:12.8px"><span style="font-size:12.8px">> OCF_TRACE_RA=1 environment variable, which is what shell-based</span><br style="font-size:12.8px"><span style="font-size:12.8px">> agents built on top ocf-shellfuncs sourcable shell library from</span><br style="font-size:12.8px"><span style="font-size:12.8px">> resource-agents project, hence incl. also agents it ships,</span><br style="font-size:12.8px"><span style="font-size:12.8px">> customarily do).</span><br style="font-size:12.8px"><span style="font-size:12.8px">> Yes, thank, it's helpful.</span><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665251] node-1.domain.tld???????crmd:?</span><span style="font-size:12.8px"><wbr>??notice:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > process_lrm_event: Operation ovndb_servers_monitor_0: ok</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > (node=node-1.domain.tld, call=185, rc=0, cib-update=88,</span><br style="font-size:12.8px"><span style="font-size:12.8px">> confirmed=true)</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > <29>Nov 23 23:06:03 node-1 crmd[665251]:???notice:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> process_lrm_event:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Operation ovndb_servers_monitor_0: ok (node=node-1.domain.tld,</span><br style="font-size:12.8px"><span style="font-size:12.8px">> call=185,</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > rc=0, cib-update=88, confirmed=true)</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op: Diff: --- 0.630.2 2</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op: Diff: +++ 0.630.3 (null)</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op: +??/cib:??@num_updates=3</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_perform_op: ++</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ></span><br style="font-size:12.8px"><span style="font-size:12.8px">> /cib/status/node_state[@id='1'</span><span style="font-size:12.8px"><wbr>]/transient_attributes[@id='1'</span><span style="font-size:12.8px"><wbr>]/instanc</span><br style="font-size:12.8px"><span style="font-size:12.8px">> e_attributes[@id='status-1']:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > <nvpair id="status-1-master-ovndb_</span><span style="font-size:12.8px">serv<wbr>ers" name="master-</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ovndb_servers"</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > value="5"/></span><br style="font-size:12.8px"><span style="font-size:12.8px">> > Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><span style="font-size:12.8px"><wbr>????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > cib_process_request: Completed cib_modify operation for section</span><br style="font-size:12.8px"><span style="font-size:12.8px">> status: OK</span><br style="font-size:12.8px"><span style="font-size:12.8px">> > (rc=0, origin=node-3.domain.tld/</span><span style="font-size:12.8px">attrd<wbr>/80, version=0.630.3)</span><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">> Also depends if there's anything interesting after this point...</span><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">> ______________________________</span><span style="font-size:12.8px"><wbr>_________________</span><br style="font-size:12.8px"><span style="font-size:12.8px">> Users mailing list: </span><a href="mailto:Users@clusterlabs.org" style="font-size:12.8px" target="_blank">Users@clusterlabs.org</a><br style="font-size:12.8px"><span style="font-size:12.8px">> </span><a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" style="font-size:12.8px" target="_blank">http://lists.clusterlabs.org<wbr>/mailman/listinfo/users</a><br style="font-size:12.8px"><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">> Project Home: </span><a href="http://www.clusterlabs.org/" rel="noreferrer" style="font-size:12.8px" target="_blank">http://www.clusterlabs.o<wbr>rg</a><br style="font-size:12.8px"><span style="font-size:12.8px">> Getting started: </span><a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch" rel="noreferrer" style="font-size:12.8px" target="_blank">http://www.clusterlab<wbr>s.org/doc/Cluster_from_Scratch</a><span style="font-size:12.8px"><wbr>.</span><br style="font-size:12.8px"><span style="font-size:12.8px">> pdf</span><br style="font-size:12.8px"><span style="font-size:12.8px">> Bugs: </span><a href="http://bugs.clusterlabs.org/" rel="noreferrer" style="font-size:12.8px" target="_blank">http://bugs.clusterlabs.<wbr>org</a><br style="font-size:12.8px"><span style="font-size:12.8px">--</span><br style="font-size:12.8px"><span style="font-size:12.8px">Ken Gaillot <</span><a href="mailto:kgaillot@redhat.com" style="font-size:12.8px" target="_blank">kgaillot@redhat.com</a><span style="font-size:12.8px">></span><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>