<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">2013/1/23 Kashif Jawed Siddiqui <span dir="ltr"><<a href="mailto:kashifjs@huawei.com" target="_blank">kashifjs@huawei.com</a>></span><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-size:10pt;font-family:Tahoma">
<p>There is a Pacemaker bug which cannot be replaced due to legacy tracking and backward compatibility</p><div class="im">
<p> </p>
<div>colocation FS_WITH_DRBD inf: IP-AND-FS ms_drbd:Master</div>
</div><div>order DRBD_BEF_FS inf: IP-AND-FS:start ms_drbd:promote </div>
<div> </div>
<div>if colocation and order is specified between 2 Resources then it means the 2nd one is the first and 1st one is the next</div>
<div> </div>
<div>For example,
<div class="im"><div>colocation FS_WITH_DRBD inf: IP-AND-FS ms_drbd:Master</div>
</div><div>//Actually means first start IP-AND-FS then make ms_drbd as Master</div>
<div>// But in behavior, it is first make ms_drbd as Master and then start IP-AND-FS </div>
<div></div></div></div></div></blockquote><div><br></div><div style>For colocation that is true. But.....</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="direction:ltr;font-size:10pt;font-family:Tahoma">
<div>Also for order, it is the same behavior...</div>
<div><div class="im">
<div>order DRBD_BEF_FS inf: IP-AND-FS:start ms_drbd:promote </div>
</div><div>
<div>//Actually means first start IP-AND-FS then promote ms_drbd </div>
<div>// But in behavior, it is first promote ms_drbd and then start IP-AND-FS</div></div></div></div></blockquote><div><br></div><div style>For order it working as we expected, so for your example first <span style="color:rgb(80,0,80);font-family:Tahoma;font-size:13px">IP-AND-FS:start then </span><span style="color:rgb(80,0,80);font-family:Tahoma;font-size:13px">ms_drbd:promote what is wrong in my case, because filesystem is on top of drbd.</span></div>
<div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div style="direction:ltr;font-size:10pt;font-family:Tahoma">
<div>
<div>But when you define order and colocation between 3 or more resources, then the interpretation is fine...</div>
<div> </div>
<div> </div>
<div>Therefore in your scenario, you must configure (as I said)</div>
<div><div class="im">
<div>colocation FS_WITH_DRBD inf: IP-AND-FS ms_drbd:Master</div>
</div><div>order DRBD_BEF_FS inf: IP-AND-FS:start ms_drbd:promote </div>
<div> </div>
<div>Though it means opposite, but the behavior is as you expect.(owing to Pacemaker backward compatibility)</div>
<div> </div>
<div> </div></div></div></div></div></blockquote><div style>When I've hanged it, I could not shut down cluster propely. I've got a lot of error in syslogs:</div><div style><br></div><div style><div>[776658.542263] block drbd1: state = { cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown r--- }</div>
<div>[776658.542352] block drbd1: wanted = { cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }</div><div>[776659.561595] block drbd1: State change failed: Device is held open by someone</div><div>[776659.561653] block drbd1: state = { cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown r--- }</div>
<div>[776659.561742] block drbd1: wanted = { cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }</div><div><br></div><div style>because pacemaker wanted to down drbd before unmount filesystem.</div><div style>
<br></div><div style><br></div><div style>To sum up, colocation rule and order from my config is correct.</div><div style><br></div><div style><br></div><div style>I wonder why I cant migrate group resource without force parameter. I use 1.0.9 version of pacemaker, maybe it is some kind of bug, I'll try newer version.</div>
<div style><br></div><div style><br></div></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><div style="direction:ltr;font-size:10pt;font-family:Tahoma"><div><div>
</div>
</div><div class="im">
<div>
<p> </p>
<div style="font-family:Tahoma;font-size:13px">
<div style="margin:0px"><font face="Calibri,sans-serif"><span style="font-size:11pt">Regards,</span></font></div>
<div style="margin:0px"><font face="Calibri,sans-serif"><span style="font-size:11pt"><font color="black" face="Arial,sans-serif"><span style="font-size:10pt">Kashif Jawed Siddiqui<br>
</span></font></span></font></div>
<font face="Calibri,sans-serif"><span style="font-size:11pt">
<div style="margin:0px"><font color="#808080" face="Arial"></font> </div>
<div style="margin:0px"><font color="gray" size="1" face="Arial,sans-serif"><span style="font-size:7.5pt">
<p style="margin:0in 0in 0pt" class=""><span style="font-family:Arial,sans-serif;font-size:10pt"><font color="#000000">***************************************************************************************<br>
This e-mail and attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended recipient's) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!</font></span></p>
</span></font></div>
</span></font></div>
</div>
</div><div style="font-size:16px;font-family:'Times New Roman'">
<hr>
<div style="direction:ltr"><font color="#000000" face="Tahoma"><b>From:</b> and k [<a href="mailto:not4mad@gmail.com" target="_blank">not4mad@gmail.com</a>]<br>
<b>Sent:</b> Wednesday, January 23, 2013 7:31 PM<br>
<b>To:</b> The Pacemaker cluster resource manager<br>
<b>Subject:</b> Re: [Pacemaker] [PACEMAKER] Why cant't migrate group resource with collation in drbd resource<br>
</font><br>
</div><div><div class="h5">
<div></div>
<div>
<div dir="ltr"><br>
<div><span style="font-family:arial,sans-serif;white-space:nowrap;font-size:13px"><br>
</span></div>
<div><span style="font-family:arial,sans-serif;white-space:nowrap;font-size:13px"><br>
</span></div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">2013/1/23 emmanuel segura <span dir="ltr"><<a href="mailto:emi2fast@gmail.com" target="_blank">emi2fast@gmail.com</a>></span><br>
<blockquote style="border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid;margin:0px 0px 0px 0.8ex;padding-left:1ex" class="gmail_quote">
Ummm<br>
<br>
Fist the IP-AND-FS? but what happen if the FS is on drbd?<br>
</blockquote>
<div><br>
</div>
<div><span style="font-family:arial,sans-serif;white-space:nowrap;font-size:13px">Emmanuel, you are right, filesystem is on top of drbd device, so i cant run that group before drbd promote.</span><br>
<div><span style="font-family:arial,sans-serif;white-space:nowrap;font-size:13px"><br>
</span></div>
<div><font face="arial, sans-serif"><span style="white-space:nowrap">I also noticed that, I can migrate it by force with use:</span></font><br>
<br>
<font face="arial, sans-serif"><span style="white-space:nowrap">crm_resource -M -f -r fs_r1 -H drbd02</span></font><br>
</div>
</div>
<div><font face="arial, sans-serif"><span style="white-space:nowrap"><br>
</span></font></div>
<div><font face="arial, sans-serif"><span style="white-space:nowrap">master is promoted correctly, and group is succesfully migrated.</span></font></div>
<div><font face="arial, sans-serif"><span style="white-space:nowrap"><br>
</span></font></div>
<div><font face="arial, sans-serif"><span style="white-space:nowrap"><br>
</span></font></div>
<div><font face="arial, sans-serif"><span style="white-space:nowrap">But i wonder why it doesn't work without force (-f) parameter ?? Any idea ?</span></font></div>
<div> </div>
<div><br>
</div>
<div><br>
</div>
<blockquote style="border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid;margin:0px 0px 0px 0.8ex;padding-left:1ex" class="gmail_quote">
<br>
Thanks<br>
<br>
<div class="gmail_quote">
<div>
<div>2013/1/23 Kashif Jawed Siddiqui <span dir="ltr"><<a href="mailto:kashifjs@huawei.com" target="_blank">kashifjs@huawei.com</a>></span><br>
</div>
</div>
<blockquote style="border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid;margin:0px 0px 0px 0.8ex;padding-left:1ex" class="gmail_quote">
<div>
<div>
<div>
<div style="font-family:Tahoma;direction:ltr;font-size:10pt">
<p>You must change the order </p>
<p> </p>
<div>#order DRBD_BEF_FS inf: ms_drbd:promote IP-AND-FS:start </div>
<div> </div>
<div>
<div>order DRBD_BEF_FS inf: IP-AND-FS:start ms_drbd:promote </div>
<div> </div>
<div>//First start IP-AND-FS, only then promote ms_drbd</div>
</div>
<div>
<p> </p>
<div style="font-family:Tahoma;font-size:13px">
<div style="margin:0px"><font face="Calibri,sans-serif"><span style="font-size:11pt">Regards,</span></font></div>
<div style="margin:0px"><font face="Calibri,sans-serif"><span style="font-size:11pt"><font color="black" face="Arial,sans-serif"><span style="font-size:10pt">Kashif Jawed Siddiqui<br>
</span></font></span></font></div>
<font face="Calibri,sans-serif"><span style="font-size:11pt">
<div style="margin:0px"> </div>
<div style="margin:0px"><font color="gray" size="1" face="Arial,sans-serif"><span style="font-size:7.5pt">
<p style="margin:0in 0in 0pt"><span style="font-family:Arial,sans-serif;font-size:10pt"><font color="#000000">***************************************************************************************<br>
This e-mail and attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended recipient's) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!</font></span></p>
</span></font></div>
</span></font></div>
</div>
<div style="font-family:'Times New Roman';font-size:16px">
<hr>
<div style="direction:ltr"><font color="#000000" face="Tahoma"><b>From:</b> and k [<a href="mailto:not4mad@gmail.com" target="_blank">not4mad@gmail.com</a>]<br>
<b>Sent:</b> Wednesday, January 23, 2013 4:34 PM<br>
<b>To:</b> <a href="mailto:pacemaker@oss.clusterlabs.org" target="_blank">pacemaker@oss.clusterlabs.org</a><br>
<b>Subject:</b> [Pacemaker] [PACEMAKER] Why cant't migrate group resource with collation in drbd resource<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr">Hello Everybody,
<div><br>
</div>
<div>I've got a problem (but I am not quite sure if it is not a feature in pacemaker ) that's why I decided to write on that mailing list.</div>
<div><br>
</div>
<div>It comes about migrate resource with collation in drbd resource.</div>
<div><br>
</div>
<div>I've got group including virtual IP and filesystem which is collated with ms resource drbd in mster-slave configuration.</div>
<div><br>
</div>
<div>
<div>============</div>
<div>Last updated: Wed Jan 23 03:40:55 2013</div>
<div>Stack: openais</div>
<div>Current DC: drbd01 - partition with quorum</div>
<div>Version: 1.0.9-74392a28b7f31d7ddc86689598bd23114f58978b</div>
<div>============</div>
<div><br>
</div>
<div>Online: [ drbd01 drbd02 ]</div>
<div><br>
</div>
<div> Master/Slave Set: ms_drbd</div>
<div> Masters: [ drbd01 ]</div>
<div> Slaves: [ drbd02 ]</div>
<div> Resource Group: IP-AND-FS</div>
<div> fs_r1 (ocf::heartbeat:Filesystem): Started drbd01</div>
<div> VIRTUAL-IP (ocf::heartbeat:IPaddr): Started drbd01</div>
</div>
<div><br>
</div>
<div>I would like to migrate that group manually to another node which is slave. So i type in: crm resource migrate IP-AND-FS drbd02</div>
<div><br>
</div>
<div>after that configuration include additional line:</div>
<div><br>
</div>
<div>location cli-prefer-IP-AND-FS IP-AND-FS \</div>
<div> rule $id="cli-prefer-rule-IP-AND-FS" inf: #uname eq drbd02</div>
<div><br>
</div>
<div>and in logs i see:<br>
<br>
<div>Jan 23 11:30:12 drbd02 cibadmin: [1126]: info: Invoked: cibadmin -Ql -o resources</div>
<div>Jan 23 11:30:12 drbd02 cibadmin: [1129]: info: Invoked: cibadmin -Ql -o nodes</div>
<div>Jan 23 11:30:12 drbd02 cibadmin: [1131]: info: Invoked: cibadmin -Ql -o resources</div>
<div>Jan 23 11:30:12 drbd02 cibadmin: [1133]: info: Invoked: cibadmin -Ql -o nodes</div>
<div>Jan 23 11:30:12 drbd02 cibadmin: [1135]: info: Invoked: cibadmin -Ql -o resources</div>
<div>Jan 23 11:30:12 drbd02 cibadmin: [1137]: info: Invoked: cibadmin -Ql -o nodes</div>
<div>Jan 23 11:30:14 drbd02 cibadmin: [1166]: info: Invoked: cibadmin -Ql -o resources</div>
<div>Jan 23 11:30:14 drbd02 cibadmin: [1168]: info: Invoked: cibadmin -Ql -o nodes</div>
<div>Jan 23 11:30:14 drbd02 cibadmin: [1170]: info: Invoked: cibadmin -Ql -o resources</div>
<div>Jan 23 11:30:14 drbd02 cibadmin: [1172]: info: Invoked: cibadmin -Ql -o nodes</div>
<div>Jan 23 11:30:16 drbd02 cibadmin: [1174]: info: Invoked: cibadmin -Ql -o resources</div>
<div>Jan 23 11:30:16 drbd02 cibadmin: [1176]: info: Invoked: cibadmin -Ql -o nodes</div>
<div>Jan 23 11:30:16 drbd02 cibadmin: [1178]: info: Invoked: cibadmin -Ql -o resources</div>
<div>Jan 23 11:30:16 drbd02 cibadmin: [1180]: info: Invoked: cibadmin -Ql -o nodes</div>
<div>Jan 23 11:30:40 drbd02 cibadmin: [1211]: info: Invoked: cibadmin -Ql -o nodes</div>
<div>Jan 23 11:30:40 drbd02 crm_resource: [1213]: info: Invoked: crm_resource -M -r IP-AND-FS --node=drbd02</div>
<div>Jan 23 11:30:40 drbd02 cib: [1214]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-73.raw</div>
<div>Jan 23 11:30:40 drbd02 cib: [1214]: info: write_cib_contents: Wrote version 0.225.0 of the CIB to disk (digest: 166251193cbe1e0b9314ab07358accca)</div>
<div>Jan 23 11:30:40 drbd02 cib: [1214]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.tk72Ft (digest: /var/lib/heartbeat/crm/cib.hF2UsS)</div>
<div>Jan 23 11:30:44 drbd02 cib: [30098]: info: cib_stats: Processed 153 operations (1437.00us average, 0% utilization) in the last 10min</div>
<div><br>
</div>
<div>but nothing happened resource group is still active on drbd01 node, as well as there was no new master promotion. </div>
<div><br>
</div>
<div>Shouldn't pacemaker automatically promote second node to master and move my resource group ? </div>
<div><br>
</div>
<div><br>
</div>
<div>Below is my test configuration, i will be appreciate for help:</div>
<div><br>
</div>
<div>
<div>crm(live)# configure show</div>
<div>node drbd01 \</div>
<div> attributes standby="off"</div>
<div>node drbd02 \</div>
<div> attributes standby="off"</div>
<div>primitive VIRTUAL-IP ocf:heartbeat:IPaddr \</div>
<div> params ip="10.11.11.111"</div>
<div>primitive drbd ocf:linbit:drbd \</div>
<div> params drbd_resource="r1" \</div>
<div> op start interval="0" timeout="240" \</div>
<div> op stop interval="0" timeout="100" \</div>
<div> op monitor interval="59s" role="Master" timeout="30s" \</div>
<div> op monitor interval="60s" role="Slave" timeout="30s"</div>
<div>primitive fs_r1 ocf:heartbeat:Filesystem \</div>
<div> params device="/dev/drbd1" directory="/mnt" fstype="ext3" \</div>
<div> op start interval="0" timeout="60" \</div>
<div> op stop interval="0" timeout="120" \</div>
<div> meta allow-migrate="true"</div>
<div>group IP-AND-FS fs_r1 VIRTUAL-IP \</div>
<div> meta target-role="Started"</div>
<div>ms ms_drbd drbd \</div>
<div> meta master-node-max="1" clone-max="2" clone-node-max="1" globally-unique="false" notify="true" target-role="Master"</div>
<div>location cli-prefer-IP-AND-FS IP-AND-FS \</div>
<div> rule $id="cli-prefer-rule-IP-AND-FS" inf: #uname eq drbd02</div>
<div>colocation FS_WITH_DRBD inf: IP-AND-FS ms_drbd:Master</div>
<div>order DRBD_BEF_FS inf: ms_drbd:promote IP-AND-FS:start</div>
<div>property $id="cib-bootstrap-options" \</div>
<div> dc-version="1.0.9-74392a28b7f31d7ddc86689598bd23114f58978b" \</div>
<div> cluster-infrastructure="openais" \</div>
<div> expected-quorum-votes="2" \</div>
<div> stonith-enabled="false" \</div>
<div> no-quorum-policy="ignore" \</div>
<div> last-lrm-refresh="1358868655" \</div>
<div> default-resource-stickiness="1"</div>
<div><br>
</div>
<div>Regards</div>
<div>Andrew</div>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">
Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
</blockquote>
</div>
<span><font color="#888888"><br>
<br clear="all">
<br>
-- <br>
esta es mi vida e me la vivo hasta que dios quiera </font></span><br>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">
Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div></div></div>
</div>
</div>
<br>_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br></blockquote></div><br></div></div>