<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;
mso-fareast-language:EN-US;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:2.0cm 42.5pt 2.0cm 3.0cm;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:293339260;
mso-list-type:hybrid;
mso-list-template-ids:1394786226 68747279 68747289 68747291 68747279 68747289 68747291 68747279 68747289 68747291;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1
{mso-list-id:1201170041;
mso-list-type:hybrid;
mso-list-template-ids:274220430 68747279 68747289 68747291 68747279 68747289 68747291 68747279 68747289 68747291;}
@list l1:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l2
{mso-list-id:1344553020;
mso-list-type:hybrid;
mso-list-template-ids:-1934567004 68747279 68747289 68747291 68747279 68747289 68747291 68747279 68747289 68747291;}
@list l2:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l2:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l2:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l2:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l2:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l2:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l2:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l2:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l2:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l3
{mso-list-id:1569072551;
mso-list-type:hybrid;
mso-list-template-ids:-1603250850 68747279 68747289 68747291 68747279 68747289 68747291 68747279 68747289 68747291;}
@list l3:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l3:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l3:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l3:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l3:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l3:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l3:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l3:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l3:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=RU link="#0563C1" vlink="#954F72" style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span lang=EN-US>Hi All,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>We use Pacemaker in specific scenario, where complex network environment, including VLANs, IPs and routes managed by external system and integrated by glue code by:<o:p></o:p></span></p><ol style='margin-top:0cm' start=1 type=1><li class=MsoListParagraph style='margin-left:0cm;mso-list:l0 level1 lfo1'><span lang=EN-US>Load CIB database config section with </span><span lang=EN-US style='font-family:"Courier New"'>cibadmin –query --scope=configuration</span><span lang=EN-US><o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l0 level1 lfo1'><span lang=EN-US>Add/delete prototypes and constraints<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l0 level1 lfo1'><span lang=EN-US>Apply the new config by </span><span lang=EN-US style='font-family:"Courier New"'>cibadmin –replace --scope=configuration --xml-pip --sync-call</span><span lang=EN-US><o:p></o:p></span></li></ol><p class=MsoNormal><span lang=EN-US>CIB taken from stdout, new cib load to by stdin, all done by Python code<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>All types handed with standard ocf:heartbeat resource scripts.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>VLANs defined as clones to ensure it is up on all nodes.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Then, order constraints given to start IP after vlan-clone (to ensure VLAN exists), then route after proper IP.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>This works very good on mass-create, but we got some problems on mass-delete.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>For my understanding of Pacemaker architecture and behavior: if it got the new config, it recalculates resource allocation, build up the target map with respect to [co]location constraints and then schedule changes with respect to order constraints. So, if we delete at once VLANS, IPs and routes, we also have to delete its constraints. Then, the scheduling of resource stop will NOT take order constraints from OLD config into consideration. Then, all the stops for VLAN, IPs and routes will start in random order. However:<o:p></o:p></span></p><ol style='margin-top:0cm' start=1 type=1><li class=MsoListParagraph style='margin-left:0cm;mso-list:l2 level1 lfo2'><span lang=EN-US>If VLAN deletes (stops) it also deletes all IPs, bound to the interface. And all routes.<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l2 level1 lfo2'><span lang=EN-US>Then, IP resource trying to remove IP address which already deleted, and then files. Moreover, as scripts run in parallel, it may see IP active when it checks, but there already no IP when it tries to delete. As it failed, it left as an orphan (stopped, blocked) and only be clear with cleanup command. And this also ruins future CIB updates.<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l2 level1 lfo2'><span lang=EN-US>About same logic between IP and routes.<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l2 level1 lfo2'><span lang=EN-US><o:p> </o:p></span></li></ol><p class=MsoNormal><span lang=EN-US>After realizing this, I have changed the logic, use two stages:<o:p></o:p></span></p><ol style='margin-top:0cm' start=1 type=1><li class=MsoListParagraph style='margin-left:0cm;mso-list:l3 level1 lfo3'><span lang=EN-US>Read CIB<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l3 level1 lfo3'><span lang=EN-US>Disable all resources to delete (setting target_role=Stopped) and send it with cibadmin<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l3 level1 lfo3'><span lang=EN-US>Delete all the resources from CIB and send it with cibadmin<o:p></o:p></span></li></ol><p class=MsoNormal><span lang=EN-US>My idea was that Pacemaker will plan and do resource shutdown at step 2 with respect to order constraints which are still in the config. And then, it is safe to delete already stopped resources.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>But I see the same troubles with this two-stage approach. Sometimes some resources fail to stop because referenced entity is already deleted.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>It seems like one of two:<o:p></o:p></span></p><ol style='margin-top:0cm' start=1 type=1><li class=MsoListParagraph style='margin-left:0cm;mso-list:l1 level1 lfo4'><span lang=EN-US>Pacemaker does not respect order constraints when we put the new config section directly<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l1 level1 lfo4'><span lang=EN-US>I misunderstand </span><span lang=EN-US style='font-family:"Courier New"'>--sync-call </span><span lang=EN-US>cibadmin option, and it won’t wait for new config really applied and returns immediately, therefore delete starts before all stops compete. I did not find any explanation, and my guess was it should wait for changes applied by pacemaker, but I am not sure.<o:p></o:p></span></li></ol><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>I need an advice about this situation and more information about –sync-call option. Is it right approach or I need extra delay? Or wait for everything stop by request state once and once?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>I will be very grateful for any ideas or information!<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Sincerely,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Alex<o:p></o:p></span></p></div></body></html>