<div class="zcontentRow"> <p>adding "sleep 5" before return in the stop func fixed the issue. so I suspect there must be concurrency bug somewhere in the code. just FYI.<br></p><p><br></p><div class="zMailSign"><p style="font-size: 7.0px; line-height: normal; widows: 1;"><span style="font-size:10px;color:#58595B;font-family:微软雅黑;font-size: 7.0px;"></span></p><span style="line-height: normal; widows: 1; font-size: 7.0px;;color:#58595b;font-size:10px"></span></div><div class="zMailFrom"></div><div><div class="zhistoryRow" style="display:block"><div class="zhistoryDes" style="width: 100%; height: 28px; line-height: 28px; background-color: #E0E5E9; color: #1388FF; text-align: center;" language-data="HistoryOrgTxt">原始邮件</div><div id="zwriteHistoryContainer"><div class="control-group zhistoryPanel"><div class="zhistoryHeader" style="padding: 8px; background-color: #F5F6F8;"><div><strong language-data="HistorySenderTxt">发件人：</strong><span class="zreadUserName"> ＜kgaillot@redhat.com＞;</span></div><div><strong language-data="HistoryTOTxt">收件人：</strong><span class="zreadUserName" style="display: inline-block;">何海龙10164561;</span></div><div><strong language-data="HistoryCCTxt">抄送人：</strong><span class="zreadUserName" style="display: inline-block;"> ＜users@clusterlabs.org＞;</span></div><div><strong language-data="HistoryDateTxt">日 期 ：</strong><span class="">2017年02月15日 23:22</span></div><div><strong language-data="HistorySubjectTxt">主 题 ：</strong><span class="zreadTitle"><strong>Re: 答复: Re: 答复: Re: [ClusterLabs] clone resource not get restarted on fail</strong></span></div></div><p class="zhistoryContent"><br></p><div>On&nbsp;02/15/2017&nbsp;03:57&nbsp;AM,&nbsp;he.hailong5@zte.com.cn&nbsp;wrote:<br>＞&nbsp;I&nbsp;just&nbsp;tried&nbsp;using&nbsp;colocation,&nbsp;it&nbsp;dosen't&nbsp;work.<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;I&nbsp;failed&nbsp;the&nbsp;node&nbsp;paas-controller-3,&nbsp;but&nbsp;sdclient_vip&nbsp;didn't&nbsp;get&nbsp;moved:<br><br>The&nbsp;colocation&nbsp;would&nbsp;work,&nbsp;but&nbsp;the&nbsp;problem&nbsp;you're&nbsp;having&nbsp;with&nbsp;router&nbsp;and<br>apigateway&nbsp;is&nbsp;preventing&nbsp;it&nbsp;from&nbsp;getting&nbsp;that&nbsp;far.&nbsp;In&nbsp;other&nbsp;words,<br>router&nbsp;and&nbsp;apigateway&nbsp;are&nbsp;still&nbsp;running&nbsp;on&nbsp;the&nbsp;node&nbsp;(they&nbsp;have&nbsp;not&nbsp;been<br>successfully&nbsp;stopped),&nbsp;so&nbsp;the&nbsp;colocation&nbsp;is&nbsp;still&nbsp;valid.<br><br>I&nbsp;suspect&nbsp;that&nbsp;the&nbsp;return&nbsp;codes&nbsp;from&nbsp;your&nbsp;custom&nbsp;resource&nbsp;agents&nbsp;may&nbsp;be<br>the&nbsp;issue.&nbsp;Make&nbsp;sure&nbsp;that&nbsp;your&nbsp;agents&nbsp;conform&nbsp;to&nbsp;these&nbsp;guidelines:<br><br>http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#ap-ocf<br><br>In&nbsp;particular,&nbsp;"start"&nbsp;should&nbsp;not&nbsp;return&nbsp;until&nbsp;a&nbsp;monitor&nbsp;operation&nbsp;would<br>return&nbsp;success,&nbsp;"stop"&nbsp;should&nbsp;not&nbsp;return&nbsp;until&nbsp;a&nbsp;monitor&nbsp;would&nbsp;return<br>"not&nbsp;running",&nbsp;and&nbsp;"monitor"&nbsp;should&nbsp;return&nbsp;"not&nbsp;running"&nbsp;if&nbsp;called&nbsp;on&nbsp;a<br>host&nbsp;where&nbsp;the&nbsp;service&nbsp;hasn't&nbsp;started&nbsp;yet.&nbsp;Be&nbsp;sure&nbsp;you&nbsp;are&nbsp;returning&nbsp;the<br>proper&nbsp;OCF_*&nbsp;codes&nbsp;according&nbsp;to&nbsp;the&nbsp;table&nbsp;in&nbsp;the&nbsp;link&nbsp;above.<br><br>If&nbsp;the&nbsp;documentation&nbsp;is&nbsp;unclear,&nbsp;please&nbsp;ask&nbsp;here&nbsp;about&nbsp;anything&nbsp;you&nbsp;are<br>unsure&nbsp;of.<br><br>＞&nbsp;<br>＞&nbsp;Online:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;paas-controller-3&nbsp;]<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;router_vip&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-1&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;sdclient_vip&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-3&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;apigateway_vip&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-2&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;sdclient_rep&nbsp;[sdclient]<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Stopped:&nbsp;[&nbsp;paas-controller-3&nbsp;]<br>＞&nbsp;<br>＞&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;router_rep&nbsp;[router]<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;router&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(ocf::heartbeat:router):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started<br>＞&nbsp;paas-controller-3&nbsp;FAILED&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>＞&nbsp;<br>＞&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;apigateway_rep&nbsp;[apigateway]<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;apigateway&nbsp;(ocf::heartbeat:apigateway):&nbsp;&nbsp;&nbsp;&nbsp;Started<br>＞&nbsp;paas-controller-3&nbsp;FAILED&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;here&nbsp;is&nbsp;the&nbsp;configuration:<br>＞&nbsp;<br>＞&nbsp;＞crm&nbsp;configure&nbsp;show<br>＞&nbsp;<br>＞&nbsp;node&nbsp;$id="336855579"&nbsp;paas-controller-1<br>＞&nbsp;<br>＞&nbsp;node&nbsp;$id="336855580"&nbsp;paas-controller-2<br>＞&nbsp;<br>＞&nbsp;node&nbsp;$id="336855581"&nbsp;paas-controller-3<br>＞&nbsp;<br>＞&nbsp;primitive&nbsp;apigateway&nbsp;ocf:heartbeat:apigateway&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;interval="2s"&nbsp;timeout="20s"&nbsp;on-fail="restart"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="200s"&nbsp;on-fail="restart"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="9999h"&nbsp;on-fail="restart"<br>＞&nbsp;<br>＞&nbsp;primitive&nbsp;apigateway_vip&nbsp;ocf:heartbeat:IPaddr2&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;params&nbsp;ip="20.20.2.7"&nbsp;cidr_netmask="24"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;timeout="20s"&nbsp;interval="2s"&nbsp;depth="0"<br>＞&nbsp;<br>＞&nbsp;primitive&nbsp;router&nbsp;ocf:heartbeat:router&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;interval="2s"&nbsp;timeout="20s"&nbsp;on-fail="restart"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="200s"&nbsp;on-fail="restart"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="9999h"&nbsp;on-fail="restart"<br>＞&nbsp;<br>＞&nbsp;primitive&nbsp;router_vip&nbsp;ocf:heartbeat:IPaddr2&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;params&nbsp;ip="10.10.1.7"&nbsp;cidr_netmask="24"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;timeout="20s"&nbsp;interval="2s"&nbsp;depth="0"<br>＞&nbsp;<br>＞&nbsp;primitive&nbsp;sdclient&nbsp;ocf:heartbeat:sdclient&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;interval="2s"&nbsp;timeout="20s"&nbsp;on-fail="restart"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="200s"&nbsp;on-fail="restart"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="9999h"&nbsp;on-fail="restart"<br>＞&nbsp;<br>＞&nbsp;primitive&nbsp;sdclient_vip&nbsp;ocf:heartbeat:IPaddr2&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;params&nbsp;ip="10.10.1.8"&nbsp;cidr_netmask="24"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;timeout="20s"&nbsp;interval="2s"&nbsp;depth="0"<br>＞&nbsp;<br>＞&nbsp;clone&nbsp;apigateway_rep&nbsp;apigateway<br>＞&nbsp;<br>＞&nbsp;clone&nbsp;router_rep&nbsp;router<br>＞&nbsp;<br>＞&nbsp;clone&nbsp;sdclient_rep&nbsp;sdclient<br>＞&nbsp;<br>＞&nbsp;colocation&nbsp;apigateway_colo&nbsp;+inf:&nbsp;apigateway_vip&nbsp;apigateway_rep:Started<br>＞&nbsp;<br>＞&nbsp;colocation&nbsp;router_colo&nbsp;+inf:&nbsp;router_vip&nbsp;router_rep:Started<br>＞&nbsp;<br>＞&nbsp;colocation&nbsp;sdclient_colo&nbsp;+inf:&nbsp;sdclient_vip&nbsp;sdclient_rep:Started<br>＞&nbsp;<br>＞&nbsp;property&nbsp;$id="cib-bootstrap-options"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;dc-version="1.1.10-42f2063"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;cluster-infrastructure="corosync"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;stonith-enabled="false"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;no-quorum-policy="stop"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;start-failure-is-fatal="false"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;last-lrm-refresh="1486981647"<br>＞&nbsp;<br>＞&nbsp;op_defaults&nbsp;$id="op_defaults-options"&nbsp;\<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;on-fail="restart"<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;原始邮件<br>＞&nbsp;*发件人：*何海龙10164561<br>＞&nbsp;*收件人：*＜kgaillot@redhat.com＞;<br>＞&nbsp;*抄送人：*＜users@clusterlabs.org＞;<br>＞&nbsp;*日&nbsp;期&nbsp;：*2017年02月15日&nbsp;10:54<br>＞&nbsp;*主&nbsp;题&nbsp;：**答复:&nbsp;Re:&nbsp;答复:&nbsp;Re:&nbsp;[ClusterLabs]&nbsp;clone&nbsp;resource&nbsp;not&nbsp;get<br>＞&nbsp;restarted&nbsp;on&nbsp;fail*<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;Is&nbsp;there&nbsp;a&nbsp;reason&nbsp;not&nbsp;to&nbsp;use&nbsp;a&nbsp;colocation&nbsp;constraint&nbsp;instead?&nbsp;If&nbsp;X_vip<br>＞&nbsp;is&nbsp;colocated&nbsp;with&nbsp;X,&nbsp;it&nbsp;will&nbsp;be&nbsp;moved&nbsp;if&nbsp;X&nbsp;fails.<br>＞&nbsp;<br>＞&nbsp;[hhl]:&nbsp;the&nbsp;movement&nbsp;should&nbsp;take&nbsp;place&nbsp;as&nbsp;well&nbsp;if&nbsp;X&nbsp;stopped&nbsp;(the&nbsp;start&nbsp;is<br>＞&nbsp;on-going).&nbsp;I&nbsp;don't&nbsp;know&nbsp;if&nbsp;the&nbsp;colocation&nbsp;would&nbsp;satisfy&nbsp;this&nbsp;requirement.<br>＞&nbsp;<br>＞&nbsp;I&nbsp;don't&nbsp;see&nbsp;any&nbsp;reason&nbsp;in&nbsp;your&nbsp;configuration&nbsp;why&nbsp;the&nbsp;services&nbsp;wouldn't<br>＞&nbsp;be&nbsp;restarted.&nbsp;It's&nbsp;possible&nbsp;the&nbsp;cluster&nbsp;tried&nbsp;to&nbsp;restart&nbsp;the&nbsp;service,<br>＞&nbsp;but&nbsp;the&nbsp;stop&nbsp;action&nbsp;failed.&nbsp;Since&nbsp;you&nbsp;have&nbsp;stonith&nbsp;disabled,&nbsp;the&nbsp;cluster<br>＞&nbsp;can't&nbsp;recover&nbsp;from&nbsp;a&nbsp;failed&nbsp;stop&nbsp;action.<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;[hhl]:&nbsp;the&nbsp;ocf&nbsp;logs&nbsp;showed&nbsp;the&nbsp;pacemaker&nbsp;never&nbsp;entered&nbsp;the&nbsp;stop&nbsp;function<br>＞&nbsp;in&nbsp;this&nbsp;case.<br>＞&nbsp;<br>＞&nbsp;Is&nbsp;there&nbsp;a&nbsp;reason&nbsp;you&nbsp;disabled&nbsp;quorum?&nbsp;With&nbsp;3&nbsp;nodes,&nbsp;if&nbsp;they&nbsp;get&nbsp;split<br>＞&nbsp;into&nbsp;groups&nbsp;of&nbsp;1&nbsp;node&nbsp;and&nbsp;2&nbsp;nodes,&nbsp;quorum&nbsp;is&nbsp;what&nbsp;keeps&nbsp;the&nbsp;groups&nbsp;from<br>＞&nbsp;both&nbsp;starting&nbsp;all&nbsp;resources.<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;[hhl]:&nbsp;I&nbsp;enabled&nbsp;the&nbsp;quorum&nbsp;and&nbsp;had&nbsp;a&nbsp;retry,&nbsp;the&nbsp;same&nbsp;happens.<br>＞&nbsp;<br>＞&nbsp;b.t.w,&nbsp;I&nbsp;repeat&nbsp;sevaral&nbsp;times&nbsp;today,&nbsp;and&nbsp;found&nbsp;when&nbsp;I&nbsp;trigger&nbsp;the<br>＞&nbsp;condition&nbsp;on&nbsp;one&nbsp;node&nbsp;that&nbsp;would&nbsp;fail&nbsp;all&nbsp;the&nbsp;clone&nbsp;resources,&nbsp;only&nbsp;one<br>＞&nbsp;would&nbsp;get&nbsp;restared,&nbsp;the&nbsp;other&nbsp;two&nbsp;would&nbsp;fail&nbsp;to&nbsp;restart.<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;＞&nbsp;trigger&nbsp;the&nbsp;failure&nbsp;conditon&nbsp;on&nbsp;paas-controller-1<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;Online:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;paas-controller-3&nbsp;]<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;router_vip&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-2&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;sdclient_vip&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-3&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;apigateway_vip&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-3&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;sdclient_rep&nbsp;[sdclient]<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-2&nbsp;paas-controller-3&nbsp;]<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Stopped:&nbsp;[&nbsp;paas-controller-1&nbsp;]<br>＞&nbsp;<br>＞&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;router_rep&nbsp;[router]<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;router&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(ocf::heartbeat:router):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started<br>＞&nbsp;paas-controller-1&nbsp;FAILED&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-2&nbsp;paas-controller-3&nbsp;]<br>＞&nbsp;<br>＞&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;apigateway_rep&nbsp;[apigateway]<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;apigateway&nbsp;(ocf::heartbeat:apigateway):&nbsp;&nbsp;&nbsp;&nbsp;Started<br>＞&nbsp;paas-controller-1&nbsp;FAILED&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-2&nbsp;paas-controller-3&nbsp;]<br>＞&nbsp;<br>＞&nbsp;&nbsp;<br>＞&nbsp;<br>＞&nbsp;＞&nbsp;trigger&nbsp;the&nbsp;failure&nbsp;conditon&nbsp;on&nbsp;paas-controller-3<br>＞&nbsp;<br>＞&nbsp;Online:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;paas-controller-3&nbsp;]<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;router_vip&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-2&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;sdclient_vip&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-3&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;apigateway_vip&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-3&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;sdclient_rep&nbsp;[sdclient]<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sdclient&nbsp;&nbsp;&nbsp;(ocf::heartbeat:sdclient):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started<br>＞&nbsp;paas-controller-3&nbsp;FAILED&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>＞&nbsp;<br>＞&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;router_rep&nbsp;[router]<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Stopped:&nbsp;[&nbsp;paas-controller-3&nbsp;]<br>＞&nbsp;<br>＞&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;apigateway_rep&nbsp;[apigateway]<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;apigateway&nbsp;(ocf::heartbeat:apigateway):&nbsp;&nbsp;&nbsp;&nbsp;Started<br>＞&nbsp;paas-controller-3&nbsp;FAILED&nbsp;<br>＞&nbsp;<br>＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;*发件人：*＜kgaillot@redhat.com＞;<br>＞&nbsp;*收件人：*何海龙10164561;<br>＞&nbsp;*抄送人：*＜users@clusterlabs.org＞;<br>＞&nbsp;*日&nbsp;期&nbsp;：*2017年02月15日&nbsp;06:14<br>＞&nbsp;*主&nbsp;题&nbsp;：**Re:&nbsp;答复:&nbsp;Re:&nbsp;[ClusterLabs]&nbsp;clone&nbsp;resource&nbsp;not&nbsp;get&nbsp;restarted<br>＞&nbsp;on&nbsp;fail*<br>＞&nbsp;<br>＞&nbsp;<br>＞&nbsp;On&nbsp;02/13/2017&nbsp;07:08&nbsp;PM,&nbsp;he.hailong5@zte.com.cn&nbsp;wrote:<br>＞&nbsp;＞&nbsp;Hi,<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;crm&nbsp;configure&nbsp;show<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;+&nbsp;crm&nbsp;configure&nbsp;show<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;node&nbsp;$id="336855579"&nbsp;paas-controller-1<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;node&nbsp;$id="336855580"&nbsp;paas-controller-2<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;node&nbsp;$id="336855581"&nbsp;paas-controller-3<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;primitive&nbsp;apigateway&nbsp;ocf:heartbeat:apigateway&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;interval="2s"&nbsp;timeout="20s"&nbsp;on-fail="restart"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="200s"&nbsp;on-fail="restart"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="9999h"&nbsp;on-fail="restart"<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;primitive&nbsp;apigateway_vip&nbsp;ocf:heartbeat:IPaddr2&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;params&nbsp;ip="20.20.2.7"&nbsp;cidr_netmask="24"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;timeout="20s"&nbsp;interval="2s"&nbsp;depth="0"<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;primitive&nbsp;router&nbsp;ocf:heartbeat:router&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;interval="2s"&nbsp;timeout="20s"&nbsp;on-fail="restart"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="200s"&nbsp;on-fail="restart"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="9999h"&nbsp;on-fail="restart"<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;primitive&nbsp;router_vip&nbsp;ocf:heartbeat:IPaddr2&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;params&nbsp;ip="10.10.1.7"&nbsp;cidr_netmask="24"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;timeout="20s"&nbsp;interval="2s"&nbsp;depth="0"<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;primitive&nbsp;sdclient&nbsp;ocf:heartbeat:sdclient&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;interval="2s"&nbsp;timeout="20s"&nbsp;on-fail="restart"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="200s"&nbsp;on-fail="restart"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="9999h"&nbsp;on-fail="restart"<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;primitive&nbsp;sdclient_vip&nbsp;ocf:heartbeat:IPaddr2&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;params&nbsp;ip="10.10.1.8"&nbsp;cidr_netmask="24"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;timeout="20s"&nbsp;interval="2s"&nbsp;depth="0"<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;clone&nbsp;apigateway_rep&nbsp;apigateway<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;clone&nbsp;router_rep&nbsp;router<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;clone&nbsp;sdclient_rep&nbsp;sdclient<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;location&nbsp;apigateway_loc&nbsp;apigateway_vip&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;rule&nbsp;$id="apigateway_loc-rule"&nbsp;+inf:&nbsp;apigateway_workable&nbsp;eq&nbsp;1<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;location&nbsp;router_loc&nbsp;router_vip&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;rule&nbsp;$id="router_loc-rule"&nbsp;+inf:&nbsp;router_workable&nbsp;eq&nbsp;1<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;location&nbsp;sdclient_loc&nbsp;sdclient_vip&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;rule&nbsp;$id="sdclient_loc-rule"&nbsp;+inf:&nbsp;sdclient_workable&nbsp;eq&nbsp;1<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;property&nbsp;$id="cib-bootstrap-options"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;dc-version="1.1.10-42f2063"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;cluster-infrastructure="corosync"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;stonith-enabled="false"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;no-quorum-policy="ignore"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;start-failure-is-fatal="false"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;last-lrm-refresh="1486981647"<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;op_defaults&nbsp;$id="op_defaults-options"&nbsp;\<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;on-fail="restart"<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;-------------------------------------------------------------------------------------------------<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;and&nbsp;B.T.W,&nbsp;I&nbsp;am&nbsp;using&nbsp;"crm_attribute&nbsp;-N&nbsp;$HOSTNAME&nbsp;-q&nbsp;-l&nbsp;reboot&nbsp;--name<br>＞&nbsp;＞&nbsp;＜prefix＞_workable&nbsp;-v&nbsp;＜1&nbsp;or&nbsp;0＞"&nbsp;in&nbsp;the&nbsp;monitor&nbsp;to&nbsp;update&nbsp;the<br>＞&nbsp;＞&nbsp;transient&nbsp;attributes,&nbsp;which&nbsp;control&nbsp;the&nbsp;vip&nbsp;location.<br>＞&nbsp;<br>＞&nbsp;Is&nbsp;there&nbsp;a&nbsp;reason&nbsp;not&nbsp;to&nbsp;use&nbsp;a&nbsp;colocation&nbsp;constraint&nbsp;instead?&nbsp;If&nbsp;X_vip<br>＞&nbsp;is&nbsp;colocated&nbsp;with&nbsp;X,&nbsp;it&nbsp;will&nbsp;be&nbsp;moved&nbsp;if&nbsp;X&nbsp;fails.<br>＞&nbsp;<br>＞&nbsp;I&nbsp;don't&nbsp;see&nbsp;any&nbsp;reason&nbsp;in&nbsp;your&nbsp;configuration&nbsp;why&nbsp;the&nbsp;services&nbsp;wouldn't<br>＞&nbsp;be&nbsp;restarted.&nbsp;It's&nbsp;possible&nbsp;the&nbsp;cluster&nbsp;tried&nbsp;to&nbsp;restart&nbsp;the&nbsp;service,<br>＞&nbsp;but&nbsp;the&nbsp;stop&nbsp;action&nbsp;failed.&nbsp;Since&nbsp;you&nbsp;have&nbsp;stonith&nbsp;disabled,&nbsp;the&nbsp;cluster<br>＞&nbsp;can't&nbsp;recover&nbsp;from&nbsp;a&nbsp;failed&nbsp;stop&nbsp;action.<br>＞&nbsp;<br>＞&nbsp;Is&nbsp;there&nbsp;a&nbsp;reason&nbsp;you&nbsp;disabled&nbsp;quorum?&nbsp;With&nbsp;3&nbsp;nodes,&nbsp;if&nbsp;they&nbsp;get&nbsp;split<br>＞&nbsp;into&nbsp;groups&nbsp;of&nbsp;1&nbsp;node&nbsp;and&nbsp;2&nbsp;nodes,&nbsp;quorum&nbsp;is&nbsp;what&nbsp;keeps&nbsp;the&nbsp;groups&nbsp;from<br>＞&nbsp;both&nbsp;starting&nbsp;all&nbsp;resources.<br>＞&nbsp;<br>＞&nbsp;＞&nbsp;and&nbsp;also&nbsp;found,&nbsp;the&nbsp;vip&nbsp;resource&nbsp;won't&nbsp;get&nbsp;moved&nbsp;if&nbsp;the&nbsp;related&nbsp;clone<br>＞&nbsp;＞&nbsp;resource&nbsp;failed&nbsp;to&nbsp;restart.<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;原始邮件<br>＞&nbsp;＞&nbsp;*发件人：*＜kgaillot@redhat.com＞;<br>＞&nbsp;＞&nbsp;*收件人：*＜users@clusterlabs.org＞;<br>＞&nbsp;＞&nbsp;*日&nbsp;期&nbsp;：*2017年02月13日&nbsp;23:04<br>＞&nbsp;＞&nbsp;*主&nbsp;题&nbsp;：**Re:&nbsp;[ClusterLabs]&nbsp;clone&nbsp;resource&nbsp;not&nbsp;get&nbsp;restarted&nbsp;on&nbsp;fail*<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;On&nbsp;02/13/2017&nbsp;07:57&nbsp;AM,&nbsp;he.hailong5@zte.com.cn&nbsp;wrote:<br>＞&nbsp;＞&nbsp;＞&nbsp;Pacemaker&nbsp;1.1.10<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;Corosync&nbsp;2.3.3<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;this&nbsp;is&nbsp;a&nbsp;3&nbsp;nodes&nbsp;cluster&nbsp;configured&nbsp;with&nbsp;3&nbsp;clone&nbsp;resources,&nbsp;each<br>＞&nbsp;＞&nbsp;＞&nbsp;attached&nbsp;wih&nbsp;a&nbsp;vip&nbsp;resource&nbsp;of&nbsp;IPAddr2:<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;＞crm&nbsp;status<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;Online:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;paas-controller-3&nbsp;]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;router_vip&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-<br>＞&nbsp;controller-1&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;sdclient_vip&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-<br>＞&nbsp;controller-3&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;apigateway_vip&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-<br>＞&nbsp;controller-2&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;sdclient_rep&nbsp;[sdclient]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;paas-<br>＞&nbsp;controller-3&nbsp;]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;router_rep&nbsp;[router]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;paas-<br>＞&nbsp;controller-3&nbsp;]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;apigateway_rep&nbsp;[apigateway]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;paas-<br>＞&nbsp;controller-3&nbsp;]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;It&nbsp;is&nbsp;observed&nbsp;that&nbsp;sometimes&nbsp;the&nbsp;clone&nbsp;resource&nbsp;is&nbsp;stuck&nbsp;to&nbsp;monitor<br>＞&nbsp;＞&nbsp;＞&nbsp;when&nbsp;the&nbsp;service&nbsp;fails:<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;router_vip&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-<br>＞&nbsp;controller-1&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;sdclient_vip&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-<br>＞&nbsp;controller-2&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;apigateway_vip&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-<br>＞&nbsp;controller-3&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;sdclient_rep&nbsp;[sdclient]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Stopped:&nbsp;[&nbsp;paas-controller-3&nbsp;]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;router_rep&nbsp;[router]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;router&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(ocf::heartbeat:router):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started<br>＞&nbsp;＞&nbsp;＞&nbsp;paas-controller-3&nbsp;FAILED&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;apigateway_rep&nbsp;[apigateway]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;apigateway&nbsp;(ocf::heartbeat:apigateway):&nbsp;&nbsp;&nbsp;&nbsp;Started<br>＞&nbsp;＞&nbsp;＞&nbsp;paas-controller-3&nbsp;FAILED&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;in&nbsp;the&nbsp;example&nbsp;above.&nbsp;the&nbsp;sdclient_rep&nbsp;get&nbsp;restarted&nbsp;on&nbsp;node&nbsp;3,&nbsp;while<br>＞&nbsp;＞&nbsp;＞&nbsp;the&nbsp;other&nbsp;two&nbsp;hang&nbsp;at&nbsp;monitoring&nbsp;on&nbsp;node&nbsp;3,&nbsp;here&nbsp;are&nbsp;the&nbsp;ocf&nbsp;logs:<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;abnormal&nbsp;(apigateway_rep):<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:27:53&nbsp;[23586]===print_log&nbsp;test_monitor&nbsp;run_func&nbsp;main===<br>＞&nbsp;＞&nbsp;＞&nbsp;Starting&nbsp;health&nbsp;check.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:27:53&nbsp;[23586]===print_log&nbsp;test_monitor&nbsp;run_func&nbsp;main===<br>＞&nbsp;＞&nbsp;＞&nbsp;health&nbsp;check&nbsp;succeed.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:27:55&nbsp;[24010]===print_log&nbsp;test_monitor&nbsp;run_func&nbsp;main===<br>＞&nbsp;＞&nbsp;＞&nbsp;Starting&nbsp;health&nbsp;check.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:27:55&nbsp;[24010]===print_log&nbsp;test_monitor&nbsp;run_func&nbsp;main===<br>＞&nbsp;＞&nbsp;＞&nbsp;Failed:&nbsp;docker&nbsp;daemon&nbsp;is&nbsp;not&nbsp;running.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:27:57&nbsp;[24095]===print_log&nbsp;test_monitor&nbsp;run_func&nbsp;main===<br>＞&nbsp;＞&nbsp;＞&nbsp;Starting&nbsp;health&nbsp;check.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:27:57&nbsp;[24095]===print_log&nbsp;test_monitor&nbsp;run_func&nbsp;main===<br>＞&nbsp;＞&nbsp;＞&nbsp;Failed:&nbsp;docker&nbsp;daemon&nbsp;is&nbsp;not&nbsp;running.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:27:59&nbsp;[24159]===print_log&nbsp;test_monitor&nbsp;run_func&nbsp;main===<br>＞&nbsp;＞&nbsp;＞&nbsp;Starting&nbsp;health&nbsp;check.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:27:59&nbsp;[24159]===print_log&nbsp;test_monitor&nbsp;run_func&nbsp;main===<br>＞&nbsp;＞&nbsp;＞&nbsp;Failed:&nbsp;docker&nbsp;daemon&nbsp;is&nbsp;not&nbsp;running.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;normal&nbsp;(sdclient_rep):<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:27:52&nbsp;[23507]===print_log&nbsp;sdclient_monitor&nbsp;run_func<br>＞&nbsp;＞&nbsp;＞&nbsp;main===&nbsp;health&nbsp;check&nbsp;succeed.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:27:54&nbsp;[23630]===print_log&nbsp;sdclient_monitor&nbsp;run_func<br>＞&nbsp;＞&nbsp;＞&nbsp;main===&nbsp;Starting&nbsp;health&nbsp;check.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:27:54&nbsp;[23630]===print_log&nbsp;sdclient_monitor&nbsp;run_func<br>＞&nbsp;＞&nbsp;＞&nbsp;main===&nbsp;Failed:&nbsp;docker&nbsp;daemon&nbsp;is&nbsp;not&nbsp;running.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:27:55&nbsp;[23710]===print_log&nbsp;sdclient_stop&nbsp;run_func&nbsp;main===<br>＞&nbsp;＞&nbsp;＞&nbsp;Starting&nbsp;stop&nbsp;the&nbsp;container.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:27:55&nbsp;[23710]===print_log&nbsp;sdclient_stop&nbsp;run_func&nbsp;main===<br>＞&nbsp;＞&nbsp;＞&nbsp;docker&nbsp;daemon&nbsp;lost,&nbsp;pretend&nbsp;stop&nbsp;succeed.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:27:55&nbsp;[23763]===print_log&nbsp;sdclient_start&nbsp;run_func&nbsp;main===<br>＞&nbsp;＞&nbsp;＞&nbsp;Starting&nbsp;run&nbsp;the&nbsp;container.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:27:55&nbsp;[23763]===print_log&nbsp;sdclient_start&nbsp;run_func&nbsp;main===<br>＞&nbsp;＞&nbsp;＞&nbsp;docker&nbsp;daemon&nbsp;lost,&nbsp;try&nbsp;again&nbsp;in&nbsp;5&nbsp;secs.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:28:00&nbsp;[23763]===print_log&nbsp;sdclient_start&nbsp;run_func&nbsp;main===<br>＞&nbsp;＞&nbsp;＞&nbsp;docker&nbsp;daemon&nbsp;lost,&nbsp;try&nbsp;again&nbsp;in&nbsp;5&nbsp;secs.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;2017-02-13&nbsp;18:28:05&nbsp;[23763]===print_log&nbsp;sdclient_start&nbsp;run_func&nbsp;main===<br>＞&nbsp;＞&nbsp;＞&nbsp;docker&nbsp;daemon&nbsp;lost,&nbsp;try&nbsp;again&nbsp;in&nbsp;5&nbsp;secs.<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;If&nbsp;I&nbsp;disable&nbsp;2&nbsp;clone&nbsp;resource,&nbsp;the&nbsp;switch&nbsp;over&nbsp;test&nbsp;for&nbsp;one&nbsp;clone<br>＞&nbsp;＞&nbsp;＞&nbsp;resource&nbsp;works&nbsp;as&nbsp;expected:&nbsp;fail&nbsp;the&nbsp;service&nbsp;-＞&nbsp;monitor&nbsp;fails&nbsp;-＞&nbsp;stop<br>＞&nbsp;＞&nbsp;＞&nbsp;-＞&nbsp;start<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;Online:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;paas-controller-3&nbsp;]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;sdclient_vip&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-<br>＞&nbsp;controller-2&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;sdclient_rep&nbsp;[sdclient]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Stopped:&nbsp;[&nbsp;paas-controller-3&nbsp;]<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;＞&nbsp;what's&nbsp;the&nbsp;reason&nbsp;behind????&nbsp;<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;Can&nbsp;you&nbsp;show&nbsp;the&nbsp;configuration&nbsp;of&nbsp;the&nbsp;three&nbsp;clones,&nbsp;their&nbsp;operations,<br>＞&nbsp;＞&nbsp;and&nbsp;any&nbsp;constraints?<br>＞&nbsp;＞&nbsp;<br>＞&nbsp;＞&nbsp;Normally,&nbsp;the&nbsp;response&nbsp;is&nbsp;controlled&nbsp;by&nbsp;the&nbsp;monitor&nbsp;operation's&nbsp;on-fail<br>＞&nbsp;＞&nbsp;attribute&nbsp;(which&nbsp;defaults&nbsp;to&nbsp;restart).<br><br></div><p><br></p></div></div></div></div><p><br></p> </div>