[ClusterLabs] 答复: Re: 答复: Re: clone resource not get restarted on fail
Ken Gaillot
kgaillot at redhat.com
Wed Feb 15 10:21:50 EST 2017
On 02/15/2017 03:57 AM, he.hailong5 at zte.com.cn wrote:
> I just tried using colocation, it dosen't work.
>
>
> I failed the node paas-controller-3, but sdclient_vip didn't get moved:
The colocation would work, but the problem you're having with router and
apigateway is preventing it from getting that far. In other words,
router and apigateway are still running on the node (they have not been
successfully stopped), so the colocation is still valid.
I suspect that the return codes from your custom resource agents may be
the issue. Make sure that your agents conform to these guidelines:
http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#ap-ocf
In particular, "start" should not return until a monitor operation would
return success, "stop" should not return until a monitor would return
"not running", and "monitor" should return "not running" if called on a
host where the service hasn't started yet. Be sure you are returning the
proper OCF_* codes according to the table in the link above.
If the documentation is unclear, please ask here about anything you are
unsure of.
>
> Online: [ paas-controller-1 paas-controller-2 paas-controller-3 ]
>
>
> router_vip (ocf::heartbeat:IPaddr2): Started paas-controller-1
>
> sdclient_vip (ocf::heartbeat:IPaddr2): Started paas-controller-3
>
> apigateway_vip (ocf::heartbeat:IPaddr2): Started paas-controller-2
>
> Clone Set: sdclient_rep [sdclient]
>
> Started: [ paas-controller-1 paas-controller-2 ]
>
> Stopped: [ paas-controller-3 ]
>
> Clone Set: router_rep [router]
>
> router (ocf::heartbeat:router): Started
> paas-controller-3 FAILED
>
> Started: [ paas-controller-1 paas-controller-2 ]
>
> Clone Set: apigateway_rep [apigateway]
>
> apigateway (ocf::heartbeat:apigateway): Started
> paas-controller-3 FAILED
>
> Started: [ paas-controller-1 paas-controller-2 ]
>
>
> here is the configuration:
>
> ＞crm configure show
>
> node $id="336855579" paas-controller-1
>
> node $id="336855580" paas-controller-2
>
> node $id="336855581" paas-controller-3
>
> primitive apigateway ocf:heartbeat:apigateway \
>
> op monitor interval="2s" timeout="20s" on-fail="restart" \
>
> op stop interval="0" timeout="200s" on-fail="restart" \
>
> op start interval="0" timeout="9999h" on-fail="restart"
>
> primitive apigateway_vip ocf:heartbeat:IPaddr2 \
>
> params ip="20.20.2.7" cidr_netmask="24" \
>
> op start interval="0" timeout="20" \
>
> op stop interval="0" timeout="20" \
>
> op monitor timeout="20s" interval="2s" depth="0"
>
> primitive router ocf:heartbeat:router \
>
> op monitor interval="2s" timeout="20s" on-fail="restart" \
>
> op stop interval="0" timeout="200s" on-fail="restart" \
>
> op start interval="0" timeout="9999h" on-fail="restart"
>
> primitive router_vip ocf:heartbeat:IPaddr2 \
>
> params ip="10.10.1.7" cidr_netmask="24" \
>
> op start interval="0" timeout="20" \
>
> op stop interval="0" timeout="20" \
>
> op monitor timeout="20s" interval="2s" depth="0"
>
> primitive sdclient ocf:heartbeat:sdclient \
>
> op monitor interval="2s" timeout="20s" on-fail="restart" \
>
> op stop interval="0" timeout="200s" on-fail="restart" \
>
> op start interval="0" timeout="9999h" on-fail="restart"
>
> primitive sdclient_vip ocf:heartbeat:IPaddr2 \
>
> params ip="10.10.1.8" cidr_netmask="24" \
>
> op start interval="0" timeout="20" \
>
> op stop interval="0" timeout="20" \
>
> op monitor timeout="20s" interval="2s" depth="0"
>
> clone apigateway_rep apigateway
>
> clone router_rep router
>
> clone sdclient_rep sdclient
>
> colocation apigateway_colo +inf: apigateway_vip apigateway_rep:Started
>
> colocation router_colo +inf: router_vip router_rep:Started
>
> colocation sdclient_colo +inf: sdclient_vip sdclient_rep:Started
>
> property $id="cib-bootstrap-options" \
>
> dc-version="1.1.10-42f2063" \
>
> cluster-infrastructure="corosync" \
>
> stonith-enabled="false" \
>
> no-quorum-policy="stop" \
>
> start-failure-is-fatal="false" \
>
> last-lrm-refresh="1486981647"
>
> op_defaults $id="op_defaults-options" \
>
> on-fail="restart"
>
>
>
>
>
> 原始邮件
> *发件人：*何海龙10164561
> *收件人：*＜kgaillot at redhat.com＞;
> *抄送人：*＜users at clusterlabs.org＞;
> *日 期 ：*2017年02月15日 10:54
> *主 题 ：**答复: Re: 答复: Re: [ClusterLabs] clone resource not get
> restarted on fail*
>
>
> Is there a reason not to use a colocation constraint instead? If X_vip
> is colocated with X, it will be moved if X fails.
>
> [hhl]: the movement should take place as well if X stopped (the start is
> on-going). I don't know if the colocation would satisfy this requirement.
>
> I don't see any reason in your configuration why the services wouldn't
> be restarted. It's possible the cluster tried to restart the service,
> but the stop action failed. Since you have stonith disabled, the cluster
> can't recover from a failed stop action.
>
>
> [hhl]: the ocf logs showed the pacemaker never entered the stop function
> in this case.
>
> Is there a reason you disabled quorum? With 3 nodes, if they get split
> into groups of 1 node and 2 nodes, quorum is what keeps the groups from
> both starting all resources.
>
>
> [hhl]: I enabled the quorum and had a retry, the same happens.
>
> b.t.w, I repeat sevaral times today, and found when I trigger the
> condition on one node that would fail all the clone resources, only one
> would get restared, the other two would fail to restart.
>
>
> ＞ trigger the failure conditon on paas-controller-1
>
>
> Online: [ paas-controller-1 paas-controller-2 paas-controller-3 ]
>
>
> router_vip (ocf::heartbeat:IPaddr2): Started paas-controller-2
>
> sdclient_vip (ocf::heartbeat:IPaddr2): Started paas-controller-3
>
> apigateway_vip (ocf::heartbeat:IPaddr2): Started paas-controller-3
>
> Clone Set: sdclient_rep [sdclient]
>
> Started: [ paas-controller-2 paas-controller-3 ]
>
> Stopped: [ paas-controller-1 ]
>
> Clone Set: router_rep [router]
>
> router (ocf::heartbeat:router): Started
> paas-controller-1 FAILED
>
> Started: [ paas-controller-2 paas-controller-3 ]
>
> Clone Set: apigateway_rep [apigateway]
>
> apigateway (ocf::heartbeat:apigateway): Started
> paas-controller-1 FAILED
>
> Started: [ paas-controller-2 paas-controller-3 ]
>
>
>
> ＞ trigger the failure conditon on paas-controller-3
>
> Online: [ paas-controller-1 paas-controller-2 paas-controller-3 ]
>
>
> router_vip (ocf::heartbeat:IPaddr2): Started paas-controller-2
>
> sdclient_vip (ocf::heartbeat:IPaddr2): Started paas-controller-3
>
> apigateway_vip (ocf::heartbeat:IPaddr2): Started paas-controller-3
>
> Clone Set: sdclient_rep [sdclient]
>
> sdclient (ocf::heartbeat:sdclient): Started
> paas-controller-3 FAILED
>
> Started: [ paas-controller-1 paas-controller-2 ]
>
> Clone Set: router_rep [router]
>
> Started: [ paas-controller-1 paas-controller-2 ]
>
> Stopped: [ paas-controller-3 ]
>
> Clone Set: apigateway_rep [apigateway]
>
> apigateway (ocf::heartbeat:apigateway): Started
> paas-controller-3 FAILED
>
> Started: [ paas-controller-1 paas-controller-2 ]
>
>
>
>
>
>
> *发件人：*＜kgaillot at redhat.com＞;
> *收件人：*何海龙10164561;
> *抄送人：*＜users at clusterlabs.org＞;
> *日 期 ：*2017年02月15日 06:14
> *主 题 ：**Re: 答复: Re: [ClusterLabs] clone resource not get restarted
> on fail*
>
>
> On 02/13/2017 07:08 PM, he.hailong5 at zte.com.cn wrote:
> ＞ Hi,
> ＞
> ＞
> ＞ ＞ crm configure show
> ＞
> ＞ + crm configure show
> ＞
> ＞ node $id="336855579" paas-controller-1
> ＞
> ＞ node $id="336855580" paas-controller-2
> ＞
> ＞ node $id="336855581" paas-controller-3
> ＞
> ＞ primitive apigateway ocf:heartbeat:apigateway \
> ＞
> ＞ op monitor interval="2s" timeout="20s" on-fail="restart" \
> ＞
> ＞ op stop interval="0" timeout="200s" on-fail="restart" \
> ＞
> ＞ op start interval="0" timeout="9999h" on-fail="restart"
> ＞
> ＞ primitive apigateway_vip ocf:heartbeat:IPaddr2 \
> ＞
> ＞ params ip="20.20.2.7" cidr_netmask="24" \
> ＞
> ＞ op start interval="0" timeout="20" \
> ＞
> ＞ op stop interval="0" timeout="20" \
> ＞
> ＞ op monitor timeout="20s" interval="2s" depth="0"
> ＞
> ＞ primitive router ocf:heartbeat:router \
> ＞
> ＞ op monitor interval="2s" timeout="20s" on-fail="restart" \
> ＞
> ＞ op stop interval="0" timeout="200s" on-fail="restart" \
> ＞
> ＞ op start interval="0" timeout="9999h" on-fail="restart"
> ＞
> ＞ primitive router_vip ocf:heartbeat:IPaddr2 \
> ＞
> ＞ params ip="10.10.1.7" cidr_netmask="24" \
> ＞
> ＞ op start interval="0" timeout="20" \
> ＞
> ＞ op stop interval="0" timeout="20" \
> ＞
> ＞ op monitor timeout="20s" interval="2s" depth="0"
> ＞
> ＞ primitive sdclient ocf:heartbeat:sdclient \
> ＞
> ＞ op monitor interval="2s" timeout="20s" on-fail="restart" \
> ＞
> ＞ op stop interval="0" timeout="200s" on-fail="restart" \
> ＞
> ＞ op start interval="0" timeout="9999h" on-fail="restart"
> ＞
> ＞ primitive sdclient_vip ocf:heartbeat:IPaddr2 \
> ＞
> ＞ params ip="10.10.1.8" cidr_netmask="24" \
> ＞
> ＞ op start interval="0" timeout="20" \
> ＞
> ＞ op stop interval="0" timeout="20" \
> ＞
> ＞ op monitor timeout="20s" interval="2s" depth="0"
> ＞
> ＞ clone apigateway_rep apigateway
> ＞
> ＞ clone router_rep router
> ＞
> ＞ clone sdclient_rep sdclient
> ＞
> ＞ location apigateway_loc apigateway_vip \
> ＞
> ＞ rule $id="apigateway_loc-rule" +inf: apigateway_workable eq 1
> ＞
> ＞ location router_loc router_vip \
> ＞
> ＞ rule $id="router_loc-rule" +inf: router_workable eq 1
> ＞
> ＞ location sdclient_loc sdclient_vip \
> ＞
> ＞ rule $id="sdclient_loc-rule" +inf: sdclient_workable eq 1
> ＞
> ＞ property $id="cib-bootstrap-options" \
> ＞
> ＞ dc-version="1.1.10-42f2063" \
> ＞
> ＞ cluster-infrastructure="corosync" \
> ＞
> ＞ stonith-enabled="false" \
> ＞
> ＞ no-quorum-policy="ignore" \
> ＞
> ＞ start-failure-is-fatal="false" \
> ＞
> ＞ last-lrm-refresh="1486981647"
> ＞
> ＞ op_defaults $id="op_defaults-options" \
> ＞
> ＞ on-fail="restart"
> ＞
> ＞ -------------------------------------------------------------------------------------------------
> ＞
> ＞
> ＞ and B.T.W, I am using "crm_attribute -N $HOSTNAME -q -l reboot --name
> ＞ ＜prefix＞_workable -v ＜1 or 0＞" in the monitor to update the
> ＞ transient attributes, which control the vip location.
>
> Is there a reason not to use a colocation constraint instead? If X_vip
> is colocated with X, it will be moved if X fails.
>
> I don't see any reason in your configuration why the services wouldn't
> be restarted. It's possible the cluster tried to restart the service,
> but the stop action failed. Since you have stonith disabled, the cluster
> can't recover from a failed stop action.
>
> Is there a reason you disabled quorum? With 3 nodes, if they get split
> into groups of 1 node and 2 nodes, quorum is what keeps the groups from
> both starting all resources.
>
> ＞ and also found, the vip resource won't get moved if the related clone
> ＞ resource failed to restart.
> ＞
> ＞
> ＞ 原始邮件
> ＞ *发件人：*＜kgaillot at redhat.com＞;
> ＞ *收件人：*＜users at clusterlabs.org＞;
> ＞ *日 期 ：*2017年02月13日 23:04
> ＞ *主 题 ：**Re: [ClusterLabs] clone resource not get restarted on fail*
> ＞
> ＞
> ＞ On 02/13/2017 07:57 AM, he.hailong5 at zte.com.cn wrote:
> ＞ ＞ Pacemaker 1.1.10
> ＞ ＞
> ＞ ＞ Corosync 2.3.3
> ＞ ＞
> ＞ ＞
> ＞ ＞ this is a 3 nodes cluster configured with 3 clone resources, each
> ＞ ＞ attached wih a vip resource of IPAddr2:
> ＞ ＞
> ＞ ＞
> ＞ ＞ ＞crm status
> ＞ ＞
> ＞ ＞
> ＞ ＞ Online: [ paas-controller-1 paas-controller-2 paas-controller-3 ]
> ＞ ＞
> ＞ ＞
> ＞ ＞ router_vip (ocf::heartbeat:IPaddr2): Started paas-
> controller-1
> ＞ ＞
> ＞ ＞ sdclient_vip (ocf::heartbeat:IPaddr2): Started paas-
> controller-3
> ＞ ＞
> ＞ ＞ apigateway_vip (ocf::heartbeat:IPaddr2): Started paas-
> controller-2
> ＞ ＞
> ＞ ＞ Clone Set: sdclient_rep [sdclient]
> ＞ ＞
> ＞ ＞ Started: [ paas-controller-1 paas-controller-2 paas-
> controller-3 ]
> ＞ ＞
> ＞ ＞ Clone Set: router_rep [router]
> ＞ ＞
> ＞ ＞ Started: [ paas-controller-1 paas-controller-2 paas-
> controller-3 ]
> ＞ ＞
> ＞ ＞ Clone Set: apigateway_rep [apigateway]
> ＞ ＞
> ＞ ＞ Started: [ paas-controller-1 paas-controller-2 paas-
> controller-3 ]
> ＞ ＞
> ＞ ＞
> ＞ ＞ It is observed that sometimes the clone resource is stuck to monitor
> ＞ ＞ when the service fails:
> ＞ ＞
> ＞ ＞
> ＞ ＞ router_vip (ocf::heartbeat:IPaddr2): Started paas-
> controller-1
> ＞ ＞
> ＞ ＞ sdclient_vip (ocf::heartbeat:IPaddr2): Started paas-
> controller-2
> ＞ ＞
> ＞ ＞ apigateway_vip (ocf::heartbeat:IPaddr2): Started paas-
> controller-3
> ＞ ＞
> ＞ ＞ Clone Set: sdclient_rep [sdclient]
> ＞ ＞
> ＞ ＞ Started: [ paas-controller-1 paas-controller-2 ]
> ＞ ＞
> ＞ ＞ Stopped: [ paas-controller-3 ]
> ＞ ＞
> ＞ ＞ Clone Set: router_rep [router]
> ＞ ＞
> ＞ ＞ router (ocf::heartbeat:router): Started
> ＞ ＞ paas-controller-3 FAILED
> ＞ ＞
> ＞ ＞ Started: [ paas-controller-1 paas-controller-2 ]
> ＞ ＞
> ＞ ＞ Clone Set: apigateway_rep [apigateway]
> ＞ ＞
> ＞ ＞ apigateway (ocf::heartbeat:apigateway): Started
> ＞ ＞ paas-controller-3 FAILED
> ＞ ＞
> ＞ ＞ Started: [ paas-controller-1 paas-controller-2 ]
> ＞ ＞
> ＞ ＞
> ＞ ＞ in the example above. the sdclient_rep get restarted on node 3, while
> ＞ ＞ the other two hang at monitoring on node 3, here are the ocf logs:
> ＞ ＞
> ＞ ＞
> ＞ ＞ abnormal (apigateway_rep):
> ＞ ＞
> ＞ ＞ 2017-02-13 18:27:53 [23586]===print_log test_monitor run_func main===
> ＞ ＞ Starting health check.
> ＞ ＞
> ＞ ＞ 2017-02-13 18:27:53 [23586]===print_log test_monitor run_func main===
> ＞ ＞ health check succeed.
> ＞ ＞
> ＞ ＞ 2017-02-13 18:27:55 [24010]===print_log test_monitor run_func main===
> ＞ ＞ Starting health check.
> ＞ ＞
> ＞ ＞ 2017-02-13 18:27:55 [24010]===print_log test_monitor run_func main===
> ＞ ＞ Failed: docker daemon is not running.
> ＞ ＞
> ＞ ＞ 2017-02-13 18:27:57 [24095]===print_log test_monitor run_func main===
> ＞ ＞ Starting health check.
> ＞ ＞
> ＞ ＞ 2017-02-13 18:27:57 [24095]===print_log test_monitor run_func main===
> ＞ ＞ Failed: docker daemon is not running.
> ＞ ＞
> ＞ ＞ 2017-02-13 18:27:59 [24159]===print_log test_monitor run_func main===
> ＞ ＞ Starting health check.
> ＞ ＞
> ＞ ＞ 2017-02-13 18:27:59 [24159]===print_log test_monitor run_func main===
> ＞ ＞ Failed: docker daemon is not running.
> ＞ ＞
> ＞ ＞
> ＞ ＞ normal (sdclient_rep):
> ＞ ＞
> ＞ ＞ 2017-02-13 18:27:52 [23507]===print_log sdclient_monitor run_func
> ＞ ＞ main=== health check succeed.
> ＞ ＞
> ＞ ＞ 2017-02-13 18:27:54 [23630]===print_log sdclient_monitor run_func
> ＞ ＞ main=== Starting health check.
> ＞ ＞
> ＞ ＞ 2017-02-13 18:27:54 [23630]===print_log sdclient_monitor run_func
> ＞ ＞ main=== Failed: docker daemon is not running.
> ＞ ＞
> ＞ ＞ 2017-02-13 18:27:55 [23710]===print_log sdclient_stop run_func main===
> ＞ ＞ Starting stop the container.
> ＞ ＞
> ＞ ＞ 2017-02-13 18:27:55 [23710]===print_log sdclient_stop run_func main===
> ＞ ＞ docker daemon lost, pretend stop succeed.
> ＞ ＞
> ＞ ＞ 2017-02-13 18:27:55 [23763]===print_log sdclient_start run_func main===
> ＞ ＞ Starting run the container.
> ＞ ＞
> ＞ ＞ 2017-02-13 18:27:55 [23763]===print_log sdclient_start run_func main===
> ＞ ＞ docker daemon lost, try again in 5 secs.
> ＞ ＞
> ＞ ＞ 2017-02-13 18:28:00 [23763]===print_log sdclient_start run_func main===
> ＞ ＞ docker daemon lost, try again in 5 secs.
> ＞ ＞
> ＞ ＞ 2017-02-13 18:28:05 [23763]===print_log sdclient_start run_func main===
> ＞ ＞ docker daemon lost, try again in 5 secs.
> ＞ ＞
> ＞ ＞
> ＞ ＞ If I disable 2 clone resource, the switch over test for one clone
> ＞ ＞ resource works as expected: fail the service -＞ monitor fails -＞ stop
> ＞ ＞ -＞ start
> ＞ ＞
> ＞ ＞
> ＞ ＞ Online: [ paas-controller-1 paas-controller-2 paas-controller-3 ]
> ＞ ＞
> ＞ ＞
> ＞ ＞ sdclient_vip (ocf::heartbeat:IPaddr2): Started paas-
> controller-2
> ＞ ＞
> ＞ ＞ Clone Set: sdclient_rep [sdclient]
> ＞ ＞
> ＞ ＞ Started: [ paas-controller-1 paas-controller-2 ]
> ＞ ＞
> ＞ ＞ Stopped: [ paas-controller-3 ]
> ＞ ＞
> ＞ ＞
> ＞ ＞ what's the reason behind????
> ＞
> ＞ Can you show the configuration of the three clones, their operations,
> ＞ and any constraints?
> ＞
> ＞ Normally, the response is controlled by the monitor operation's on-fail
> ＞ attribute (which defaults to restart).
More information about the Users
mailing list