<div dir="ltr">Hi Damiano,<br><br>As a workaround I can suggest the following solution (in short, add infinite score colocation rule for the first resource in the group with the last resource in the group).<br><br><b>How it works:</b><br>After that it will work as you expect:<br>1) If at least one resource in the group fails the group will fail all resources which are located below the affected resource<br>2) If the last resource of the group is stopped the first resource of the group will be stopped as well<br>3) If the first resource of the group is stopped all the rest running resources will be stopped as well<br><br>Details:<br><br><b>Here is entire configuration:</b><br><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> sudo crm configure show<br>node 2: tt738741-ip1 \<br> attributes cloud-mode=0 site=Main webdispatcher-192.168.3.211=1<br>node 3: tt738741-ip2 \<br> attributes cloud-mode=0 site=Main webdispatcher-192.168.3.211=1<br>primitive haproxy-192.168.3.211 ocf:my:haproxy \<br> params vipaddress=192.168.3.211 \<br> op monitor interval=10s timeout=30s \<br> op start interval=0 timeout=30s \<br> op stop interval=0 timeout=60s \<br> meta failure-timeout=120s migration-threshold=3 target-role=Started<br>primitive vip-192.168.3.211 ocf:my:IPaddr2 \<br> params ip=192.168.3.211 iflabel=wd cidr_netmask=28 \<br> op monitor interval=10s timeout=20s \<br> op start interval=0 timeout=30s \<br> op stop interval=0 timeout=30s \<br> meta failure-timeout=120s migration-threshold=3 target-role=Started<br>primitive vip-alias-192.168.3.216 ocf:my:IPaddr2 \<br> params ip=192.168.3.216 iflabel=wd cidr_netmask=28 \<br> op monitor interval=10s timeout=20s \<br> op start interval=0 timeout=30s \<br> op stop interval=0 timeout=30s \<br> meta failure-timeout=120s migration-threshold=3<br>group grp_webdispatcher-192.168.3.211 vip-192.168.3.211 vip-alias-192.168.3.216 haproxy-192.168.3.211 \<br> meta<br>location allow-grp_webdispatcher-192.168.3.211 grp_webdispatcher-192.168.3.211 \<br> rule 100: webdispatcher-192.168.3.211 eq 1<br>colocation colocate-group-resources inf: vip-192.168.3.211 haproxy-192.168.3.211<br>location deny-grp_webdispatcher-192.168.3.211 grp_webdispatcher-192.168.3.211 \<br> rule -inf: webdispatcher-192.168.3.211 ne 1<br>property cib-bootstrap-options: \<br> have-watchdog=false \<br> dc-version=1.1.15-11.el7-e174ec8 \<br> cluster-infrastructure=corosync \<br> stonith-enabled=false \<br> no-quorum-policy=ignore \<br> symmetric-cluster=false \<br> default-resource-stickiness=1 \<br> cluster-recheck-interval=60s \<br> dc-deadtime=30 \<br> pe-input-series-max=5000 \<br> pe-error-series-max=5000 \<br> pe-warn-series-max=5000 \<br> shutdown-escalation=2min \<br> maintenance-mode=false \<br> last-lrm-refresh=1563900766<br>rsc_defaults rsc_defaults-options:</blockquote><br><br><b>Here is how colocation rule looks like in XML format:</b><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><constraints><br> <rsc_colocation id="colocate-group-resources" rsc="vip-192.168.3.211" with-rsc="haproxy-192.168.3.211" score="INFINITY"/><br></constraints></blockquote><br><b>Where</b>:<br>"colocate-group-resources" - any string id for the rule you like<br>"vip-192.168.3.211" - id of the first resource in the group<br>"haproxy-192.168.3.211" - id of the last resource in the group<br>score="INFINITY" - means do not tolerate if rule is not satisfied and stop vip-192.168.3.211 if haproxy-192.168.3.211 is not running<br><br><b>see</b> <a href="https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/2.0/html-single/Pacemaker_Explained/index.html#_colocation_properties">https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/2.0/html-single/Pacemaker_Explained/index.html#_colocation_properties</a><br><br><br><b>How to add:</b><br>1) Via "sudo crm configure edit"<br>- insert "colocation colocate-group-resources inf: vip-192.168.3.211 haproxy-192.168.3.211" after groups declaration<br>- save file and exit<br>- you'll see warnings, but it still works:<br>WARNING: colocate-group-resources: resource vip-192.168.3.211 is grouped, constraints should apply to the group<br>WARNING: colocate-group-resources: resource haproxy-192.168.3.211 is grouped, constraints should apply to the group<br><br>2) Via cibadmin by replacing cib.xml(crm configure edit xml didn't work for me)<br>sudo cibadmin -Q > ~/orig.cib.xml<br>cp ~/orig.cib.xml ~/new.cib.xml<br>head -1 ~/new.cib.xml | grep admin_epoch<br># in your editor<br># increase admin_epoch by 1<br># and insert the above XML into <constraints>..</constraints> tag if exists or create a new one after </resources> tag.<br>nano ~/new.cib.xml<br><br># replace current CIB (cluster information base) with a new one:<br>sudo cibadmin --replace --xml-file ~/new.cib.xml<br><br><br><b>Test results:</b><br><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> sudo crm status<br>...<br> Resource Group: grp_webdispatcher-192.168.3.211<br> vip-192.168.3.211 (ocf::my:IPaddr2): <b>Started</b> tt738741-ip1<br> vip-alias-192.168.3.216 (ocf::my:IPaddr2): <b>Started</b> tt738741-ip1<br> haproxy-192.168.3.211 (ocf::my:haproxy): <b>Started</b> tt738741-ip1</blockquote><br><br>> sudo crm resource <b>stop</b> haproxy-192.168.3.211<br><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> sudo crm status<br>...<br> Resource Group: grp_webdispatcher-192.168.3.211<br> vip-192.168.3.211 (ocf::my:IPaddr2): <b>Stopped</b><br> vip-alias-192.168.3.216 (ocf::my:IPaddr2): <b>Stopped</b><br> haproxy-192.168.3.211 (ocf::my:haproxy): <b>Stopped</b> (disabled)</blockquote><br><br><br>> sudo crm resource <b>start</b> haproxy-192.168.3.211<br><br><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> sudo crm status<br>...<br> Resource Group: grp_webdispatcher-192.168.3.211<br> vip-192.168.3.211 (ocf::my:IPaddr2): <b>Started</b> tt738741-ip1<br> vip-alias-192.168.3.216 (ocf::my:IPaddr2): <b>Started</b> tt738741-ip1<br> haproxy-192.168.3.211 (ocf::my:haproxy): <b>Started</b> tt738741-ip1</blockquote><div><br></div><div>Regards,</div><div>Igor </div></div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 28, 2021 at 8:31 PM Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I've opened a feature request for this:<br>
<br>
<a href="https://bugs.clusterlabs.org/show_bug.cgi?id=5465" rel="noreferrer" target="_blank">https://bugs.clusterlabs.org/show_bug.cgi?id=5465</a><br>
<br>
Realistically, developer time is tight for the foreseeable future, so<br>
it's more a wish-list item unless someone volunteers to work on it.<br>
<br>
On Thu, 2021-01-28 at 17:42 +0100, damiano giuliani wrote:<br>
> Hi Ulrich, thanks for the answer, <br>
> as Ken explained me, there isnt any way to prevent earlier members<br>
> from running<br>
> if a later member has no available node, <br>
> if no node is available for the failed member, then it will just<br>
> remain<br>
> stopped,and the earlier<br>
> members will stay active where they are. <br>
> i really hope was a solution or workaorund for this, but as ken<br>
> clarify, pacemaker cant hadle this exceptions.<br>
> <br>
> Many thanks for your quick and effective support.<br>
> <br>
> Have a good evening!<br>
> <br>
> Damiano<br>
> <br>
> <br>
> Il giorno gio 28 gen 2021 alle ore 11:15 Ulrich Windl <<br>
> <a href="mailto:Ulrich.Windl@rz.uni-regensburg.de" target="_blank">Ulrich.Windl@rz.uni-regensburg.de</a>> ha scritto:<br>
> > >>> damiano giuliani <<a href="mailto:damianogiuliani87@gmail.com" target="_blank">damianogiuliani87@gmail.com</a>> schrieb am<br>
> > 27.01.2021 um<br>
> > 19:25<br>
> > in Nachricht<br>
> > <CAG=zYNOx-R=wKbhtm=4N7qaoYKE=<a href="mailto:ofORVQ7jA0jr17oYjgqOhQ@mail.gmail.com" target="_blank">ofORVQ7jA0jr17oYjgqOhQ@mail.gmail.com</a><br>
> > >:<br>
> > > Hi Andrei, Thanks for ur help.<br>
> > > if one of my resource in the group fails or the primary node<br>
> > went down (<br>
> > > in my case acspcmk-02 ), the probe notices it and pacemaker tries<br>
> > to<br>
> > > restart the whole resource group on the second node.<br>
> > > if the second node cant run one of my grouped resources, it tries<br>
> > to stop<br>
> > > them.<br>
> > <br>
> > And what exactly is what you want? The behavior described it how<br>
> > the cluster<br>
> > handles it normally.<br>
> > <br>
> > > <br>
> > > <br>
> > > i attached my cluster status; my primary node ( acspcmk-02 )<br>
> > fails and the<br>
> > > resource group tries to restart on the acspcmk-01, i keep broken<br>
> > the<br>
> > > resource "lta-subscription-backend-ope-s3" on purpose and as you<br>
> > can see<br>
> > > some grouped resources are still started..<br>
> > > i would like to know how achive a condition that the resource<br>
> > group must<br>
> > > start properly for each resources, if not stop all the group<br>
> > without some<br>
> > > services still up and running.<br>
> > > <br>
> > > <br>
> > > 2 nodes configured<br>
> > > 28 resources configured<br>
> > > <br>
> > > Online: [ acspcmk-01 ]<br>
> > > OFFLINE: [ acspcmk-02 ]<br>
> > > <br>
> > > Full list of resources:<br>
> > > <br>
> > > Clone Set: lta-odata-frontend-ope-s1-clone [lta-odata-frontend-<br>
> > ope-s1]<br>
> > > Started: [ acspcmk-01 ]<br>
> > > Stopped: [ acspcmk-02 ]<br>
> > > Clone Set: lta-odata-frontend-ope-s2-clone [lta-odata-frontend-<br>
> > ope-s2]<br>
> > > Started: [ acspcmk-01 ]<br>
> > > Stopped: [ acspcmk-02 ]<br>
> > > Clone Set: lta-odata-frontend-ope-s3-clone [lta-odata-frontend-<br>
> > ope-s3]<br>
> > > Started: [ acspcmk-01 ]<br>
> > > Stopped: [ acspcmk-02 ]<br>
> > > Clone Set: s1ltaestimationtime-clone [s1ltaestimationtime]<br>
> > > Started: [ acspcmk-01 ]<br>
> > > Stopped: [ acspcmk-02 ]<br>
> > > Clone Set: s2ltaestimationtime-clone [s2ltaestimationtime]<br>
> > > Started: [ acspcmk-01 ]<br>
> > > Stopped: [ acspcmk-02 ]<br>
> > > Clone Set: s3ltaestimationtime-clone [s3ltaestimationtime]<br>
> > > Started: [ acspcmk-01 ]<br>
> > > Stopped: [ acspcmk-02 ]<br>
> > > Clone Set: openresty-clone [openresty]<br>
> > > Started: [ acspcmk-01 ]<br>
> > > Stopped: [ acspcmk-02 ]<br>
> > > Resource Group: LTA_SINGLE_RESOURCES<br>
> > > VIP (ocf::heartbeat:IPaddr2): Started acspcmk-<br>
> > 01<br>
> > > lta-subscription-backend-ope-s1<br>
> > > (systemd:lta-subscription-backend-ope-s1): Started acspcmk-<br>
> > 01<br>
> > > lta-subscription-backend-ope-s2<br>
> > > (systemd:lta-subscription-backend-ope-s2): Started acspcmk-<br>
> > 01<br>
> > > lta-subscription-backend-ope-s3<br>
> > > (systemd:lta-subscription-backend-ope-s3): Stopped<br>
> > > s1ltaquotaservice (systemd:s1ltaquotaservice): Stopped<br>
> > > s2ltaquotaservice (systemd:s2ltaquotaservice): Stopped<br>
> > > s3ltaquotaservice (systemd:s3ltaquotaservice): Stopped<br>
> > > s1ltarolling (systemd:s1ltarolling): Stopped<br>
> > > s2ltarolling (systemd:s2ltarolling): Stopped<br>
> > > s3ltarolling (systemd:s3ltarolling): Stopped<br>
> > > s1srvnotificationdispatcher<br>
> > > (systemd:s1srvnotificationdispatcher): Stopped<br>
> > > s2srvnotificationdispatcher<br>
> > > (systemd:s2srvnotificationdispatcher): Stopped<br>
> > > s3srvnotificationdispatcher<br>
> > > (systemd:s3srvnotificationdispatcher): Stopped<br>
> > > <br>
> > > Failed Resource Actions:<br>
> > > * lta-subscription-backend-ope-s3_start_0 on acspcmk-01 'unknown<br>
> > error'<br>
> > > (1): call=466, status=complete, exitreason='',<br>
> > > last-rc-change='Wed Jan 27 13:00:21 2021', queued=0ms,<br>
> > exec=2128ms<br>
> > > <br>
> > > Daemon Status:<br>
> > > corosync: active/disabled<br>
> > > pacemaker: active/disabled<br>
> > > pcsd: active/enabled<br>
> > > sbd: active/enabled<br>
> > > <br>
> > > <br>
> > > I hope i explained my problem at my best,<br>
> > > <br>
> > > Thanks for your time and help.<br>
> > > <br>
> > > Good Evening<br>
> > > <br>
> > > Damiano<br>
> > > <br>
> > > Il giorno mer 27 gen 2021 alle ore 19:03 Andrei Borzenkov <<br>
> > > <a href="mailto:arvidjaar@gmail.com" target="_blank">arvidjaar@gmail.com</a>> ha scritto:<br>
> > > <br>
> > >> 27.01.2021 19:06, damiano giuliani пишет:<br>
> > >> > Hi all im pretty new to the clusters, im struggling trying to<br>
> > configure<br>
> > a<br>
> > >> > bounch of resources and test how they failover.my need is to<br>
> > start and<br>
> > >> > manage a group of resources as one (in order to archive this a<br>
> > resource<br>
> > >> > group has been created), and if one of them cant run and still<br>
> > fails,<br>
> > the<br>
> > >> > cluster will try to restart the resource group in the<br>
> > secondary node, if<br>
> > >> it<br>
> > >> > cant run the all the resource toghter disable all the resource<br>
> > group.<br>
> > >> > i would like to know if there is a way to set the cluster to<br>
> > disable all<br>
> > >> > the resources of the group (or the group itself) if it cant be<br>
> > run all<br>
> > >> the<br>
> > >> > resoruces somewhere.<br>
> > >> ><br>
> > >><br>
> > >> That's what pacemaker group does. I am not sure what you mean<br>
> > with<br>
> > >> "disable all resources". If resource fail count on a node<br>
> > exceeds<br>
> > >> threshold, this node is banned from running resource. If<br>
> > resource failed<br>
> > >> on every node, no node can run it until you clear fail count.<br>
> > >><br>
> > >> "Disable resource" in pacemaker would mean setting its target-<br>
> > role to<br>
> > >> stopped. That does not happen automatically (at least I am not<br>
> > aware of<br>
> > >> it).<br>
> > >> _______________________________________________<br>
> > >> Manage your subscription:<br>
> > >> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a> <br>
> > >><br>
> > >> ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a> <br>
> > >><br>
> > <br>
> > <br>
> > <br>
> > _______________________________________________<br>
> > Manage your subscription:<br>
> > <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
> > <br>
> > ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
> <br>
> _______________________________________________<br>
> Manage your subscription:<br>
> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
> <br>
> ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
-- <br>
Ken Gaillot <<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>><br>
<br>
_______________________________________________<br>
Manage your subscription:<br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
<br>
ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
</blockquote></div>