[ClusterLabs] I want to have some resource monitored and based on that make an acton. Is it possible?

Thu Mar 12 10:54:29 EDT 2020

On Wed, 2020-03-11 at 20:01 +0200, Roman Hershkovich wrote:
> But colocation of dbprobe won't pull trigger of webserver ? Or
> because that it is below in order - it will just restart services ?

I'm not sure what you mean. Resource failures don't cause node fencing
unless you explicitly ask for it with on-fail=fence. Resource failures
don't cause failover to another node unless migration-threshold is set
for the resource (or it fails a million times).

If you want the db monitor failure to cause the web server to restart,
order the web server after the db monitor. Otherwise, the db monitor
failure won't have any effect on the web server.

> On Wed, Mar 11, 2020, 18:41 Ken Gaillot <kgaillot at redhat.com> wrote:
> > On Wed, 2020-03-11 at 16:08 +0200, Roman Hershkovich wrote:
> > > Great, thank you very much for explanation. Regarding returning
> > error
> > > - i did not knew.
> > > So, basically i can have a service, that will probe for master
> > DB, in
> > > case of its transfer - service will update /etc/hosts and return
> > > error, which will be caught by pcs and it will restart whole
> > > dependent set ? Sounds good.
> > > But how i can do 2 "main resources" ? I have webserver AND
> > > db_monitor. In case of failure of webserver - should all start on
> > > node b, but in case of DB change - only underlying resources ...
> > > Should i make webserver outside of set? 
> > 
> > If you want the webserver to move to another node after a single
> > failure (of the webserver itself), set its migration-threshold to
> > 1. If
> > you want other resources to move with it, colocate them with the
> > webserver.
> > 
> > The db monitor won't affect that -- if the db monitor fails,
> > anything
> > ordered after it will restart.
> > 
> > > On Wed, Mar 11, 2020 at 3:57 PM Ken Gaillot <kgaillot at redhat.com>
> > > wrote:
> > > > On Wed, 2020-03-11 at 02:27 +0200, Roman Hershkovich wrote:
> > > > > Yes.
> > > > > I have only 1 APP active at same time, and so I want this app
> > to
> > > > be
> > > > > restarted whenever DB changes. Another one is a "standby"
> > APP,
> > > > where
> > > > > all resources are shut.
> > > > > So i thought about adding some "service" script, which will
> > probe
> > > > a
> > > > > DB , and in case if it finds a CHANGE - will trigger pcs to
> > > > reload a
> > > > > set of resources, where one of resource would be a systemctl
> > > > file,
> > > > > which will continue to run a script, so in case of next
> > change of
> > > > DB
> > > > > - it will restart APP set again. Is it sounds reasonable? (i
> > > > don't
> > > > > care of errors. I mean - i do, i want to log, but i'm ok to
> > see
> > > > them)
> > > > 
> > > > That sounds fine, but I'd trigger the restart by returning an
> > error
> > > > code from the db-monitoring script, rather than directly
> > attempt to
> > > > restart the resources via pcs. If you order the other resources
> > > > after
> > > > the db-monitoring script, pacemaker will automatically restart
> > them
> > > > when the db-monitoring script returns an error.
> > > > 
> > > > > In addition - i thought maybe bringing PAF here could be
> > useful -
> > > > but
> > > > > this is even more complex ... 
> > > > 
> > > > If bringing the db into the cluster is a possibility, that
> > would
> > > > probably be more reliable, with a quicker response too.
> > > > 
> > > > In that case you would simply order the dependent resources
> > after
> > > > the
> > > > database master promotion. pcs example: pcs constraint order
> > > > promote
> > > > DB-RSC then start DEPENDENT-RSC
> > > > 
> > > > > On Tue, Mar 10, 2020 at 10:28 PM Ken Gaillot <
> > kgaillot at redhat.com
> > > > >
> > > > > wrote:
> > > > > > On Tue, 2020-03-10 at 21:03 +0200, Roman Hershkovich wrote:
> > > > > > > DB servers are not in PCS cluster. Basically you say that
> > i
> > > > need
> > > > > > to
> > > > > > > add them to PCS cluster and then start them? but in case
> > if
> > > > DB1
> > > > > > fails
> > > > > > > - DB2 autopromoted and not required start of service
> > again>
> > > > > > > 
> > > > > > > Regarding colocation rule - i'm kind of missing logic how
> > it
> > > > > > works -
> > > > > > > how i can "colocate" 1 of 2 APP servers to be around a
> > master
> > > > DB
> > > > > > ? 
> > > > > > 
> > > > > > If I understand correctly, what you want is that both apps
> > are
> > > > > > restarted if the master changes?
> > > > > > 
> > > > > > I'm thinking you'll need a custom OCF agent for the app
> > > > servers.
> > > > > > The
> > > > > > monitor action, in addition to checking the app's status,
> > could
> > > > > > also
> > > > > > check which db is master, and return an error if it's
> > changed
> > > > since
> > > > > > the
> > > > > > last monitor. (The start action would have to record the
> > > > initial
> > > > > > master.) Pacemaker will restart the app to recover from the
> > > > error.
> > > > > > 
> > > > > > That is a little hacky because you'll have errors in the
> > status
> > > > > > every
> > > > > > time the master moves, but maybe that's worth knowing in
> > your
> > > > > > situation
> > > > > > anyway.
> > > > > > 
> > > > > > > On Tue, Mar 10, 2020 at 8:42 PM Strahil Nikolov <
> > > > > > > hunter86_bg at yahoo.com> wrote:
> > > > > > > > On March 10, 2020 7:31:27 PM GMT+02:00, Roman
> > Hershkovich <
> > > > > > > > warpik at gmail.com> wrote:
> > > > > > > > >I have 2 DB servers (master/slave with replica) and 2
> > APP
> > > > > > servers.
> > > > > > > > >2 APP servers managed by pacemaker  (active/passive) ,
> > but
> > > > i
> > > > > > want
> > > > > > > > also
> > > > > > > > >to
> > > > > > > > >monitor "which DB is master".  I can't use VIP (which
> > > > could be
> > > > > > > > sticked
> > > > > > > > >on
> > > > > > > > >master DB) - it is very limited virtual environment.
> > > > > > > > >
> > > > > > > > >Is it possible to create a rule or some other
> > scenario, so
> > > > in
> > > > > > case
> > > > > > > > if
> > > > > > > > >master moved - pacemaker will restart APP (app does
> > not
> > > > > > support
> > > > > > > > >failover) ?
> > > > > > > > 
> > > > > > > > Hi Roman,
> > > > > > > > 
> > > > > > > > If you set an order rule that  starts  first the
> > master 
> > > > and
> > > > > > then
> > > > > > > > the app, during a failover  the app will be stoped  and
> > > > once
> > > > > > the
> > > > > > > > master  is switched  (slave is promoted) the  app will
> > be
> > > > > > started
> > > > > > > > again.
> > > > > > > > 
> > > > > > > > Also you can consider  a  colocation rule that all 
> > apps
> > > > are 
> > > > > > > > started  where  the master  DB is running  -  so the
> > > > lattency
> > > > > > will
> > > > > > > > be minimal.
> > > > > > > > 
> > > > > > > > Best Regards,
> > > > > > > > Strahil Nikolov
> > > > _______________________________________________
> > > > Manage your subscription:
> > > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > > 
> > > > ClusterLabs home: https://www.clusterlabs.org/
> > _______________________________________________
> > Manage your subscription:
> > https://lists.clusterlabs.org/mailman/listinfo/users
> > 
> > ClusterLabs home: https://www.clusterlabs.org/
-- 
Ken Gaillot <kgaillot at redhat.com>