From elias at po-mayak.ru  Sun Aug  2 23:26:21 2020
From: elias at po-mayak.ru (=?utf-8?B?0JjQu9GM0Y8g0J3QsNGB0L7QvdC+0LK=?=)
Date: Mon, 3 Aug 2020 08:26:21 +0500
Subject: [ClusterLabs] Clear Pending Fencing Action
Message-ID: <20200803032623.75154600275@iwtm.local>

Hello!
  
After troubleshooting 2-Node cluster, crm_mon deprecated actions are displayed in ?Pending Fencing Action:? list.
How can I delete them.
?stonith_admin --cleanup --history=*? does not delete it.


? ?????????,
???? ???????
elias at po-mayak.ru

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200803/1b4047fd/attachment.htm>

From nwahl at redhat.com  Mon Aug  3 01:04:26 2020
From: nwahl at redhat.com (Reid Wahl)
Date: Sun, 2 Aug 2020 22:04:26 -0700
Subject: [ClusterLabs] Clear Pending Fencing Action
In-Reply-To: <20200803032623.75154600275@iwtm.local>
References: <20200803032623.75154600275@iwtm.local>
Message-ID: <CAPiuu9-cppfbxmJ3qMHrOzQ-z9zXbqTf_HvVdYAd4HKkgR3GKw@mail.gmail.com>

Hi, ????. `stonith_admin --cleanup` doesn't get rid of pending actions,
only failed ones. You might be hitting
https://bugs.clusterlabs.org/show_bug.cgi?id=5401.

I believe a simultaneous reboot of both nodes will clear the pending
actions. I don't recall whether there's any other way to clear them.

On Sun, Aug 2, 2020 at 8:26 PM ???? ??????? <elias at po-mayak.ru> wrote:

> Hello!
>
>
>
> After troubleshooting 2-Node cluster, crm_mon deprecated actions are
> displayed in ?Pending Fencing Action:? list.
>
> How can I delete them.
>
> ?stonith_admin --cleanup --history=*? does not delete it.
>
>
>
>
>
> ? ?????????,
> ???? ???????
> elias at po-mayak.ru
>
>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>


-- 
Regards,

Reid Wahl, RHCA
Software Maintenance Engineer, Red Hat
CEE - Platform Support Delivery - ClusterHA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200802/b3f17434/attachment.htm>

From Ulrich.Windl at rz.uni-regensburg.de  Mon Aug  3 01:53:36 2020
From: Ulrich.Windl at rz.uni-regensburg.de (Ulrich Windl)
Date: Mon, 03 Aug 2020 07:53:36 +0200
Subject: [ClusterLabs] Antw: Re: Antw: Re: Antw: [EXT] why is node fenced ?
In-Reply-To: <577378942.29145077.1596207358970.JavaMail.zimbra@helmholtz-muenchen.de>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <5F2276A7020000A10003A4EC@gwsmtp.uni-regensburg.de>
 <1551037644.28469209.1596140634882.JavaMail.zimbra@helmholtz-muenchen.de>
 <5F23B44F020000A10003A58B@gwsmtp.uni-regensburg.de>
 <577378942.29145077.1596207358970.JavaMail.zimbra@helmholtz-muenchen.de>
Message-ID: <5F27A660020000A10003A63B@gwsmtp.uni-regensburg.de>

>>> "Lentes, Bernd" <bernd.lentes at helmholtz-muenchen.de> schrieb am 31.07.2020
um
16:55 in Nachricht
<577378942.29145077.1596207358970.JavaMail.zimbra at helmholtz-muenchen.de>:

> 
> ----- On Jul 31, 2020, at 8:03 AM, Ulrich Windl 
> Ulrich.Windl at rz.uni-regensburg.de wrote:
> 
> 
>>>> 
>>>> My guess is that ha-idg-1 was fenced because a failed migration from
>>> ha-idg-2
>>>> is treated like a stop failure on ha-idg-2. Stop failures cause fencing.
>> You
>>>> should have tested your resource before going productive.
>>> 
>>> Migration failed at 16:59:34.
>>> Node is fenced at 17:05:35. 6 minutes later.
>>> The cluster needs 6 minutes to decide to fence the node ?
>>> I don't believe that the failed migration is the cause for the fencing.
>> 
>> What are the values for migration timeout and for stop timeout?
>> 
> 
> 
> primitive vm_nextcloud VirtualDomain \
>         params config="/mnt/share/vm_nextcloud.xml" \
>         params hypervisor="qemu:///system" \
>         params migration_transport=ssh \
>         params migrate_options="--p2p --tunnelled" \
>         op start interval=0 timeout=120 \
>         op stop interval=0 timeout=180 \  <==================
>         op monitor interval=30 timeout=25 \
>         op migrate_from interval=0 timeout=300 \ <=================
>         op migrate_to interval=0 timeout=300 \ <================
>         meta allow-migrate=true target-role=Started is-managed=true 
> maintenance=false \
>         utilization cpu=1 hv_memory=4096
> 
> 3 or 5 minutes, not 6 minutes.

My guess would rather be: 5 Minutes for migration; if it fails a stop attempt
(actually a restart) with another 3 minutes. That would be 8 minutes.

> 
> Bernd
> Helmholtz Zentrum M?nchen
> 
> Helmholtz Zentrum Muenchen
> Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
> Ingolstaedter Landstr. 1
> 85764 Neuherberg
> www.helmholtz-muenchen.de 
> Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
> Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin 
> Guenther
> Registergericht: Amtsgericht Muenchen HRB 6466
> USt-IdNr: DE 129521671
> 
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




From kgaillot at redhat.com  Tue Aug  4 16:38:38 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Tue, 04 Aug 2020 15:38:38 -0500
Subject: [ClusterLabs] Antw: [EXT] Coming in Pacemaker 2.0.5: finer
 control over resource and operation defaults
In-Reply-To: <5F1A8AAA020000A10003A3EB@gwsmtp.uni-regensburg.de>
References: <99c11c73d59560fccd472d09c3b76073dab1b73e.camel@redhat.com>
 <5F1A8AAA020000A10003A3EB@gwsmtp.uni-regensburg.de>
Message-ID: <e7b58a8de91c921f886d368de4a0476ed85570e8.camel@redhat.com>

On Fri, 2020-07-24 at 09:15 +0200, Ulrich Windl wrote:
> > > > Ken Gaillot <kgaillot at redhat.com> schrieb am 23.07.2020 um
> > > > 23:54 in
> 
> Nachricht
> <99c11c73d59560fccd472d09c3b76073dab1b73e.camel at redhat.com>:
> > Hi all,
> > 
> > Pacemaker 2.0.4 is barely out the door, and we're already looking
> > ahead
> > to 2.0.5, expected at the end of this year.
> > 
> > One of the new features, already available in the master branch,
> > will
> > be finer?grained control over resource and operation defaults.
> > 
> > Currently, you can set meta?attribute values in the CIB's
> > rsc_defaults
> > section to apply to all resources, and op_defaults to apply to all
> > operations. Rules can be used to apply defaults only during certain
> > times. For example, to set a default stickiness of INFINITY during
> > business hours and 0 outside those hours:
> > 
> >    <rsc_defaults>
> >     <meta_attributes id="business?hours" score="2">
> >       <rule id="business?hours?rule" score="0">
> >         <date_expression id="business?hours?expr"
> > operation="date_spec">
> >           <date_spec id="business?hours?spec" hours="9?17"
> > weekdays="1?5"/>
> >         </date_expression>
> >       </rule>
> >       <nvpair id="business?hours?stickiness"
> > name="resource?stickiness" 
> > value="INFINITY"/>
> >     </meta_attributes>
> >     <meta_attributes id="after?hours" score="1" >
> >       <nvpair id="after?hours?stickiness"
> > name="resource?stickiness"
> 
> value="0"/>
> >     </meta_attributes>
> >    </rsc_defaults>
> > 
> > But what if you want to change the default stickiness of just pgsql
> > databases? Or the default timeout of only start operations?
> 
> We are using a rather similar scenario, like the stickyness. However
> we
> distinguish between productive and "no so productive" (test,
> developement)
> resources. First we release the stickiness of non-essential resources
> so that
> they can be re-balanced if needed. Later when the productive
> resources are
> released, the nodes maybe balanced already using the non-essential
> resources.
> 
> At the moment we copied the rules to each resource, which is not
> nice, of
> course.
> 
> I'd appreciate:
> date_spec be defined once and reused often
> rule be defined once and reused often

Reusing a rule is already possible -- define it as usual in one place,
and in the other use <rule id-ref="original-rule-id"/>

That doesn't work for date_spec though.

> 
> > 
> > 2.0.5 will add new rule expressions for this purpose. Examples:
> > 
> >    <rsc_defaults>
> >     <meta_attributes id="pgsql?defaults" score="2">
> >       <rule id="pgsql?defaults?rule" score="0">
> >         <rsc_expression id="pgsql?defaults?expr" class="ocf" 
> > provider="heartbeat" type="pgsqlms"/>
> >       </rule>
> >       <nvpair id="pgsql?stickiness" name="resource?stickiness" 
> > value="INFINITY"/>
> >     </meta_attributes>
> >    </rsc_defaults>
> > 
> >    <op_defaults>
> >     <meta_attributes id="start?defaults" score="2">
> >       <rule id="start?defaults?rule" score="0">
> >         <op_expression id="start?defaults?expr" name="start"
> > interval="0"/>
> >       </rule>
> >       <nvpair id="start?timeout" name="timeout" value="60s"/>
> >     </meta_attributes>
> >    </op_defaults>
> > 
> > You can combine rsc_expression and op_expression in op_defaults
> > rules,
> > if for example you want to set a default stop timeout for all
> > ocf:heartbeat:docker resources.
> > 
> > This obviously can be convenient if you have many resources of the
> > same
> > type, but it has one other trick up its sleeve: this is the only
> > way
> > you can affect the meta?attributes of resources implicitly created
> > by
> > Pacemaker for bundles.
> > 
> > When you configure a bundle, Pacemaker will implicitly create
> > container
> > resources (ocf:heartbeat:docker, ocf:heartbeat:rkt, or
> > ocf:heartbeat:podman) and if appropriate, IP resources
> > (ocf:heartbeat:IPaddr2). Previously, there was no way to directly
> > affect these resources, but with these new expressions you can at
> > least
> > configure defaults that apply to them, without having to use those
> > same
> > defaults for all your resources.
> > ?? 
> > Ken Gaillot <kgaillot at redhat.com>
> > 
> > _______________________________________________
> > Manage your subscription:
> > https://lists.clusterlabs.org/mailman/listinfo/users 
> > 
> > ClusterLabs home: https://www.clusterlabs.org/ 
> 
> 
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> ClusterLabs home: https://www.clusterlabs.org/
-- 
Ken Gaillot <kgaillot at redhat.com>


From acecile at le-vert.net  Sun Aug  9 15:11:51 2020
From: acecile at le-vert.net (=?UTF-8?Q?Adam_C=c3=a9cile?=)
Date: Sun, 9 Aug 2020 21:11:51 +0200
Subject: [ClusterLabs] Automatic recover from split brain ?
Message-ID: <accce530-4ab9-ed8c-1b61-aed5184f597d@le-vert.net>

Hello,


I'm experiencing issue with corosync/pacemaker running on Debian Buster. 
Cluster has three nodes running in VMWare virtual machine and the 
cluster fails when VEEAM backups the virtual machine (I know it's doing 
bad things, like freezing completely the VM for a few minutes to make 
disk snapshot).

My biggest issue is that once the backup has been completed, the cluster 
stays in split brain state, and I'd like it to heal itself. Here current 
status:


One node is isolated:

Stack: corosync
Current DC: host2.domain.com (version 2.0.1-9e909a5bdd) - partition 
WITHOUT quorum
Last updated: Sat Aug? 8 11:59:46 2020
Last change: Fri Jul 24 07:18:12 2020 by root via cibadmin on 
host1.domain.com

3 nodes configured
6 resources configured

Online: [ host2.domain.com ]
OFFLINE: [ host3.domain.com host1.domain.com ]


Two others are seeing each others:

Stack: corosync
Current DC: host3.domain.com (version 2.0.1-9e909a5bdd) - partition with 
quorum
Last updated: Sat Aug? 8 12:07:56 2020
Last change: Fri Jul 24 07:18:12 2020 by root via cibadmin on 
host1.domain.com

3 nodes configured
6 resources configured

Online: [ host3.domain.com host1.domain.com ]
OFFLINE: [ host2.domain.com ]


The problem is that one of the resources is a floating IP address which 
is currently assigned to two different hosts...


Can you help me configuring the cluster correctly so this cannot occurs ?


Thanks in advance,

Adam.



From kgaillot at redhat.com  Mon Aug 10 16:19:10 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Mon, 10 Aug 2020 15:19:10 -0500
Subject: [ClusterLabs] Automatic recover from split brain ?
In-Reply-To: <accce530-4ab9-ed8c-1b61-aed5184f597d@le-vert.net>
References: <accce530-4ab9-ed8c-1b61-aed5184f597d@le-vert.net>
Message-ID: <ca26c80bc537a679710ecaeb3f6a2a52a53e106d.camel@redhat.com>

On Sun, 2020-08-09 at 21:11 +0200, Adam C?cile wrote:
> Hello,
> 
> 
> I'm experiencing issue with corosync/pacemaker running on Debian
> Buster. 
> Cluster has three nodes running in VMWare virtual machine and the 
> cluster fails when VEEAM backups the virtual machine (I know it's
> doing 
> bad things, like freezing completely the VM for a few minutes to
> make 
> disk snapshot).
> 
> My biggest issue is that once the backup has been completed, the
> cluster 
> stays in split brain state, and I'd like it to heal itself. Here 

Fencing is how the cluster prevents split-brain. When one node is lost,
the other nodes will not recover any resources from it until it's
fenced. For VMWare there's a fence_vmware_soap fence agent.

However that's intended for failure scenarios, not a planned outage
like a backup snapshot.

For planned outages, you can set the cluster-wide
property "maintenance-mode" to true. The cluster won't start, monitor,
or stop resources while in maintenance mode. You can use rules to
automatically put the cluster in maintenance mode at specific times.

However I believe even in maintenance mode, the node will get fenced if
it drops out of the corosync membership. Ideally you'd put the cluster
in maintenance mode, stop pacemaker and corosync on the node, do the
backup, then start pacemaker and corosync, wait for them to come up,
and take the cluster out of maintenance mode.

Alternatively, if you want the resources to move to other nodes while
the backup is being done, you could put the node in standby rather than
set maintenance mode.

> current 
> status:
> 
> 
> One node is isolated:
> 
> Stack: corosync
> Current DC: host2.domain.com (version 2.0.1-9e909a5bdd) - partition 
> WITHOUT quorum
> Last updated: Sat Aug  8 11:59:46 2020
> Last change: Fri Jul 24 07:18:12 2020 by root via cibadmin on 
> host1.domain.com
> 
> 3 nodes configured
> 6 resources configured
> 
> Online: [ host2.domain.com ]
> OFFLINE: [ host3.domain.com host1.domain.com ]
> 
> 
> Two others are seeing each others:
> 
> Stack: corosync
> Current DC: host3.domain.com (version 2.0.1-9e909a5bdd) - partition
> with 
> quorum
> Last updated: Sat Aug  8 12:07:56 2020
> Last change: Fri Jul 24 07:18:12 2020 by root via cibadmin on 
> host1.domain.com
> 
> 3 nodes configured
> 6 resources configured
> 
> Online: [ host3.domain.com host1.domain.com ]
> OFFLINE: [ host2.domain.com ]
> 
> 
> The problem is that one of the resources is a floating IP address
> which 
> is currently assigned to two different hosts...
> 
> 
> Can you help me configuring the cluster correctly so this cannot
> occurs ?
> 
> 
> Thanks in advance,
> 
> Adam.
> 
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> ClusterLabs home: https://www.clusterlabs.org/
-- 
Ken Gaillot <kgaillot at redhat.com>


From acecile at le-vert.net  Sat Aug  8 06:10:08 2020
From: acecile at le-vert.net (=?UTF-8?Q?Adam_C=c3=a9cile?=)
Date: Sat, 8 Aug 2020 12:10:08 +0200
Subject: [ClusterLabs] Automatic recover from split brain ?
Message-ID: <b899e0c9-0844-7643-5f07-59e47c9a806d@le-vert.net>

Hello,


I'm experiencing issue with corosync/pacemaker running on Debian Buster. 
Cluster has three nodes running in VMWare virtual machine and the 
cluster fails when VEEAM backups the virtual machine (I know it's doing 
bad things, like freezing completely the VM for a few minutes to make 
disk snapshot).

My biggest issue is that once the backup has been completed, the cluster 
stays in split brain state, and I'd like it to heal itself. Here current 
status:


One node is isolated:

Stack: corosync
Current DC: host2.domain.com (version 2.0.1-9e909a5bdd) - partition 
WITHOUT quorum
Last updated: Sat Aug? 8 11:59:46 2020
Last change: Fri Jul 24 07:18:12 2020 by root via cibadmin on 
host1.domain.com

3 nodes configured
6 resources configured

Online: [ host2.domain.com ]
OFFLINE: [ host3.domain.com host1.domain.com ]


Two others are seeing each others:

Stack: corosync
Current DC: host3.domain.com (version 2.0.1-9e909a5bdd) - partition with 
quorum
Last updated: Sat Aug? 8 12:07:56 2020
Last change: Fri Jul 24 07:18:12 2020 by root via cibadmin on 
host1.domain.com

3 nodes configured
6 resources configured

Online: [ host3.domain.com host1.domain.com ]
OFFLINE: [ host2.domain.com ]


The problem is that one of the resources is a floating IP address which 
is currently assigned to two different hosts...


Can you help me configuring the cluster correctly so this cannot occurs ?


Thanks in advance,

Adam.


From bernd.lentes at helmholtz-muenchen.de  Thu Aug  6 05:06:35 2020
From: bernd.lentes at helmholtz-muenchen.de (Lentes, Bernd)
Date: Thu, 6 Aug 2020 11:06:35 +0200 (CEST)
Subject: [ClusterLabs] why is node fenced ?
In-Reply-To: <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
Message-ID: <505919068.32962787.1596704795719.JavaMail.zimbra@helmholtz-muenchen.de>


----- Am 29. Jul 2020 um 18:53 schrieb kgaillot kgaillot at redhat.com:

 
> Since the ha-idg-2 is now shutting down, ha-idg-1 becomes DC.

The other way round.

>> Jul 20 17:05:33 [10690] ha-idg-2    pengine:  warning:
>> unpack_rsc_op_failure:   Processing failed migrate_to of vm_nextcloud
>> on ha-idg-1: unknown error | rc=1
>> Jul 20 17:05:33 [10690] ha-idg-2    pengine:  warning:
>> unpack_rsc_op_failure:   Processing failed start of vm_nextcloud on
>> ha-idg-2: unknown error | rc
>> 
>> Jul 20 17:05:33 [10690] ha-idg-2    pengine:     info:
>> native_color:    Resource vm_nextcloud cannot run anywhere
>> logical
>> 
>> Jul 20 17:05:33 [10690] ha-idg-2    pengine:  warning:
>> custom_action:   Action vm_nextcloud_stop_0 on ha-idg-1 is unrunnable
>> (pending)
>> ???
> 
> So this appears to be the problem. From these logs I would guess the
> successful stop on ha-idg-1 did not get written to the CIB for some
> reason. I'd look at the pe input from this transition on ha-idg-2 to
> confirm that.
> 
> Without the DC knowing about the stop, it tries to schedule a new one,
> but the node is shutting down so it can't do it, which means it has to
> be fenced.
> 
>> Jul 20 17:05:35 [10690] ha-idg-2    pengine:  warning:
>> custom_action:   Action vm_nextcloud_stop_0 on ha-idg-1 is unrunnable
>> (offline)
>> Jul 20 17:05:35 [10690] ha-idg-2    pengine:  warning:
>> pe_fence_node:   Cluster node ha-idg-1 will be fenced: resource
>> actions are unrunnable
>> Jul 20 17:05:35 [10690] ha-idg-2    pengine:  warning:
>> stage6:  Scheduling Node ha-idg-1 for STONITH
>> Jul 20 17:05:35 [10690] ha-idg-2    pengine:     info:
>> native_stop_constraints: vm_nextcloud_stop_0 is implicit after ha-
>> idg-1 is fenced
>> Jul 20 17:05:35 [10690] ha-idg-2    pengine:   notice:
>> LogNodeActions:   * Fence (Off) ha-idg-1 'resource actions are
>> unrunnable'
>> 
>> 
>> Why does it say "Jul 20 17:05:35 [10690] ha-idg-
>> 2    pengine:  warning: custom_action:   Action vm_nextcloud_stop_0
>> on ha-idg-1 is unrunnable (offline)" although
>> "Jul 20 17:04:06 [23768] ha-idg-1       crmd:   notice:
>> process_lrm_event:       Result of stop operation for vm_nextcloud on
>> ha-idg-1: 0 (ok) | call=3197 key=vm_nextcloud_stop_0 confirmed=true
>> cib-update=5960"
>> says that stop was ok ?

Bernd
Helmholtz Zentrum M?nchen

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin Guenther
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671



From kgaillot at redhat.com  Mon Aug 10 12:47:24 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Mon, 10 Aug 2020 11:47:24 -0500
Subject: [ClusterLabs] Coming in Pacemaker 2.0.5: on-fail=demote /
 no-quorum-policy=demote
Message-ID: <cb172f00a8ed1abdaa1b9cdc4c01c1a5c7aaf7c1.camel@redhat.com>

Hi all,

Looking ahead to the Pacemaker 2.0.5 release expected at the end of
this year, here is a new feature already in the master branch.

When configuring resource operations, Pacemaker lets you set an "on-
fail" policy to specify whether to restart the resource, fence the
node, etc., if the operation fails. With 2.0.5, a new possible value
will be "demote", which will mean "demote this resource but do not
fully restart it".

"Demote" will be a valid value only for promote actions, and for
recurring monitors with "role" set to "Master".

Once the resource is demoted, it will be eligible for promotion again,
so if the promotion scores have not changed, a promote on the same node
may be attempted. If this is not desired, the agent can change the
promotion scores either in the failed monitor or the demote.

The intended use case is an application where a successful demote 
assures a well-functioning service, and a full restart would be
unnecessarily heavyweight. A large database might be an example.

Similarly, Pacemaker offers the cluster-wide "no-quorum-policy" option
to specify what happens to resources when quorum is lost (the default
being to stop them). With 2.0.5, "demote" will be a possible value here
as well, and will mean "demote all promotable resources and stop all
other resources".

The intended use case is an application that cannot cause any harm
after being demoted, and may be useful in a demoted role even if there
is no quorum. A database that operates read-only when demoted and
doesn't depend on any non-promotable resources might be an example.

Happy clustering :)
-- 
Ken Gaillot <kgaillot at redhat.com>


From bernd.lentes at helmholtz-muenchen.de  Sun Aug  9 16:17:24 2020
From: bernd.lentes at helmholtz-muenchen.de (Lentes, Bernd)
Date: Sun, 9 Aug 2020 22:17:24 +0200 (CEST)
Subject: [ClusterLabs] why is node fenced ?
In-Reply-To: <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
Message-ID: <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>



----- Am 29. Jul 2020 um 18:53 schrieb kgaillot kgaillot at redhat.com:

> On Wed, 2020-07-29 at 17:26 +0200, Lentes, Bernd wrote:
>> Hi,
>> 
>> a few days ago one of my nodes was fenced and i don't know why, which
>> is something i really don't like.
>> What i did:
>> I put one node (ha-idg-1) in standby. The resources on it (most of
>> all virtual domains) were migrated to ha-idg-2,
>> except one domain (vm_nextcloud). On ha-idg-2 a mountpoint was
>> missing the xml of the domain points to.
>> Then the cluster tries to start vm_nextcloud on ha-idg-2 which of
>> course also failed.
>> Then ha-idg-1 was fenced.
>> I did a "crm history" over the respective time period, you find it
>> here:
>> https://hmgubox2.helmholtz-muenchen.de/index.php/s/529dfcXf5a72ifF
>> 
>> Here, from my point of view, the most interesting from the logs:
>> ha-idg-1:
>> Jul 20 16:59:33 [23763] ha-idg-1        cib:     info:
>> cib_perform_op:  Diff: --- 2.16196.19 2
>> Jul 20 16:59:33 [23763] ha-idg-1        cib:     info:
>> cib_perform_op:  Diff: +++ 2.16197.0 bc9a558dfbe6d7196653ce56ad1ee758
>> Jul 20 16:59:33 [23763] ha-idg-1        cib:     info:
>> cib_perform_op:  +  /cib:  @epoch=16197, @num_updates=0
>> Jul 20 16:59:33 [23763] ha-idg-1        cib:     info:
>> cib_perform_op:  +  /cib/configuration/nodes/node[@id='1084777482']/i
>> nstance_attributes[@id='nodes-108
>> 4777482']/nvpair[@id='nodes-1084777482-standby']:  @value=on
>> ha-idg-1 set to standby
>> 
>> Jul 20 16:59:34 [23768] ha-idg-1       crmd:   notice:
>> process_lrm_event:       ha-idg-1-vm_nextcloud_migrate_to_0:3169 [
>> error: Cannot access storage file
>> '/mnt/mcd/AG_BioInformatik/Technik/software_und_treiber/linux/ubuntu/
>> ubuntu-18.04.4-live-server-amd64.iso': No such file or
>> directory\nocf-exit-reason:vm_nextcloud: live migration to ha-idg-2
>> failed: 1\n ]
>> migration failed
>> 
>> Jul 20 17:04:01 [23767] ha-idg-1    pengine:    error:
>> native_create_actions:   Resource vm_nextcloud is active on 2 nodes
>> (attempting recovery)
>> ???
> 
> This is standard for a failed live migration -- the cluster doesn't
> know how far the migration actually got before failing, so it has to
> assume the VM could be active on either node. (The log message would
> make more sense saying "might be active" rather than "is active".)
> 
>> Jul 20 17:04:01 [23767] ha-idg-1    pengine:   notice:
>> LogAction:        *
>> Recover    vm_nextcloud           (             ha-idg-2 )
> 
> The recovery from that situation is a full stop on both nodes, and
> start on one of them.
> 
>> Jul 20 17:04:01 [23768] ha-idg-1       crmd:   notice:
>> te_rsc_command:  Initiating stop operation vm_nextcloud_stop_0 on ha-
>> idg-2 | action 106
>> Jul 20 17:04:01 [23768] ha-idg-1       crmd:   notice:
>> te_rsc_command:  Initiating stop operation vm_nextcloud_stop_0
>> locally on ha-idg-1 | action 2
>> 
>> Jul 20 17:04:01 [23768] ha-idg-1       crmd:     info:
>> match_graph_event:       Action vm_nextcloud_stop_0 (106) confirmed
>> on ha-idg-2 (rc=0)
>> 
>> Jul 20 17:04:06 [23768] ha-idg-1       crmd:   notice:
>> process_lrm_event:       Result of stop operation for vm_nextcloud on
>> ha-idg-1: 0 (ok) | call=3197 key=vm_nextcloud_stop_0 confirmed=true
>> cib-update=5960
> 
> It looks like both stops succeeded.
> 
>> Jul 20 17:05:29 [23761] ha-idg-1 pacemakerd:   notice:
>> crm_signal_dispatch:     Caught 'Terminated' signal | 15 (invoking
>> handler)
>> systemctl stop pacemaker.service
>> 
>> 
>> ha-idg-2:
>> Jul 20 17:04:03 [10691] ha-idg-2       crmd:   notice:
>> process_lrm_event:       Result of stop operation for vm_nextcloud on
>> ha-idg-2: 0 (ok) | call=157 key=vm_nextcloud_stop_0 confirmed=true
>> cib-update=57
>> the log from ha-idg-2 is two seconds ahead of ha-idg-1
>> 
>> Jul 20 17:04:08 [10688] ha-idg-2       lrmd:   notice:
>> log_execute:     executing - rsc:vm_nextcloud action:start
>> call_id:192
>> Jul 20 17:04:09 [10688] ha-idg-2       lrmd:   notice:
>> operation_finished:      vm_nextcloud_start_0:29107:stderr [ error:
>> Failed to create domain from /mnt/share/vm_nextcloud.xml ]
>> Jul 20 17:04:09 [10688] ha-idg-2       lrmd:   notice:
>> operation_finished:      vm_nextcloud_start_0:29107:stderr [ error:
>> Cannot access storage file
>> '/mnt/mcd/AG_BioInformatik/Technik/software_und_treiber/linux/ubuntu/
>> ubuntu-18.04.4-live-server-amd64.iso': No such file or directory ]
>> Jul 20 17:04:09 [10688] ha-idg-2       lrmd:   notice:
>> operation_finished:      vm_nextcloud_start_0:29107:stderr [ ocf-
>> exit-reason:Failed to start virtual domain vm_nextcloud. ]
>> Jul 20 17:04:09 [10688] ha-idg-2       lrmd:   notice:
>> log_finished:    finished - rsc:vm_nextcloud action:start call_id:192
>> pid:29107 exit-code:1 exec-time:581ms queue-time:0ms
>> start on ha-idg-2 failed
> 
> The start failed ...
> 
>> Jul 20 17:05:32 [10691] ha-idg-2       crmd:     info:
>> do_dc_takeover:  Taking over DC status for this partition
>> ha-idg-1 stopped pacemaker
> 
> Since the ha-idg-2 is now shutting down, ha-idg-1 becomes DC.
> 
>> Jul 20 17:05:33 [10690] ha-idg-2    pengine:  warning:
>> unpack_rsc_op_failure:   Processing failed migrate_to of vm_nextcloud
>> on ha-idg-1: unknown error | rc=1
>> Jul 20 17:05:33 [10690] ha-idg-2    pengine:  warning:
>> unpack_rsc_op_failure:   Processing failed start of vm_nextcloud on
>> ha-idg-2: unknown error | rc
>> 
>> Jul 20 17:05:33 [10690] ha-idg-2    pengine:     info:
>> native_color:    Resource vm_nextcloud cannot run anywhere
>> logical
>> 
>> Jul 20 17:05:33 [10690] ha-idg-2    pengine:  warning:
>> custom_action:   Action vm_nextcloud_stop_0 on ha-idg-1 is unrunnable
>> (pending)
>> ???
> 
> So this appears to be the problem. From these logs I would guess the
> successful stop on ha-idg-1 did not get written to the CIB for some
> reason. I'd look at the pe input from this transition on ha-idg-2 to
> confirm that.
> 
> Without the DC knowing about the stop, it tries to schedule a new one,
> but the node is shutting down so it can't do it, which means it has to
> be fenced.
> 
>> Jul 20 17:05:35 [10690] ha-idg-2    pengine:  warning:
>> custom_action:   Action vm_nextcloud_stop_0 on ha-idg-1 is unrunnable
>> (offline)
>> Jul 20 17:05:35 [10690] ha-idg-2    pengine:  warning:
>> pe_fence_node:   Cluster node ha-idg-1 will be fenced: resource
>> actions are unrunnable
>> Jul 20 17:05:35 [10690] ha-idg-2    pengine:  warning:
>> stage6:  Scheduling Node ha-idg-1 for STONITH
>> Jul 20 17:05:35 [10690] ha-idg-2    pengine:     info:
>> native_stop_constraints: vm_nextcloud_stop_0 is implicit after ha-
>> idg-1 is fenced
>> Jul 20 17:05:35 [10690] ha-idg-2    pengine:   notice:
>> LogNodeActions:   * Fence (Off) ha-idg-1 'resource actions are
>> unrunnable'
>> 
>> 
>> Why does it say "Jul 20 17:05:35 [10690] ha-idg-
>> 2    pengine:  warning: custom_action:   Action vm_nextcloud_stop_0
>> on ha-idg-1 is unrunnable (offline)" although
>> "Jul 20 17:04:06 [23768] ha-idg-1       crmd:   notice:
>> process_lrm_event:       Result of stop operation for vm_nextcloud on
>> ha-idg-1: 0 (ok) | call=3197 key=vm_nextcloud_stop_0 confirmed=true
>> cib-update=5960"
>> says that stop was ok ?

I'm stll digging in the logs trying to understand what happened.
What i'm wondering abaout:

Jul 20 17:04:26 [23768] ha-idg-1       crmd:   notice: run_graph:       Transition 4515 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-3117.bz2): Complete

Jul 20 17:05:29 [23768] ha-idg-1       crmd:     info: abort_transition_graph:  Transition 4515 aborted by status-1084777482-shutdown doing create shutdown=1595257529: Transient attribute change | cib=2.16197.72 source=abort_unless_down:317   path=/cib/status/node_state[@id='1084777482']/transient_attributes[@id='1084777482']/instance_attributes[@id='status-1084777482'] complete=true


A transition is completed and three secons later aborted. How can that be?

Bernd
Helmholtz Zentrum M?nchen

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin Guenther
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671



From mariusz.gronczewski at efigence.com  Mon Aug 10 08:21:53 2020
From: mariusz.gronczewski at efigence.com (Mariusz Gronczewski)
Date: Mon, 10 Aug 2020 14:21:53 +0200
Subject: [ClusterLabs] How to specify which IP pcs should use?
Message-ID: <20200810142153.603f56dd@hydra.home.zxz.li>

Hi,

Pacemaker 2, the current setup is

* management network with host's hostname resolving to host's
  management IP
* cluster network for Pacemaker/Corosync communication
* corosync set up with node name and IP of the cluster network

pcs status shows both nodes online, added config syncs to the other
node etc. but pcs cluster status shows one node being offline.

After a look in firewall logs it appears all of the communication is
going just fine on the cluster network but PCS tries to talk with PCSD
on port 2224 via *management* network instead of using IP set as
ring0_addr in corosync

Is "just use host's hostname regardless of config" something normal ?
Is there a separate setting to pcs about which IP it should use ?

Regards

-- 
Mariusz Gronczewski, Administrator

Efigence S. A.
ul. Wo?oska 9a, 02-583 Warszawa
T:   [+48] 22 380 13 13
NOC: [+48] 22 380 10 20
E: admin at efigence.com

From kgaillot at redhat.com  Mon Aug 10 17:59:07 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Mon, 10 Aug 2020 16:59:07 -0500
Subject: [ClusterLabs] why is node fenced ?
In-Reply-To: <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
 <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>
Message-ID: <f411d804d82fd34b7ae0974fcd9f17821a115299.camel@redhat.com>

On Sun, 2020-08-09 at 22:17 +0200, Lentes, Bernd wrote:
> 
> ----- Am 29. Jul 2020 um 18:53 schrieb kgaillot kgaillot at redhat.com:
> 
> > On Wed, 2020-07-29 at 17:26 +0200, Lentes, Bernd wrote:
> > > Hi,
> > > 
> > > a few days ago one of my nodes was fenced and i don't know why,
> > > which
> > > is something i really don't like.
> > > What i did:
> > > I put one node (ha-idg-1) in standby. The resources on it (most
> > > of
> > > all virtual domains) were migrated to ha-idg-2,
> > > except one domain (vm_nextcloud). On ha-idg-2 a mountpoint was
> > > missing the xml of the domain points to.
> > > Then the cluster tries to start vm_nextcloud on ha-idg-2 which of
> > > course also failed.
> > > Then ha-idg-1 was fenced.
> > > I did a "crm history" over the respective time period, you find
> > > it
> > > here:
> > > https://hmgubox2.helmholtz-muenchen.de/index.php/s/529dfcXf5a72ifF
> > > 
> > > Here, from my point of view, the most interesting from the logs:
> > > ha-idg-1:
> > > Jul 20 16:59:33 [23763] ha-idg-1        cib:     info:
> > > cib_perform_op:  Diff: --- 2.16196.19 2
> > > Jul 20 16:59:33 [23763] ha-idg-1        cib:     info:
> > > cib_perform_op:  Diff: +++ 2.16197.0
> > > bc9a558dfbe6d7196653ce56ad1ee758
> > > Jul 20 16:59:33 [23763] ha-idg-1        cib:     info:
> > > cib_perform_op:  +  /cib:  @epoch=16197, @num_updates=0
> > > Jul 20 16:59:33 [23763] ha-idg-1        cib:     info:
> > > cib_perform_op:  +  /cib/configuration/nodes/node[@id='1084777482
> > > ']/i
> > > nstance_attributes[@id='nodes-108
> > > 4777482']/nvpair[@id='nodes-1084777482-standby']:  @value=on
> > > ha-idg-1 set to standby
> > > 
> > > Jul 20 16:59:34 [23768] ha-idg-1       crmd:   notice:
> > > process_lrm_event:       ha-idg-1-vm_nextcloud_migrate_to_0:3169
> > > [
> > > error: Cannot access storage file
> > > '/mnt/mcd/AG_BioInformatik/Technik/software_und_treiber/linux/ubu
> > > ntu/
> > > ubuntu-18.04.4-live-server-amd64.iso': No such file or
> > > directory\nocf-exit-reason:vm_nextcloud: live migration to ha-
> > > idg-2
> > > failed: 1\n ]
> > > migration failed
> > > 
> > > Jul 20 17:04:01 [23767] ha-idg-1    pengine:    error:
> > > native_create_actions:   Resource vm_nextcloud is active on 2
> > > nodes
> > > (attempting recovery)
> > > ???
> > 
> > This is standard for a failed live migration -- the cluster doesn't
> > know how far the migration actually got before failing, so it has
> > to
> > assume the VM could be active on either node. (The log message
> > would
> > make more sense saying "might be active" rather than "is active".)
> > 
> > > Jul 20 17:04:01 [23767] ha-idg-1    pengine:   notice:
> > > LogAction:        *
> > > Recover    vm_nextcloud           (             ha-idg-2 )
> > 
> > The recovery from that situation is a full stop on both nodes, and
> > start on one of them.
> > 
> > > Jul 20 17:04:01 [23768] ha-idg-1       crmd:   notice:
> > > te_rsc_command:  Initiating stop operation vm_nextcloud_stop_0 on
> > > ha-
> > > idg-2 | action 106
> > > Jul 20 17:04:01 [23768] ha-idg-1       crmd:   notice:
> > > te_rsc_command:  Initiating stop operation vm_nextcloud_stop_0
> > > locally on ha-idg-1 | action 2
> > > 
> > > Jul 20 17:04:01 [23768] ha-idg-1       crmd:     info:
> > > match_graph_event:       Action vm_nextcloud_stop_0 (106)
> > > confirmed
> > > on ha-idg-2 (rc=0)
> > > 
> > > Jul 20 17:04:06 [23768] ha-idg-1       crmd:   notice:
> > > process_lrm_event:       Result of stop operation for
> > > vm_nextcloud on
> > > ha-idg-1: 0 (ok) | call=3197 key=vm_nextcloud_stop_0
> > > confirmed=true
> > > cib-update=5960
> > 
> > It looks like both stops succeeded.
> > 
> > > Jul 20 17:05:29 [23761] ha-idg-1 pacemakerd:   notice:
> > > crm_signal_dispatch:     Caught 'Terminated' signal | 15
> > > (invoking
> > > handler)
> > > systemctl stop pacemaker.service
> > > 
> > > 
> > > ha-idg-2:
> > > Jul 20 17:04:03 [10691] ha-idg-2       crmd:   notice:
> > > process_lrm_event:       Result of stop operation for
> > > vm_nextcloud on
> > > ha-idg-2: 0 (ok) | call=157 key=vm_nextcloud_stop_0
> > > confirmed=true
> > > cib-update=57
> > > the log from ha-idg-2 is two seconds ahead of ha-idg-1
> > > 
> > > Jul 20 17:04:08 [10688] ha-idg-2       lrmd:   notice:
> > > log_execute:     executing - rsc:vm_nextcloud action:start
> > > call_id:192
> > > Jul 20 17:04:09 [10688] ha-idg-2       lrmd:   notice:
> > > operation_finished:      vm_nextcloud_start_0:29107:stderr [
> > > error:
> > > Failed to create domain from /mnt/share/vm_nextcloud.xml ]
> > > Jul 20 17:04:09 [10688] ha-idg-2       lrmd:   notice:
> > > operation_finished:      vm_nextcloud_start_0:29107:stderr [
> > > error:
> > > Cannot access storage file
> > > '/mnt/mcd/AG_BioInformatik/Technik/software_und_treiber/linux/ubu
> > > ntu/
> > > ubuntu-18.04.4-live-server-amd64.iso': No such file or directory
> > > ]
> > > Jul 20 17:04:09 [10688] ha-idg-2       lrmd:   notice:
> > > operation_finished:      vm_nextcloud_start_0:29107:stderr [ ocf-
> > > exit-reason:Failed to start virtual domain vm_nextcloud. ]
> > > Jul 20 17:04:09 [10688] ha-idg-2       lrmd:   notice:
> > > log_finished:    finished - rsc:vm_nextcloud action:start
> > > call_id:192
> > > pid:29107 exit-code:1 exec-time:581ms queue-time:0ms
> > > start on ha-idg-2 failed
> > 
> > The start failed ...
> > 
> > > Jul 20 17:05:32 [10691] ha-idg-2       crmd:     info:
> > > do_dc_takeover:  Taking over DC status for this partition
> > > ha-idg-1 stopped pacemaker
> > 
> > Since the ha-idg-2 is now shutting down, ha-idg-1 becomes DC.
> > 
> > > Jul 20 17:05:33 [10690] ha-idg-2    pengine:  warning:
> > > unpack_rsc_op_failure:   Processing failed migrate_to of
> > > vm_nextcloud
> > > on ha-idg-1: unknown error | rc=1
> > > Jul 20 17:05:33 [10690] ha-idg-2    pengine:  warning:
> > > unpack_rsc_op_failure:   Processing failed start of vm_nextcloud
> > > on
> > > ha-idg-2: unknown error | rc
> > > 
> > > Jul 20 17:05:33 [10690] ha-idg-2    pengine:     info:
> > > native_color:    Resource vm_nextcloud cannot run anywhere
> > > logical
> > > 
> > > Jul 20 17:05:33 [10690] ha-idg-2    pengine:  warning:
> > > custom_action:   Action vm_nextcloud_stop_0 on ha-idg-1 is
> > > unrunnable
> > > (pending)
> > > ???
> > 
> > So this appears to be the problem. From these logs I would guess
> > the
> > successful stop on ha-idg-1 did not get written to the CIB for some
> > reason. I'd look at the pe input from this transition on ha-idg-2
> > to
> > confirm that.
> > 
> > Without the DC knowing about the stop, it tries to schedule a new
> > one,
> > but the node is shutting down so it can't do it, which means it has
> > to
> > be fenced.
> > 
> > > Jul 20 17:05:35 [10690] ha-idg-2    pengine:  warning:
> > > custom_action:   Action vm_nextcloud_stop_0 on ha-idg-1 is
> > > unrunnable
> > > (offline)
> > > Jul 20 17:05:35 [10690] ha-idg-2    pengine:  warning:
> > > pe_fence_node:   Cluster node ha-idg-1 will be fenced: resource
> > > actions are unrunnable
> > > Jul 20 17:05:35 [10690] ha-idg-2    pengine:  warning:
> > > stage6:  Scheduling Node ha-idg-1 for STONITH
> > > Jul 20 17:05:35 [10690] ha-idg-2    pengine:     info:
> > > native_stop_constraints: vm_nextcloud_stop_0 is implicit after
> > > ha-
> > > idg-1 is fenced
> > > Jul 20 17:05:35 [10690] ha-idg-2    pengine:   notice:
> > > LogNodeActions:   * Fence (Off) ha-idg-1 'resource actions are
> > > unrunnable'
> > > 
> > > 
> > > Why does it say "Jul 20 17:05:35 [10690] ha-idg-
> > > 2    pengine:  warning: custom_action:   Action
> > > vm_nextcloud_stop_0
> > > on ha-idg-1 is unrunnable (offline)" although
> > > "Jul 20 17:04:06 [23768] ha-idg-1       crmd:   notice:
> > > process_lrm_event:       Result of stop operation for
> > > vm_nextcloud on
> > > ha-idg-1: 0 (ok) | call=3197 key=vm_nextcloud_stop_0
> > > confirmed=true
> > > cib-update=5960"
> > > says that stop was ok ?
> 
> I'm stll digging in the logs trying to understand what happened.
> What i'm wondering abaout:
> 
> Jul 20 17:04:26 [23768] ha-idg-1       crmd:   notice:
> run_graph:       Transition 4515 (Complete=10, Pending=0, Fired=0,
> Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-
> 3117.bz2): Complete
> 
> Jul 20 17:05:29 [23768] ha-idg-1       crmd:     info:
> abort_transition_graph:  Transition 4515 aborted by status-
> 1084777482-shutdown doing create shutdown=1595257529: Transient
> attribute change | cib=2.16197.72
> source=abort_unless_down:317   path=/cib/status/node_state[@id='10847
> 77482']/transient_attributes[@id='1084777482']/instance_attributes[@i
> d='status-1084777482'] complete=true
> 
> 
> A transition is completed and three secons later aborted. How can
> that be?

The most recent transition is aborted, but since all its actions are
complete, the only effect is to trigger a new transition.

We should probably rephrase the log message. In fact, the whole
"transition" terminology is kind of obscure. It's hard to come up with
something better though.

> 
> Bernd
> Helmholtz Zentrum M?nchen
> 
> Helmholtz Zentrum Muenchen
> Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
> Ingolstaedter Landstr. 1
> 85764 Neuherberg
> www.helmholtz-muenchen.de
> Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
> Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin
> Guenther
> Registergericht: Amtsgericht Muenchen HRB 6466
> USt-IdNr: DE 129521671
-- 
Ken Gaillot <kgaillot at redhat.com>


From arvidjaar at gmail.com  Tue Aug 11 02:48:15 2020
From: arvidjaar at gmail.com (Andrei Borzenkov)
Date: Tue, 11 Aug 2020 09:48:15 +0300
Subject: [ClusterLabs] Automatic recover from split brain ?
In-Reply-To: <b899e0c9-0844-7643-5f07-59e47c9a806d@le-vert.net>
References: <b899e0c9-0844-7643-5f07-59e47c9a806d@le-vert.net>
Message-ID: <cabc2669-2b81-43a8-2a86-378986cf35ab@gmail.com>

08.08.2020 13:10, Adam C?cile ?????:
> Hello,
> 
> 
> I'm experiencing issue with corosync/pacemaker running on Debian Buster.
> Cluster has three nodes running in VMWare virtual machine and the
> cluster fails when VEEAM backups the virtual machine (I know it's doing
> bad things, like freezing completely the VM for a few minutes to make
> disk snapshot).
> 
> My biggest issue is that once the backup has been completed, the cluster
> stays in split brain state, and I'd like it to heal itself. Here current
> status:
> 
> 
> One node is isolated:
> 
> Stack: corosync
> Current DC: host2.domain.com (version 2.0.1-9e909a5bdd) - partition
> WITHOUT quorum
> Last updated: Sat Aug? 8 11:59:46 2020
> Last change: Fri Jul 24 07:18:12 2020 by root via cibadmin on
> host1.domain.com
> 
> 3 nodes configured
> 6 resources configured
> 
> Online: [ host2.domain.com ]
> OFFLINE: [ host3.domain.com host1.domain.com ]
> 
> 
> Two others are seeing each others:
> 
> Stack: corosync
> Current DC: host3.domain.com (version 2.0.1-9e909a5bdd) - partition with
> quorum
> Last updated: Sat Aug? 8 12:07:56 2020
> Last change: Fri Jul 24 07:18:12 2020 by root via cibadmin on
> host1.domain.com
> 
> 3 nodes configured
> 6 resources configured
> 
> Online: [ host3.domain.com host1.domain.com ]
> OFFLINE: [ host2.domain.com ]
> 

Show your full configuration including defined STONITH resources and
cluster options (most importantly, no-quorum-policy and stonith-enabled).

> 
> The problem is that one of the resources is a floating IP address which
> is currently assigned to two different hosts...
> 

Of course - each partition assumes another partition is dead and so it
is free to take over remaining resources.

> 
> Can you help me configuring the cluster correctly so this cannot occurs ?
> 

Define "correctly".

The most straightforward text book answer - you need to have STONITH
resources that will eliminate "lost" node. But your lost node is in the
middle of performing backup. Eliminating it may invalidate backup being
created.

So another answer would be - put cluster in maintenance mode, perform
backup, resume normal operation. Usually backup software allows hooks to
be executed before and after backup. It may work too.

Or find a way to not freeze VM during backup ... e.g. by using different
backup method?

From tojeline at redhat.com  Tue Aug 11 03:22:25 2020
From: tojeline at redhat.com (Tomas Jelinek)
Date: Tue, 11 Aug 2020 09:22:25 +0200
Subject: [ClusterLabs] How to specify which IP pcs should use?
In-Reply-To: <20200810142153.603f56dd@hydra.home.zxz.li>
References: <20200810142153.603f56dd@hydra.home.zxz.li>
Message-ID: <9d6eda77-6ea3-d3e6-95ac-e4068e8dbcd3@redhat.com>

Hi Mariusz,

You haven't mention pcs version you are running. Based on you mentioning 
running Pacemaker 2, I suppose you are running pcs 0.10.x. The text 
bellow applies to pcs 0.10.x.

Pcs doesn't depend on or use corosync.conf when connecting to other 
nodes. The reason is pcs must be able to connect to nodes not specified 
in corosync.conf, e.g. when there is no cluster created yet.

Instead, pcs has its own config file mapping node names to addresses. 
The easiest way to set it is to specify an address for each node in 'pcs 
host auth' command like this:
pcs host auth <node1> addr=<ip1> <node2> addr=<ip2> ...

Specifying addresses is not mandatory. If the addresses are omitted, pcs 
uses node names as addresses. See man pcs for more details.

To fix your issues, run 'pcs host auth' and specify all nodes and their 
addresses. Running the command on one node of your cluster should be enough.


Regards,
Tomas


Dne 10. 08. 20 v 14:21 Mariusz Gronczewski napsal(a):
> Hi,
> 
> Pacemaker 2, the current setup is
> 
> * management network with host's hostname resolving to host's
>    management IP
> * cluster network for Pacemaker/Corosync communication
> * corosync set up with node name and IP of the cluster network
> 
> pcs status shows both nodes online, added config syncs to the other
> node etc. but pcs cluster status shows one node being offline.
> 
> After a look in firewall logs it appears all of the communication is
> going just fine on the cluster network but PCS tries to talk with PCSD
> on port 2224 via *management* network instead of using IP set as
> ring0_addr in corosync
> 
> Is "just use host's hostname regardless of config" something normal ?
> Is there a separate setting to pcs about which IP it should use ?
> 
> Regards
> 


From acecile at le-vert.net  Tue Aug 11 03:34:40 2020
From: acecile at le-vert.net (=?UTF-8?Q?Adam_C=c3=a9cile?=)
Date: Tue, 11 Aug 2020 09:34:40 +0200
Subject: [ClusterLabs] Automatic recover from split brain ?
In-Reply-To: <cabc2669-2b81-43a8-2a86-378986cf35ab@gmail.com>
References: <b899e0c9-0844-7643-5f07-59e47c9a806d@le-vert.net>
 <cabc2669-2b81-43a8-2a86-378986cf35ab@gmail.com>
Message-ID: <69719d4e-d990-a2ba-4347-0e679b582d3f@le-vert.net>

On 8/11/20 8:48 AM, Andrei Borzenkov wrote:
> 08.08.2020 13:10, Adam C?cile ?????:
>> Hello,
>>
>>
>> I'm experiencing issue with corosync/pacemaker running on Debian Buster.
>> Cluster has three nodes running in VMWare virtual machine and the
>> cluster fails when VEEAM backups the virtual machine (I know it's doing
>> bad things, like freezing completely the VM for a few minutes to make
>> disk snapshot).
>>
>> My biggest issue is that once the backup has been completed, the cluster
>> stays in split brain state, and I'd like it to heal itself. Here current
>> status:
>>
>>
>> One node is isolated:
>>
>> Stack: corosync
>> Current DC: host2.domain.com (version 2.0.1-9e909a5bdd) - partition
>> WITHOUT quorum
>> Last updated: Sat Aug? 8 11:59:46 2020
>> Last change: Fri Jul 24 07:18:12 2020 by root via cibadmin on
>> host1.domain.com
>>
>> 3 nodes configured
>> 6 resources configured
>>
>> Online: [ host2.domain.com ]
>> OFFLINE: [ host3.domain.com host1.domain.com ]
>>
>>
>> Two others are seeing each others:
>>
>> Stack: corosync
>> Current DC: host3.domain.com (version 2.0.1-9e909a5bdd) - partition with
>> quorum
>> Last updated: Sat Aug? 8 12:07:56 2020
>> Last change: Fri Jul 24 07:18:12 2020 by root via cibadmin on
>> host1.domain.com
>>
>> 3 nodes configured
>> 6 resources configured
>>
>> Online: [ host3.domain.com host1.domain.com ]
>> OFFLINE: [ host2.domain.com ]
>>
> Show your full configuration including defined STONITH resources and
> cluster options (most importantly, no-quorum-policy and stonith-enabled).

Hello,

Stonith is disabled and I tried various settings for no-quorum-policy.

>> The problem is that one of the resources is a floating IP address which
>> is currently assigned to two different hosts...
>>
> Of course - each partition assumes another partition is dead and so it
> is free to take over remaining resources.
I understand that but I still don't get why once all nodes are back 
online, the cluster does not heal from resources running one multiple hosts.
>
>> Can you help me configuring the cluster correctly so this cannot occurs ?
>>
> Define "correctly".
>
> The most straightforward text book answer - you need to have STONITH
> resources that will eliminate "lost" node. But your lost node is in the
> middle of performing backup. Eliminating it may invalidate backup being
> created.
Yeah but well, no. Killing the node is worse, sensible services are 
already running in clustering mode at application level so they do not 
rely on corosync. Basically corosync is providing a floating IP for some 
external non critical access and starting systemd timers that are 
pointless to be run on multiple hosts. Nothing critical here.
>
> So another answer would be - put cluster in maintenance mode, perform
> backup, resume normal operation. Usually backup software allows hooks to
> be executed before and after backup. It may work too.
This in indeed something I might look at, but again, for my trivial 
needs it sounds a bit overkill to me.
> Or find a way to not freeze VM during backup ... e.g. by using different
> backup method?

Or tweaks some network settings so corosync does not consider the node 
as being dead too soon ? Backup won't last more than 2 minutes and the 
freeze is usually way below. I can definitely leave with cluster state 
being unknown for a couple of minutes. Is that possible ?

Removing VEEAM is indeed my last option and the one I used so far, but 
this time I was hoping someone else would be experiencing the same issue 
and could help me fixing that in a clean way.


Thanks

> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/



From gandalf.levert at gmail.com  Sun Aug  9 15:14:00 2020
From: gandalf.levert at gmail.com (Adam Cecile)
Date: Sun, 9 Aug 2020 21:14:00 +0200
Subject: [ClusterLabs] Automatic recover from split brain ?
Message-ID: <926856b0-bd63-ab6d-f617-f0ed952b3904@gmail.com>

Hello,


I'm experiencing issue with corosync/pacemaker running on Debian Buster. 
Cluster has three nodes running in VMWare virtual machine and the 
cluster fails when VEEAM backups the virtual machine (I know it's doing 
bad things, like freezing completely the VM for a few minutes to make 
disk snapshot).

My biggest issue is that once the backup has been completed, the cluster 
stays in split brain state, and I'd like it to heal itself. Here current 
status:


One node is isolated:

Stack: corosync
Current DC: host2.domain.com (version 2.0.1-9e909a5bdd) - partition 
WITHOUT quorum
Last updated: Sat Aug? 8 11:59:46 2020
Last change: Fri Jul 24 07:18:12 2020 by root via cibadmin on 
host1.domain.com

3 nodes configured
6 resources configured

Online: [ host2.domain.com ]
OFFLINE: [ host3.domain.com host1.domain.com ]


Two others are seeing each others:

Stack: corosync
Current DC: host3.domain.com (version 2.0.1-9e909a5bdd) - partition with 
quorum
Last updated: Sat Aug? 8 12:07:56 2020
Last change: Fri Jul 24 07:18:12 2020 by root via cibadmin on 
host1.domain.com

3 nodes configured
6 resources configured

Online: [ host3.domain.com host1.domain.com ]
OFFLINE: [ host2.domain.com ]


The problem is that one of the resources is a floating IP address which 
is currently assigned to two different hosts...


Can you help me configuring the cluster correctly so this cannot occurs ?


Thanks in advance,

Adam.




From arvidjaar at gmail.com  Tue Aug 11 14:30:54 2020
From: arvidjaar at gmail.com (Andrei Borzenkov)
Date: Tue, 11 Aug 2020 21:30:54 +0300
Subject: [ClusterLabs] Automatic recover from split brain ?
In-Reply-To: <69719d4e-d990-a2ba-4347-0e679b582d3f@le-vert.net>
References: <b899e0c9-0844-7643-5f07-59e47c9a806d@le-vert.net>
 <cabc2669-2b81-43a8-2a86-378986cf35ab@gmail.com>
 <69719d4e-d990-a2ba-4347-0e679b582d3f@le-vert.net>
Message-ID: <16b699d9-4c97-72a3-afd5-d56bd4ec1c28@gmail.com>

11.08.2020 10:34, Adam C?cile ?????:
> On 8/11/20 8:48 AM, Andrei Borzenkov wrote:
>> 08.08.2020 13:10, Adam C?cile ?????:
>>> Hello,
>>>
>>>
>>> I'm experiencing issue with corosync/pacemaker running on Debian Buster.
>>> Cluster has three nodes running in VMWare virtual machine and the
>>> cluster fails when VEEAM backups the virtual machine (I know it's doing
>>> bad things, like freezing completely the VM for a few minutes to make
>>> disk snapshot).
>>>
>>> My biggest issue is that once the backup has been completed, the cluster
>>> stays in split brain state, and I'd like it to heal itself. Here current
>>> status:
>>>
>>>
>>> One node is isolated:
>>>
>>> Stack: corosync
>>> Current DC: host2.domain.com (version 2.0.1-9e909a5bdd) - partition
>>> WITHOUT quorum
>>> Last updated: Sat Aug? 8 11:59:46 2020
>>> Last change: Fri Jul 24 07:18:12 2020 by root via cibadmin on
>>> host1.domain.com
>>>
>>> 3 nodes configured
>>> 6 resources configured
>>>
>>> Online: [ host2.domain.com ]
>>> OFFLINE: [ host3.domain.com host1.domain.com ]
>>>
>>>
>>> Two others are seeing each others:
>>>
>>> Stack: corosync
>>> Current DC: host3.domain.com (version 2.0.1-9e909a5bdd) - partition with
>>> quorum
>>> Last updated: Sat Aug? 8 12:07:56 2020
>>> Last change: Fri Jul 24 07:18:12 2020 by root via cibadmin on
>>> host1.domain.com
>>>
>>> 3 nodes configured
>>> 6 resources configured
>>>
>>> Online: [ host3.domain.com host1.domain.com ]
>>> OFFLINE: [ host2.domain.com ]
>>>
>> Show your full configuration including defined STONITH resources and
>> cluster options (most importantly, no-quorum-policy and stonith-enabled).
> 
> Hello,
> 
> Stonith is disabled and I tried various settings for no-quorum-policy.
> 
>>> The problem is that one of the resources is a floating IP address which
>>> is currently assigned to two different hosts...
>>>
>> Of course - each partition assumes another partition is dead and so it
>> is free to take over remaining resources.
> I understand that but I still don't get why once all nodes are back
> online, the cluster does not heal from resources running one multiple
> hosts.

In my limited testing it does - after nodes see each other pacemaker
sees resources active on multiple nodes and tries to fix it. This is
with pacemaker 2.0.3. Check logs on all nodes what happens around time
node becomes alive again.

>>
>>> Can you help me configuring the cluster correctly so this cannot
>>> occurs ?
>>>
>> Define "correctly".
>>
>> The most straightforward text book answer - you need to have STONITH
>> resources that will eliminate "lost" node. But your lost node is in the
>> middle of performing backup. Eliminating it may invalidate backup being
>> created.
> Yeah but well, no. Killing the node is worse, sensible services are
> already running in clustering mode at application level so they do not
> rely on corosync. Basically corosync is providing a floating IP for some
> external non critical access and starting systemd timers that are
> pointless to be run on multiple hosts. Nothing critical here.
>>
>> So another answer would be - put cluster in maintenance mode, perform
>> backup, resume normal operation. Usually backup software allows hooks to
>> be executed before and after backup. It may work too.
> This in indeed something I might look at, but again, for my trivial
> needs it sounds a bit overkill to me.
>> Or find a way to not freeze VM during backup ... e.g. by using different
>> backup method?
> 
> Or tweaks some network settings so corosync does not consider the node
> as being dead too soon ? Backup won't last more than 2 minutes and the
> freeze is usually way below. I can definitely leave with cluster state
> being unknown for a couple of minutes. Is that possible ?
> 

You probably can increase token time but that also directly affects how
fast pacemaker will start and also how fast it will react to node failures.

> Removing VEEAM is indeed my last option and the one I used so far, but
> this time I was hoping someone else would be experiencing the same issue
> and could help me fixing that in a clean way.
> 
> 
> Thanks
> 
>> _______________________________________________
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>
>> ClusterLabs home: https://www.clusterlabs.org/
> 
> 


From rohitsaini111.forum at gmail.com  Wed Aug 12 10:18:57 2020
From: rohitsaini111.forum at gmail.com (Rohit Saini)
Date: Wed, 12 Aug 2020 19:48:57 +0530
Subject: [ClusterLabs] Alerts for qdevice/qnetd/booth
Message-ID: <CA+ZsWTXN8srS-G-ONPvMiXk=8YqDdwgCMjn-3C+n7D85N5ASBg@mail.gmail.com>

Hi Team,

Question-1:
Similar to pcs alerts, do we have something similar for qdevice/qnetd? This
is to detect asynchronously if any of the member is unreachable/joined/left
and if that member is qdevice or qnetd.

Question-2:
Same above question for booth nodes and arbitrator. Is there any way to
receive events from booth daemon?

My main objective is to see if these daemons give events related to
their internal state transitions  and raise some alarms accordingly. For
example, boothd arbitrator is unreachable, ticket moved from x to y, etc.

Thanks,
Rohit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200812/b22d5c69/attachment.htm>

From jfriesse at redhat.com  Wed Aug 12 11:28:15 2020
From: jfriesse at redhat.com (Jan Friesse)
Date: Wed, 12 Aug 2020 17:28:15 +0200
Subject: [ClusterLabs] Alerts for qdevice/qnetd/booth
In-Reply-To: <CA+ZsWTXN8srS-G-ONPvMiXk=8YqDdwgCMjn-3C+n7D85N5ASBg@mail.gmail.com>
References: <CA+ZsWTXN8srS-G-ONPvMiXk=8YqDdwgCMjn-3C+n7D85N5ASBg@mail.gmail.com>
Message-ID: <942629ca-1035-32b4-2025-1ddd970e1b1a@redhat.com>

Hi Rohit,

Rohit Saini napsal(a):
> Hi Team,
> 
> Question-1:
> Similar to pcs alerts, do we have something similar for qdevice/qnetd? This

You mean pacemaker alerts right?

> is to detect asynchronously if any of the member is unreachable/joined/left
> and if that member is qdevice or qnetd.

Nope but actually shouldn't be that hard to implement. What exactly 
would you like to see there?

> 
> Question-2:
> Same above question for booth nodes and arbitrator. Is there any way to
> receive events from booth daemon?

Not directly (again, shouldn't be that hard to implement). But pacemaker 
alerts should be triggered when service changes state because of ticket 
grant/reject, isn't it?

> 
> My main objective is to see if these daemons give events related to
> their internal state transitions  and raise some alarms accordingly. For
> example, boothd arbitrator is unreachable, ticket moved from x to y, etc.

I don't think "boothd arbitrator is unreachable" alert is really doable. 
Ticket moved from x to y would be probably two alerts - 1. ticket 
rejected on X and 2. granted on Y.

Would you mind to elaborate a bit more on events you would like to see 
and potentially open issue for upstream project (or, if you have a RH 
subscription try to contact GSS, so I get more time to work on this issue).

Regards,
   Honza

> 
> Thanks,
> Rohit
> 
> 
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> ClusterLabs home: https://www.clusterlabs.org/
> 


From kgaillot at redhat.com  Wed Aug 12 14:57:07 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Wed, 12 Aug 2020 13:57:07 -0500
Subject: [ClusterLabs] Coming in Pacemaker 2.0.5: on-fail=demote /
 no-quorum-policy=demote
Message-ID: <b99f6ce763af5ad372c5583aead244c6419c4474.camel@redhat.com>

(Apologies if this shows up twice -- clusterlabs.org had a mail
configuration issue over the weekend, and I'm not sure if my original
message was permanently lost or will eventually show up)

Hi all,

Looking ahead to the Pacemaker 2.0.5 release expected at the end of
this year, here is a new feature already in the master branch.

When configuring resource operations, Pacemaker lets you set an "on-
fail" policy to specify whether to restart the resource, fence the
node, etc., if the operation fails. With 2.0.5, a new possible value
will be "demote", which will mean "demote this resource but do not
fully restart it".

"Demote" will be a valid value only for promote actions, and for
recurring monitors with "role" set to "Master".

Once the resource is demoted, it will be eligible for promotion again,
so if the promotion scores have not changed, a promote on the same node
may be attempted. If this is not desired, the agent can change the
promotion scores either in the failed monitor or the demote.

The intended use case is an application where a successful demote 
assures a well-functioning service, and a full restart would be
unnecessarily heavyweight. A large database might be an example.

Similarly, Pacemaker offers the cluster-wide "no-quorum-policy" option
to specify what happens to resources when quorum is lost (the default
being to stop them). With 2.0.5, "demote" will be a possible value here
as well, and will mean "demote all promotable resources and stop all
other resources".

The intended use case is an application that cannot cause any harm
after being demoted, and may be useful in a demoted role even if there
is no quorum. A database that operates read-only when demoted and
doesn't depend on any non-promotable resources might be an example.

Happy clustering :)
-- 
Ken Gaillot <kgaillot at redhat.com>


From rohitsaini111.forum at gmail.com  Thu Aug 13 03:02:45 2020
From: rohitsaini111.forum at gmail.com (Rohit Saini)
Date: Thu, 13 Aug 2020 12:32:45 +0530
Subject: [ClusterLabs] Alerts for qdevice/qnetd/booth
In-Reply-To: <942629ca-1035-32b4-2025-1ddd970e1b1a@redhat.com>
References: <CA+ZsWTXN8srS-G-ONPvMiXk=8YqDdwgCMjn-3C+n7D85N5ASBg@mail.gmail.com>
 <942629ca-1035-32b4-2025-1ddd970e1b1a@redhat.com>
Message-ID: <CA+ZsWTVEiEJmgk+7Uydq+GM_TwO+CUkBqbKyF2g6BDW76omz6w@mail.gmail.com>

Hi Honza,
Thanks for your reply. Please find the attached image below:

[image: image.png]

Yes, I am talking about pacemaker alerts only.

Please find my suggestions/requirements below:

*Booth:*
1. Node5 booth-arbitrator should be able to give event when any of the
booth node joins or leaves. booth-ip can be passed in event.
2. Event when booth-arbitrator is up successfully and has started
monitoring the booth nodes.
2. Geo site booth should be able to give event when its booth peers
joins/leaves. For example, Geo site1 gives an event when node5
booth-arbitrator joins/leaves OR site2 booth joins/leaves.  booth-ip can be
passed in event.
3. On ticket movements (revoke/grant), every booth node(Site1/2 and node5)
should give events.

Note: pacemaker alerts works in a cluster. Since, arbitrator is a
non-cluster node, not sure how exactly it will work there. But this is good
to have feature.

*Qnetd/Qdevice:*
This is similar to above.
1. Node5 qnetd should be able to raise an event when any of the cluster
node joins/leaves the quorum.
2. Event when qnetd is up successfully and has started monitoring the
cluster nodes
3. Cluster node should be able to give event when any of the quorum node
leaves/joins.

If you see on high level, then these are kind of node/resource events wrt
booth and qnetd/qdevice.

As of today wrt booth/qnetd, I don't see any provision where any of the
nodes gives any event when its peer leaves/joins. This makes it difficult
to know whether geo sites nodes can see booth-arbitrator or not. This is
true the other way around also where booth-arbitrator cannot see geo booth
sites.
I am not sure how others are doing it in today's deployment, but I see need
of monitoring of every other booth/qnet node. So that on basis of event,
appropriate alarms can be raised and action can be taken accordingly.

Please let me know if you agree on the usecases. I'll raise feature-request
on the pacemaker upstream project accordingly.

Thanks,
Rohit

On Wed, Aug 12, 2020 at 8:58 PM Jan Friesse <jfriesse at redhat.com> wrote:

> Hi Rohit,
>
> Rohit Saini napsal(a):
> > Hi Team,
> >
> > Question-1:
> > Similar to pcs alerts, do we have something similar for qdevice/qnetd?
> This
>
> You mean pacemaker alerts right?
>
> > is to detect asynchronously if any of the member is
> unreachable/joined/left
> > and if that member is qdevice or qnetd.
>
> Nope but actually shouldn't be that hard to implement. What exactly
> would you like to see there?
>
> >
> > Question-2:
> > Same above question for booth nodes and arbitrator. Is there any way to
> > receive events from booth daemon?
>
> Not directly (again, shouldn't be that hard to implement). But pacemaker
> alerts should be triggered when service changes state because of ticket
> grant/reject, isn't it?
>
> >
> > My main objective is to see if these daemons give events related to
> > their internal state transitions  and raise some alarms accordingly. For
> > example, boothd arbitrator is unreachable, ticket moved from x to y, etc.
>
> I don't think "boothd arbitrator is unreachable" alert is really doable.
> Ticket moved from x to y would be probably two alerts - 1. ticket
> rejected on X and 2. granted on Y.
>
> Would you mind to elaborate a bit more on events you would like to see
> and potentially open issue for upstream project (or, if you have a RH
> subscription try to contact GSS, so I get more time to work on this issue).
>
> Regards,
>    Honza
>
> >
> > Thanks,
> > Rohit
> >
> >
> >
> > _______________________________________________
> > Manage your subscription:
> > https://lists.clusterlabs.org/mailman/listinfo/users
> >
> > ClusterLabs home: https://www.clusterlabs.org/
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200813/53d1d8a8/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 75435 bytes
Desc: not available
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200813/53d1d8a8/attachment-0001.png>

From jfriesse at redhat.com  Thu Aug 13 03:33:46 2020
From: jfriesse at redhat.com (Jan Friesse)
Date: Thu, 13 Aug 2020 09:33:46 +0200
Subject: [ClusterLabs] Alerts for qdevice/qnetd/booth
In-Reply-To: <CA+ZsWTVEiEJmgk+7Uydq+GM_TwO+CUkBqbKyF2g6BDW76omz6w@mail.gmail.com>
References: <CA+ZsWTXN8srS-G-ONPvMiXk=8YqDdwgCMjn-3C+n7D85N5ASBg@mail.gmail.com>
 <942629ca-1035-32b4-2025-1ddd970e1b1a@redhat.com>
 <CA+ZsWTVEiEJmgk+7Uydq+GM_TwO+CUkBqbKyF2g6BDW76omz6w@mail.gmail.com>
Message-ID: <4ad3e321-4204-859f-cc2b-18f67e1faf9e@redhat.com>

Hi Rohit,

> Hi Honza,
> Thanks for your reply. Please find the attached image below:
> 
> [image: image.png]
> 
> Yes, I am talking about pacemaker alerts only.
> 
> Please find my suggestions/requirements below:
> 
> *Booth:*
> 1. Node5 booth-arbitrator should be able to give event when any of the
> booth node joins or leaves. booth-ip can be passed in event.

This is not how booth works. Ticket leader (so site booth, never 
arbitrator) executes election and get replies from other 
sites/arbitrator. Follower executes election when leader hasn't for 
configured timeout.

What I want to say is, that there is no "membership" - as in (for 
example) corosync fashion.

The best we could get is the rough estimation based on election 
request/replies.

> 2. Event when booth-arbitrator is up successfully and has started
> monitoring the booth nodes.

This is basically start of service. I think it's doable with small 
change in unit file (something like 
https://northernlightlabs.se/2014-07-05/systemd-status-mail-on-unit-failure.html)

> 2. Geo site booth should be able to give event when its booth peers
> joins/leaves. For example, Geo site1 gives an event when node5
> booth-arbitrator joins/leaves OR site2 booth joins/leaves.  booth-ip can be
> passed in event.
> 3. On ticket movements (revoke/grant), every booth node(Site1/2 and node5)
> should give events.

That would be doable

> 
> Note: pacemaker alerts works in a cluster. Since, arbitrator is a
> non-cluster node, not sure how exactly it will work there. But this is good
> to have feature.
> 
> *Qnetd/Qdevice:*
> This is similar to above.
> 1. Node5 qnetd should be able to raise an event when any of the cluster
> node joins/leaves the quorum.

Doable

> 2. Event when qnetd is up successfully and has started monitoring the
> cluster nodes

Qnetd itself is not monitoring qdevice nodes (it doesn't have list of 
nodes). It monitors node status after node joins (= it would be possible 
to trigger event on leave). So that may be enough.

> 3. Cluster node should be able to give event when any of the quorum node
> leaves/joins.

You mean qdevice should be able to trigger event when connected to qnetd?

> 
> If you see on high level, then these are kind of node/resource events wrt
> booth and qnetd/qdevice.

Yeah

> 
> As of today wrt booth/qnetd, I don't see any provision where any of the
> nodes gives any event when its peer leaves/joins. This makes it difficult
> to know whether geo sites nodes can see booth-arbitrator or not. This is

Got it. That's exactly what would be really problematic to implement, 
because of no "membership" in booth. It would be, however, possible to 
implement message when ticket was granted/rejected and have a list of 
other booths replies and what was their votes.

> true the other way around also where booth-arbitrator cannot see geo booth
> sites.
> I am not sure how others are doing it in today's deployment, but I see need
> of monitoring of every other booth/qnet node. So that on basis of event,
> appropriate alarms can be raised and action can be taken accordingly.
> 
> Please let me know if you agree on the usecases. I'll raise feature-request

I can agree on usecases, but (especially with booth) there are technical 
problems on realizing them.

> on the pacemaker upstream project accordingly.

Please use booth (https://github.com/ClusterLabs/booth) and qdevice 
(https://github.com/corosync/corosync-qdevice) upstream rather than 
pacemaker, because these requests has really nothing to do with pcmk.

Regards,
   honza

> 
> Thanks,
> Rohit
> 
> On Wed, Aug 12, 2020 at 8:58 PM Jan Friesse <jfriesse at redhat.com> wrote:
> 
>> Hi Rohit,
>>
>> Rohit Saini napsal(a):
>>> Hi Team,
>>>
>>> Question-1:
>>> Similar to pcs alerts, do we have something similar for qdevice/qnetd?
>> This
>>
>> You mean pacemaker alerts right?
>>
>>> is to detect asynchronously if any of the member is
>> unreachable/joined/left
>>> and if that member is qdevice or qnetd.
>>
>> Nope but actually shouldn't be that hard to implement. What exactly
>> would you like to see there?
>>
>>>
>>> Question-2:
>>> Same above question for booth nodes and arbitrator. Is there any way to
>>> receive events from booth daemon?
>>
>> Not directly (again, shouldn't be that hard to implement). But pacemaker
>> alerts should be triggered when service changes state because of ticket
>> grant/reject, isn't it?
>>
>>>
>>> My main objective is to see if these daemons give events related to
>>> their internal state transitions  and raise some alarms accordingly. For
>>> example, boothd arbitrator is unreachable, ticket moved from x to y, etc.
>>
>> I don't think "boothd arbitrator is unreachable" alert is really doable.
>> Ticket moved from x to y would be probably two alerts - 1. ticket
>> rejected on X and 2. granted on Y.
>>
>> Would you mind to elaborate a bit more on events you would like to see
>> and potentially open issue for upstream project (or, if you have a RH
>> subscription try to contact GSS, so I get more time to work on this issue).
>>
>> Regards,
>>     Honza
>>
>>>
>>> Thanks,
>>> Rohit
>>>
>>>
>>>
>>> _______________________________________________
>>> Manage your subscription:
>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>> ClusterLabs home: https://www.clusterlabs.org/
>>>
>>
>>
> 


From luckydogxf at gmail.com  Thu Aug 13 04:33:41 2020
From: luckydogxf at gmail.com (luckydog xf)
Date: Thu, 13 Aug 2020 16:33:41 +0800
Subject: [ClusterLabs] Pacemaker-remote: Connection to cluster failed:
 Transport endpoint is not connected
Message-ID: <CAAoCYOzb_xXRxH8NO6OmAxW36htNDjY0HuGyoeOSUdbqHxS16g@mail.gmail.com>

Hi, guys,

  I'm running SLES12 sp3 and pacemaker-remote-1.1.16-4.8.x86_64, a few
months ago, compute nodes of openstack are running well.  But today when I
setup  a new compute node, it's said,"
-------------
Aug 13 16:31:04 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:   notice:
lrmd_init_remote_tls_server:    Starting a tls listener on port 3121.
Aug 13 16:31:05 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:   notice:
bind_and_listen:        Listening on address ::
Aug 13 16:31:05 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:     info:
qb_ipcs_us_publish:     server name: cib_ro
Aug 13 16:31:05 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:     info:
qb_ipcs_us_publish:     server name: cib_rw
Aug 13 16:31:05 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:     info:
qb_ipcs_us_publish:     server name: cib_shm
Aug 13 16:31:05 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:     info:
qb_ipcs_us_publish:     server name: attrd
Aug 13 16:31:05 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:     info:
qb_ipcs_us_publish:     server name: stonith-ng
Aug 13 16:31:05 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:     info:
qb_ipcs_us_publish:     server name: crmd
Aug 13 16:31:05 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:     info:
main:   Starting
-----------
and after I run `crm_mon -1r`, two lines are appended of pacemaker.log

Aug 13 16:31:38 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:    error:
ipc_proxy_accept:       No ipc providers available for uid 0 gid 0
Aug 13 16:31:38 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:    error:
handle_new_connection:  Error in connection setup (43122-43151-15): Remote
I/O error (121)

And the output of `crm_mon -1r` is

Connection to cluster failed: Transport endpoint is not connected

My environment is almost the same with other servers. So what's up ?
Thanks,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200813/4f931114/attachment.htm>

From luckydogxf at gmail.com  Thu Aug 13 05:15:39 2020
From: luckydogxf at gmail.com (luckydog xf)
Date: Thu, 13 Aug 2020 17:15:39 +0800
Subject: [ClusterLabs] Pacemaker-remote: Connection to cluster failed:
 Transport endpoint is not connected
In-Reply-To: <CAAoCYOzb_xXRxH8NO6OmAxW36htNDjY0HuGyoeOSUdbqHxS16g@mail.gmail.com>
References: <CAAoCYOzb_xXRxH8NO6OmAxW36htNDjY0HuGyoeOSUdbqHxS16g@mail.gmail.com>
Message-ID: <CAAoCYOwn3uV2wxHWuBXQwGPBbYxNeYodHVmXiCjPbzvp5dkcZw@mail.gmail.com>

Probably I didn't configure  pacemaker resource properly, now it's OK.
---
crm configure show remote-db8-ca-3a-69-60-f4
node remote-db8-ca-3a-69-60-f4:remote \
        attributes OpenStack-role=compute standby=off
primitive remote-db8-ca-3a-69-60-f4 ocf:pacemaker:remote \
        params reconnect_interval=60s server=db8-ca-3a-69-60-f4.ipa.pthl.hk
\
        op monitor interval=30s \
        op start interval=0 timeout=60s \
        op stop interval=0 timeout=60s \
        meta target-role=Started
---

On Thu, Aug 13, 2020 at 4:33 PM luckydog xf <luckydogxf at gmail.com> wrote:

> Hi, guys,
>
>   I'm running SLES12 sp3 and pacemaker-remote-1.1.16-4.8.x86_64, a few
> months ago, compute nodes of openstack are running well.  But today when I
> setup  a new compute node, it's said,"
> -------------
> Aug 13 16:31:04 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:   notice:
> lrmd_init_remote_tls_server:    Starting a tls listener on port 3121.
> Aug 13 16:31:05 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:   notice:
> bind_and_listen:        Listening on address ::
> Aug 13 16:31:05 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:     info:
> qb_ipcs_us_publish:     server name: cib_ro
> Aug 13 16:31:05 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:     info:
> qb_ipcs_us_publish:     server name: cib_rw
> Aug 13 16:31:05 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:     info:
> qb_ipcs_us_publish:     server name: cib_shm
> Aug 13 16:31:05 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:     info:
> qb_ipcs_us_publish:     server name: attrd
> Aug 13 16:31:05 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:     info:
> qb_ipcs_us_publish:     server name: stonith-ng
> Aug 13 16:31:05 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:     info:
> qb_ipcs_us_publish:     server name: crmd
> Aug 13 16:31:05 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:     info:
> main:   Starting
> -----------
> and after I run `crm_mon -1r`, two lines are appended of pacemaker.log
>
> Aug 13 16:31:38 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:    error:
> ipc_proxy_accept:       No ipc providers available for uid 0 gid 0
> Aug 13 16:31:38 [43122] db8-ca-3a-69-60-f4 pacemaker_remoted:    error:
> handle_new_connection:  Error in connection setup (43122-43151-15): Remote
> I/O error (121)
>
> And the output of `crm_mon -1r` is
>
> Connection to cluster failed: Transport endpoint is not connected
>
> My environment is almost the same with other servers. So what's up ?
> Thanks,
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200813/d1e85666/attachment.htm>

From clusterlabs at t.poki.me  Thu Aug 13 06:32:32 2020
From: clusterlabs at t.poki.me (clusterlabs at t.poki.me)
Date: Thu, 13 Aug 2020 12:32:32 +0200
Subject: [ClusterLabs] Coming in Pacemaker 2.0.5: on-fail=demote /
 no-quorum-policy=demote
In-Reply-To: <b99f6ce763af5ad372c5583aead244c6419c4474.camel@redhat.com>
References: <b99f6ce763af5ad372c5583aead244c6419c4474.camel@redhat.com>
Message-ID: <03646c86-ae3e-c76c-c6cd-589ea2f5d97a@poki.me>

Hello,

wanted to point out one thing that occurred to me when thinking about
the paragraph below.

On 8/12/20 8:57 PM, Ken Gaillot wrote:
> Similarly, Pacemaker offers the cluster-wide "no-quorum-policy" option
> to specify what happens to resources when quorum is lost (the default
> being to stop them). With 2.0.5, "demote" will be a possible value here
> as well, and will mean "demote all promotable resources and stop all
> other resources".
> 
> The intended use case is an application that cannot cause any harm
> after being demoted, and may be useful in a demoted role even if there
> is no quorum. A database that operates read-only when demoted and
> doesn't depend on any non-promotable resources might be an example.

Perhaps not that expected corollary with this cluster-wide setting
(only acknowledged when the cluster is upgraded in its entirety?), if
I understand it correctly, is that previously promoted resource will
get stopped anyway once it depends on a simple resource that doesn't
specify "on-fail" on its own (putting global/resource defaults aside).

It is this implicit resource "composability" (a.k.a. resource trees,
with some tweaks applicable right at this "composed service" level)
idea of long declined RGManager (RIP) that is still quite an appealing
and natural way (despite having less expressive power in general) one
can think of composed services (i.e. the behaviour of a final 	
brings-me-value unit rather than sum of moving parts behind it).
Through the prism of this tree model, if it could be proved that no
other simple resources share dependency on any of the prerequisites
with this promoted resource to be demoted because of "on fail" event,
it would be intuitive to expect they will be kept running and hence
will prevent this promoted resource from consequentially being stopped
despite just a weaker form of its demotion is the first and foremost
choice requested by the user.  Similarly with clones and other
promotable prerequisites, except it might be wise to demote them as
well if it would not conflict with demotion of the dependent promoted
resource that just suffered a failure.

Rationale for this is that prerequisite resource _solely_ consumed
  by
resources that don't need quorum as well hardly needs quorum on its own,
otherwise this is a conflicting/fishy configuration in some way.
(It would likewise be interesting and configuration-proofing-friendly
to investigate such "composing rules of soundness".)

I see there are some practical limits to these semi-recursive and
potentially explosive graph problems, but there's also a question
what's an intuitively expected behaviour, and possible disconnect
could at least be addressed in the documentation.

[Alternatively, some kind of "back-propagation" (intuitively,
"this terminal resource has powers to steer the behaviour of its
prerequisites, as the configuration of this terminal resource is
what's wanted by the outer surroundings of this box, afterall)
flag could be devised such that it would override any behaviour of
the prerequisite resources on a subset of conflicting options, like
on-fail, given that there is no conflict with any other resource
dependent on the same resource.]

Thanks & cheers

-- 
poki

From clusterlabs at t.poki.me  Thu Aug 13 11:34:30 2020
From: clusterlabs at t.poki.me (clusterlabs at t.poki.me)
Date: Thu, 13 Aug 2020 17:34:30 +0200
Subject: [ClusterLabs] Coming in Pacemaker 2.0.5: on-fail=demote /
 no-quorum-policy=demote
In-Reply-To: <03646c86-ae3e-c76c-c6cd-589ea2f5d97a@poki.me>
References: <b99f6ce763af5ad372c5583aead244c6419c4474.camel@redhat.com>
 <03646c86-ae3e-c76c-c6cd-589ea2f5d97a@poki.me>
Message-ID: <7ee8dae1-3f73-80b5-a36c-0b3f6c311b5a@poki.me>

Sorry, followed too much into the intermixing of two rather unrelated
things, when failure occurs and when quorum is lost.  I've meant to
dedicate the comment solely to the latter, but managed to cross that
line.  Corrections below:

On 8/13/20 12:32 PM, clusterlabs at t.poki.me wrote:
> wanted to point out one thing that occurred to me when thinking about
> the paragraph below.
> 
> On 8/12/20 8:57 PM, Ken Gaillot wrote:
>> Similarly, Pacemaker offers the cluster-wide "no-quorum-policy" option
>> to specify what happens to resources when quorum is lost (the default
>> being to stop them). With 2.0.5, "demote" will be a possible value here
>> as well, and will mean "demote all promotable resources and stop all
>> other resources".
>>
>> The intended use case is an application that cannot cause any harm
>> after being demoted, and may be useful in a demoted role even if there
>> is no quorum. A database that operates read-only when demoted and
>> doesn't depend on any non-promotable resources might be an example.
> 
> Perhaps not that expected corollary with this cluster-wide setting
> (only acknowledged when the cluster is upgraded in its entirety?), if
> I understand it correctly, is that previously promoted resource will
> get stopped anyway once it depends on a simple resource that doesn't
> specify "on-fail" on its own (putting global/resource defaults aside).

Scratch trailing " that ...", there's no individual per-resource
(nor per-resource-operation) override.

Would be, though, interesting to consider allowing for an escape
"no-quorum-policy=as-active-state-failure" that would turn loss
of quorum into started/promoted/demoted state maintenance failure,
which could then trigger individual on-fail response.  Who knows.

> It is this implicit resource "composability" (a.k.a. resource trees,
> with some tweaks applicable right at this "composed service" level)
> idea of long declined RGManager (RIP) that is still quite an appealing
> and natural way (despite having less expressive power in general) one
> can think of composed services (i.e. the behaviour of a final
> brings-me-value unit rather than sum of moving parts behind it).
> Through the prism of this tree model, if it could be proved that no
> other simple resources share dependency on any of the prerequisites
> with this promoted resource to be demoted because of "on fail" event,

s/on fail/quorum lost/

> it would be intuitive to expect they will be kept running and hence
> will prevent this promoted resource from consequentially being stopped
> despite just a weaker form of its demotion is the first and foremost
> choice requested by the user.? Similarly with clones and other
> promotable prerequisites, except it might be wise to demote them as
> well if it would not conflict with demotion of the dependent promoted
> resource that just suffered a failure.
> 
> Rationale for this is that prerequisite resource _solely_ consumed by
> resources that don't need quorum as well hardly needs quorum on its own,
> otherwise this is a conflicting/fishy configuration in some way.
> (It would likewise be interesting and configuration-proofing-friendly
> to investigate such "composing rules of soundness".)
> 
> I see there are some practical limits to these semi-recursive and
> potentially explosive graph problems, but there's also a question
> what's an intuitively expected behaviour, and possible disconnect
> could at least be addressed in the documentation.
> 
> [Alternatively, some kind of "back-propagation" (intuitively,
> "this terminal resource has powers to steer the behaviour of its
> prerequisites, as the configuration of this terminal resource is
> what's wanted by the outer surroundings of this box, afterall)
> flag could be devised such that it would override any behaviour of
> the prerequisite resources on a subset of conflicting options, like
> on-fail, given that there is no conflict with any other resource

Again, scrach "like on-fail", in this case it would perhaps mean
altering individual yet externally not directly configurable approach
of Pacemaker towards the resource in particular circumstance.

(It's part of the conceivability problem amongst different audiences,
since not every internal flag/tracking and conditional handling maps
to user configurable item -- main pillar of explaining the mechanics
to users -- in particular context.)

> dependent on the same resource.]
> 
> Thanks & cheers

-- 
poki

From rohitsaini111.forum at gmail.com  Fri Aug 14 00:22:48 2020
From: rohitsaini111.forum at gmail.com (Rohit Saini)
Date: Fri, 14 Aug 2020 09:52:48 +0530
Subject: [ClusterLabs] Alerts for qdevice/qnetd/booth
In-Reply-To: <4ad3e321-4204-859f-cc2b-18f67e1faf9e@redhat.com>
References: <CA+ZsWTXN8srS-G-ONPvMiXk=8YqDdwgCMjn-3C+n7D85N5ASBg@mail.gmail.com>
 <942629ca-1035-32b4-2025-1ddd970e1b1a@redhat.com>
 <CA+ZsWTVEiEJmgk+7Uydq+GM_TwO+CUkBqbKyF2g6BDW76omz6w@mail.gmail.com>
 <4ad3e321-4204-859f-cc2b-18f67e1faf9e@redhat.com>
Message-ID: <CA+ZsWTUvCPU53PP+WZV45c2Jk01n+08bkany4qYD=t4Qz4yEfA@mail.gmail.com>

Thanks Honza. I have raised these on both upstream projects.
I will leave upto implementer how best this can be done, considering the
technical limitations you mentioned.

https://github.com/corosync/corosync-qdevice/issues/13
https://github.com/ClusterLabs/booth/issues/99

Thanks,
Rohit

On Thu, Aug 13, 2020 at 1:03 PM Jan Friesse <jfriesse at redhat.com> wrote:

> Hi Rohit,
>
> > Hi Honza,
> > Thanks for your reply. Please find the attached image below:
> >
> > [image: image.png]
> >
> > Yes, I am talking about pacemaker alerts only.
> >
> > Please find my suggestions/requirements below:
> >
> > *Booth:*
> > 1. Node5 booth-arbitrator should be able to give event when any of the
> > booth node joins or leaves. booth-ip can be passed in event.
>
> This is not how booth works. Ticket leader (so site booth, never
> arbitrator) executes election and get replies from other
> sites/arbitrator. Follower executes election when leader hasn't for
> configured timeout.
>
> What I want to say is, that there is no "membership" - as in (for
> example) corosync fashion.
>
> The best we could get is the rough estimation based on election
> request/replies.
>
> > 2. Event when booth-arbitrator is up successfully and has started
> > monitoring the booth nodes.
>
> This is basically start of service. I think it's doable with small
> change in unit file (something like
>
> https://northernlightlabs.se/2014-07-05/systemd-status-mail-on-unit-failure.html
> )
>
> > 2. Geo site booth should be able to give event when its booth peers
> > joins/leaves. For example, Geo site1 gives an event when node5
> > booth-arbitrator joins/leaves OR site2 booth joins/leaves.  booth-ip can
> be
> > passed in event.
> > 3. On ticket movements (revoke/grant), every booth node(Site1/2 and
> node5)
> > should give events.
>
> That would be doable
>
> >
> > Note: pacemaker alerts works in a cluster. Since, arbitrator is a
> > non-cluster node, not sure how exactly it will work there. But this is
> good
> > to have feature.
> >
> > *Qnetd/Qdevice:*
> > This is similar to above.
> > 1. Node5 qnetd should be able to raise an event when any of the cluster
> > node joins/leaves the quorum.
>
> Doable
>
> > 2. Event when qnetd is up successfully and has started monitoring the
> > cluster nodes
>
> Qnetd itself is not monitoring qdevice nodes (it doesn't have list of
> nodes). It monitors node status after node joins (= it would be possible
> to trigger event on leave). So that may be enough.
>
> > 3. Cluster node should be able to give event when any of the quorum node
> > leaves/joins.
>
> You mean qdevice should be able to trigger event when connected to qnetd?
>
> >
> > If you see on high level, then these are kind of node/resource events wrt
> > booth and qnetd/qdevice.
>
> Yeah
>
> >
> > As of today wrt booth/qnetd, I don't see any provision where any of the
> > nodes gives any event when its peer leaves/joins. This makes it difficult
> > to know whether geo sites nodes can see booth-arbitrator or not. This is
>
> Got it. That's exactly what would be really problematic to implement,
> because of no "membership" in booth. It would be, however, possible to
> implement message when ticket was granted/rejected and have a list of
> other booths replies and what was their votes.
>
> > true the other way around also where booth-arbitrator cannot see geo
> booth
> > sites.
> > I am not sure how others are doing it in today's deployment, but I see
> need
> > of monitoring of every other booth/qnet node. So that on basis of event,
> > appropriate alarms can be raised and action can be taken accordingly.
> >
> > Please let me know if you agree on the usecases. I'll raise
> feature-request
>
> I can agree on usecases, but (especially with booth) there are technical
> problems on realizing them.
>
> > on the pacemaker upstream project accordingly.
>
> Please use booth (https://github.com/ClusterLabs/booth) and qdevice
> (https://github.com/corosync/corosync-qdevice) upstream rather than
> pacemaker, because these requests has really nothing to do with pcmk.
>
> Regards,
>    honza
>
> >
> > Thanks,
> > Rohit
> >
> > On Wed, Aug 12, 2020 at 8:58 PM Jan Friesse <jfriesse at redhat.com> wrote:
> >
> >> Hi Rohit,
> >>
> >> Rohit Saini napsal(a):
> >>> Hi Team,
> >>>
> >>> Question-1:
> >>> Similar to pcs alerts, do we have something similar for qdevice/qnetd?
> >> This
> >>
> >> You mean pacemaker alerts right?
> >>
> >>> is to detect asynchronously if any of the member is
> >> unreachable/joined/left
> >>> and if that member is qdevice or qnetd.
> >>
> >> Nope but actually shouldn't be that hard to implement. What exactly
> >> would you like to see there?
> >>
> >>>
> >>> Question-2:
> >>> Same above question for booth nodes and arbitrator. Is there any way to
> >>> receive events from booth daemon?
> >>
> >> Not directly (again, shouldn't be that hard to implement). But pacemaker
> >> alerts should be triggered when service changes state because of ticket
> >> grant/reject, isn't it?
> >>
> >>>
> >>> My main objective is to see if these daemons give events related to
> >>> their internal state transitions  and raise some alarms accordingly.
> For
> >>> example, boothd arbitrator is unreachable, ticket moved from x to y,
> etc.
> >>
> >> I don't think "boothd arbitrator is unreachable" alert is really doable.
> >> Ticket moved from x to y would be probably two alerts - 1. ticket
> >> rejected on X and 2. granted on Y.
> >>
> >> Would you mind to elaborate a bit more on events you would like to see
> >> and potentially open issue for upstream project (or, if you have a RH
> >> subscription try to contact GSS, so I get more time to work on this
> issue).
> >>
> >> Regards,
> >>     Honza
> >>
> >>>
> >>> Thanks,
> >>> Rohit
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Manage your subscription:
> >>> https://lists.clusterlabs.org/mailman/listinfo/users
> >>>
> >>> ClusterLabs home: https://www.clusterlabs.org/
> >>>
> >>
> >>
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200814/1c00d750/attachment.htm>

From bernd.lentes at helmholtz-muenchen.de  Fri Aug 14 06:17:51 2020
From: bernd.lentes at helmholtz-muenchen.de (Lentes, Bernd)
Date: Fri, 14 Aug 2020 12:17:51 +0200 (CEST)
Subject: [ClusterLabs] why is node fenced ?
In-Reply-To: <f411d804d82fd34b7ae0974fcd9f17821a115299.camel@redhat.com>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
 <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>
 <f411d804d82fd34b7ae0974fcd9f17821a115299.camel@redhat.com>
Message-ID: <1954017432.49729402.1597400271575.JavaMail.zimbra@helmholtz-muenchen.de>



----- On Aug 10, 2020, at 11:59 PM, kgaillot kgaillot at redhat.com wrote:
> The most recent transition is aborted, but since all its actions are
> complete, the only effect is to trigger a new transition.
> 
> We should probably rephrase the log message. In fact, the whole
> "transition" terminology is kind of obscure. It's hard to come up with
> something better though.
> 
Hi Ken,

i don't get it. How can s.th. be aborted which is already completed ?

Bernd
Helmholtz Zentrum M?nchen

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin Guenther
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671



From gbulfon at sonicle.com  Fri Aug 14 09:09:04 2020
From: gbulfon at sonicle.com (Gabriele Bulfon)
Date: Fri, 14 Aug 2020 15:09:04 +0200 (CEST)
Subject: [ClusterLabs] Antw: [EXT] Stonith failing
In-Reply-To: <68622439.459.1596025362188@www>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www>
Message-ID: <1118615335.1895.1597410544148@www>

Thanks to all your suggestions, I now have the systems with stonith configured on ipmi.
?
Two questions:
- how can I simulate a stonith situation to check that everything is ok?
- considering that I have both nodes with stonith against the other node, once the two nodes can communicate, how can I be sure the two nodes will not try to stonith each other?
?
:)
Thanks!
Gabriele
?
?
Sonicle S.r.l.?
:?
http://www.sonicle.com
Music:?
http://www.gabrielebulfon.com
Quantum Mechanics :?
http://www.cdbaby.com/cd/gabrielebulfon
Da:
Gabriele Bulfon
A:
Cluster Labs - All topics related to open-source clustering welcomed
Data:
29 luglio 2020 14.22.42 CEST
Oggetto:
Re: [ClusterLabs] Antw: [EXT] Stonith failing
?
It is a ZFS based illumos system.
I don't think SBD is an option.
Is there a reliable ZFS based stonith?
?
Gabriele
?
?
Sonicle S.r.l.?
:?
http://www.sonicle.com
Music:?
http://www.gabrielebulfon.com
Quantum Mechanics :?
http://www.cdbaby.com/cd/gabrielebulfon
Da:
Andrei Borzenkov
A:
Cluster Labs - All topics related to open-source clustering welcomed
Data:
29 luglio 2020 9.46.09 CEST
Oggetto:
Re: [ClusterLabs] Antw: [EXT] Stonith failing
?
On Wed, Jul 29, 2020 at 9:01 AM Gabriele Bulfon
gbulfon at sonicle.com
wrote:
That one was taken from a specific implementation on Solaris 11.
The situation is a dual node server with shared storage controller: both nodes see the same disks concurrently.
Here we must be sure that the two nodes are not going to import/mount the same zpool at the same time, or we will encounter data corruption:
?
ssh based "stonith" cannot guarantee it.
?
node 1 will be perferred for pool 1, node 2 for pool 2, only in case one of the node goes down or is taken offline the resources should be first free by the leaving node and taken by the other node.
?
Would you suggest one of the available stonith in this case?
?
?
IPMI, managed PDU, SBD ...
In practice, the only stonith method that works in case of complete node outage including any power supply is SBD.
_______________________________________________Manage your subscription:https://lists.clusterlabs.org/mailman/listinfo/usersClusterLabs home: https://www.clusterlabs.org/
_______________________________________________Manage your subscription:https://lists.clusterlabs.org/mailman/listinfo/usersClusterLabs home: https://www.clusterlabs.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200814/f3593e7a/attachment-0001.htm>

From bernd.lentes at helmholtz-muenchen.de  Fri Aug 14 14:37:43 2020
From: bernd.lentes at helmholtz-muenchen.de (Lentes, Bernd)
Date: Fri, 14 Aug 2020 20:37:43 +0200 (CEST)
Subject: [ClusterLabs] why is node fenced ?
In-Reply-To: <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
 <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>
Message-ID: <1230152204.50050895.1597430263067.JavaMail.zimbra@helmholtz-muenchen.de>

----- On Aug 9, 2020, at 10:17 PM, Bernd Lentes bernd.lentes at helmholtz-muenchen.de wrote:


>> So this appears to be the problem. From these logs I would guess the
>> successful stop on ha-idg-1 did not get written to the CIB for some
>> reason. I'd look at the pe input from this transition on ha-idg-2 to
>> confirm that.
>> 
>> Without the DC knowing about the stop, it tries to schedule a new one,
>> but the node is shutting down so it can't do it, which means it has to
>> be fenced.

I checked all relevant pe-files in this time period.
This is what i found out (i just write the important entries):

ha-idg-1:~/why-fenced/ha-idg-1/pengine # crm_simulate -S -x pe-input-3116 -G transition-3116.xml -D transition-3116.dot
Current cluster status:
 ...
 vm_nextcloud   (ocf::heartbeat:VirtualDomain): Started ha-idg-1
Transition Summary:
 ...
* Migrate    vm_nextcloud           ( ha-idg-1 -> ha-idg-2 )
Executing cluster transition:
 * Resource action: vm_nextcloud    migrate_from on ha-idg-2 <======= migrate vm_nextcloud
 * Resource action: vm_nextcloud    stop on ha-idg-1 
 * Pseudo action:   vm_nextcloud_start_0
Revised cluster status:
Node ha-idg-1 (1084777482): standby
Online: [ ha-idg-2 ]
vm_nextcloud   (ocf::heartbeat:VirtualDomain): Started ha-idg-2


ha-idg-1:~/why-fenced/ha-idg-1/pengine # crm_simulate -S -x pe-error-48 -G transition-4514.xml -D transition-4514.dot
Current cluster status:
Node ha-idg-1 (1084777482): standby
Online: [ ha-idg-2 ]
...
 vm_nextcloud   (ocf::heartbeat:VirtualDomain): FAILED[ ha-idg-2 ha-idg-1 ] <====== migration failed
Transition Summary:
..
 * Recover    vm_nextcloud            (             ha-idg-2 )
Executing cluster transition:
 * Resource action: vm_nextcloud    stop on ha-idg-2
 * Resource action: vm_nextcloud    stop on ha-idg-1 
 * Resource action: vm_nextcloud    start on ha-idg-2
 * Resource action: vm_nextcloud    monitor=30000 on ha-idg-2
Revised cluster status:
 vm_nextcloud   (ocf::heartbeat:VirtualDomain): Started ha-idg-2

ha-idg-1:~/why-fenced/ha-idg-1/pengine # crm_simulate -S -x pe-input-3117 -G transition-3117.xml -D transition-3117.dot
Current cluster status:
Node ha-idg-1 (1084777482): standby
Online: [ ha-idg-2 ]
 vm_nextcloud   (ocf::heartbeat:VirtualDomain): FAILED ha-idg-2 <====== start on ha-idg-2 failed
Transition Summary:
 * Stop       vm_nextcloud     ( ha-idg-2 )   due to node availability <==== stop vm_nextcloud (what means due to node availability ?)
Executing cluster transition:
 * Resource action: vm_nextcloud    stop on ha-idg-2
Revised cluster status:
 vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped

ha-idg-1:~/why-fenced/ha-idg-1/pengine # crm_simulate -S -x pe-input-3118 -G transition-4516.xml -D transition-4516.dot
Current cluster status:
Node ha-idg-1 (1084777482): standby
Online: [ ha-idg-2 ]
 vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped <============== vm_nextcloud is stopped
Transition Summary:
 * Shutdown ha-idg-1
Executing cluster transition:
 * Resource action: vm_nextcloud    stop on ha-idg-1 <==== why stop ? It is already stopped
Revised cluster status:
 vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped

ha-idg-1:~/why-fenced/ha-idg-2/pengine # crm_simulate -S -x pe-input-3545 -G transition-0.xml -D transition-0.dot
Current cluster status:
Node ha-idg-1 (1084777482): pending
Online: [ ha-idg-2 ]
 vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped <====== vm_nextcloud is stopped
Transition Summary:

Executing cluster transition:
Using the original execution date of: 2020-07-20 15:05:33Z
Revised cluster status:
vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped

ha-idg-1:~/why-fenced/ha-idg-2/pengine # crm_simulate -S -x pe-warn-749 -G transition-1.xml -D transition-1.dot
Current cluster status:
Node ha-idg-1 (1084777482): OFFLINE (standby)
Online: [ ha-idg-2 ]
 vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped <======= vm_nextcloud is stopped
Transition Summary:
 * Fence (Off) ha-idg-1 'resource actions are unrunnable'
Executing cluster transition:
 * Fencing ha-idg-1 (Off)
 * Pseudo action:   vm_nextcloud_stop_0 <======= why stop ? It is already stopped ?
Revised cluster status:
Node ha-idg-1 (1084777482): OFFLINE (standby)
Online: [ ha-idg-2 ]
 vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped

I don't understand why the cluster tries to stop a resource which is already stopped.

Bernd
Helmholtz Zentrum M?nchen

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin Guenther
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671



From gerry at infinityit.ie  Fri Aug 14 11:48:32 2020
From: gerry at infinityit.ie (Gerry Kernan)
Date: Fri, 14 Aug 2020 15:48:32 +0000
Subject: [ClusterLabs] DRBD resource not starting
Message-ID: <DB6P190MB0535C4F2F0FE9FCB8C9500F2BC400@DB6P190MB0535.EURP190.PROD.OUTLOOK.COM>

Hi
Im trying to add a drbd resource to pacemaker cluster on centos 7


But getting this error on pcs status
drbd_r0        (ocf::linbit:drbd):      ORPHANED FAILED (blocked)[ vps1 vps2 ]


if I try and the resource from cli I get out out below

[root at VPS1 drbd-utils]# pcs resource debug-start rep-r0
Operation start for rep-r0:0 (ocf:linbit:drbd) returned: 'unknown error' (1)
>  stdout: drbdsetup - Configure the DRBD kernel module.
>  stdout:
>  stdout: USAGE: drbdsetup command {arguments} [options]
>  stdout:
>  stdout: Commands:
>  stdout:     primary - Change the role of a node in a resource to primary.
>  stdout:     secondary - Change the role of a node in a resource to secondary.
>  stdout:     attach - Attach a lower-level device to an existing replicated device.
>  stdout:     disk-options - Change the disk options of an attached lower-level device.
>  stdout:     detach - Detach the lower-level device of a replicated device.
>  stdout:     connect - Attempt to (re)establish a replication link to a peer host.
>  stdout:     new-peer - Make a peer host known to a resource.
>  stdout:     del-peer - Remove a connection to a peer host.
>  stdout:     new-path - Add a path (endpoint address pair) where a peer host should be
>  stdout:         reachable.
>  stdout:     del-path - Remove a path (endpoint address pair) from a connection to a peer
>  stdout:         host.
>  stdout:     net-options - Change the network options of a connection.
>  stdout:     disconnect - Unconnect from a peer host.
>  stdout:     resize - Reexamine the lower-level device sizes to resize a replicated
>  stdout:         device.
>  stdout:     resource-options - Change the resource options of an existing resource.
>  stdout:     peer-device-options - Change peer-device options.
>  stdout:     new-current-uuid - Generate a new current UUID.
>  stdout:     invalidate - Replace the local data of a volume with that of a peer.
>  stdout:     invalidate-remote - Replace a peer's data of a volume with the local data.
>  stdout:     pause-sync - Stop resynchronizing between a local and a peer device.
>  stdout:     resume-sync - Allow resynchronization to resume on a replicated device.
>  stdout:     suspend-io - Suspend I/O on a replicated device.
>  stdout:     resume-io - Resume I/O on a replicated device.
>  stdout:     outdate - Mark the data on a lower-level device as outdated.
>  stdout:     verify - Verify the data on a lower-level device against a peer device.
>  stdout:     down - Take a resource down.
>  stdout:     role - Show the current role of a resource.
>  stdout:     cstate - Show the current state of a connection.
>  stdout:     dstate - Show the current disk state of a lower-level device.
>  stdout:     show-gi - Show the data generation identifiers for a device on a particular
>  stdout:         connection, with explanations.
>  stdout:     get-gi - Show the data generation identifiers for a device on a particular
>  stdout:         connection.
>  stdout:     show - Show the current configuration of a resource, or of all resources.
>  stdout:     status - Show the state of a resource, or of all resources.
>  stdout:     check-resize - Remember the current size of a lower-level device.
>  stdout:     events2 - Show the current state and all state changes of a resource, or of
>  stdout:         all resources.
>  stdout:     wait-sync-volume - Wait until resync finished on a volume.
>  stdout:     wait-sync-connection - Wait until resync finished on all volumes of a
>  stdout:         connection.
>  stdout:     wait-sync-resource - Wait until resync finished on all volumes.
>  stdout:     wait-connect-volume - Wait until a device on a peer is visible.
>  stdout:     wait-connect-connection - Wait until all peer volumes of connection are
>  stdout:         visible.
>  stdout:     wait-connect-resource - Wait until all connections are establised.
>  stdout:     new-resource - Create a new resource.
>  stdout:     new-minor - Create a new replicated device within a resource.
>  stdout:     del-minor - Remove a replicated device.
>  stdout:     del-resource - Remove a resource.
>  stdout:     forget-peer - Completely remove any reference to a unconnected peer from
>  stdout:         meta-data.
>  stdout:
>  stdout: Use 'drbdsetup help command' for command-specific help.
>  stdout:
>  stdout:
>  stdout: invalid command
>  stdout:
>  stdout:
>  stdout:
>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 318: USAGE:: command not found
>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 320: Commands:: command not found
>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 321: primary: command not found
>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 322: secondary: command not found
>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 323: attach: command not found
>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 324: disk-options: command not found
>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 325: detach: command not found
>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: eval: line 326: syntax error near unexpected token `('
>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: eval: line 326: `    connect - Attempt to (re)establish a replication link to a peer host.'
>  stderr: new-minor r0 0 0: sysfs node '/sys/devices/virtual/block/drbd0' (already? still?) exists
>  stderr: r0: Failure: (161) Minor or volume exists already (delete it first)
>  stderr: Command 'drbdsetup new-minor r0 0 0' terminated with exit code 10
>  stderr: Aug 14 16:47:38 ERROR: r0: Called drbdadm -c /etc/drbd.conf -S new-minor r0
>  stderr: Aug 14 16:47:38 ERROR: r0: Exit code 10
>  stderr: Aug 14 16:47:38 ERROR: r0: Command output:

Gerry


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200814/47323f6c/attachment-0001.htm>

From hmoneta at gmail.com  Thu Aug 13 19:12:43 2020
From: hmoneta at gmail.com (Howard)
Date: Thu, 13 Aug 2020 16:12:43 -0700
Subject: [ClusterLabs] Filtering info messages from pacemaker.log
Message-ID: <CAO51vj7pe5EhY=RYXAU89zoSwT5+YAUjbO4APnYmES=fFms3MA@mail.gmail.com>

Hi there.  Really getting a lot of good from using Pacemaker.  Quick
question on filtering entries out of the pacemaker.log.

We have other streaming slots that are active for our Pacemaker managed
PostgreSQL cluster in addition to the slot handling the replication. The
/var/log/pacemaker/pacemaker.log is full of these below entries writing
four new rows every second. Is there some way to exclude this noise from
the log?

"pgsqlms(pgsqld)[xxxxxxx]:       xxxxxxxxxx  INFO: Ignoring unknown
application_name/node "PostgreSQL JDBC Driver"",

Thanks for any insight.

Howard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200813/0b4b5400/attachment.htm>

From jgdr at dalibo.com  Sat Aug 15 10:06:55 2020
From: jgdr at dalibo.com (Jehan-Guillaume de Rorthais)
Date: Sat, 15 Aug 2020 16:06:55 +0200
Subject: [ClusterLabs] Filtering info messages from pacemaker.log
In-Reply-To: <CAO51vj7pe5EhY=RYXAU89zoSwT5+YAUjbO4APnYmES=fFms3MA@mail.gmail.com>
References: <CAO51vj7pe5EhY=RYXAU89zoSwT5+YAUjbO4APnYmES=fFms3MA@mail.gmail.com>
Message-ID: <20200815160655.0bca507f@firost>

Hi,

On Thu, 13 Aug 2020 16:12:43 -0700
Howard <hmoneta at gmail.com> wrote:

> Hi there.  Really getting a lot of good from using Pacemaker.  Quick
> question on filtering entries out of the pacemaker.log.
> 
> We have other streaming slots that are active for our Pacemaker managed
> PostgreSQL cluster in addition to the slot handling the replication. The
> /var/log/pacemaker/pacemaker.log is full of these below entries writing
> four new rows every second. Is there some way to exclude this noise from
> the log?
> 
> "pgsqlms(pgsqld)[xxxxxxx]:       xxxxxxxxxx  INFO: Ignoring unknown
> application_name/node "PostgreSQL JDBC Driver"",

I just created an issue in PAF repository to keep track of this issue. See:
https://github.com/ClusterLabs/PAF/issues/178

I'll try to work on this soon. Do not hesitate to post some fix suggestion if
you have some.

In the meantime, maybe you can filter out this flood with some rsyslog setup?

Thank you for your report!

From hunter86_bg at yahoo.com  Sat Aug 15 11:16:09 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Sat, 15 Aug 2020 18:16:09 +0300
Subject: [ClusterLabs] DRBD resource not starting
In-Reply-To: <DB6P190MB0535C4F2F0FE9FCB8C9500F2BC400@DB6P190MB0535.EURP190.PROD.OUTLOOK.COM>
References: <DB6P190MB0535C4F2F0FE9FCB8C9500F2BC400@DB6P190MB0535.EURP190.PROD.OUTLOOK.COM>
Message-ID: <E2F890BF-D894-401A-9D5D-41D783AE0F70@yahoo.com>

And how did you define the drbd resource?

Best Regards,
Strahil Nikolov

?? 14 ?????? 2020 ?. 18:48:32 GMT+03:00, Gerry Kernan <gerry at infinityit.ie> ??????:
>Hi
>Im trying to add a drbd resource to pacemaker cluster on centos 7
>
>
>But getting this error on pcs status
>drbd_r0        (ocf::linbit:drbd):      ORPHANED FAILED (blocked)[ vps1
>vps2 ]
>
>
>if I try and the resource from cli I get out out below
>
>[root at VPS1 drbd-utils]# pcs resource debug-start rep-r0
>Operation start for rep-r0:0 (ocf:linbit:drbd) returned: 'unknown
>error' (1)
>>  stdout: drbdsetup - Configure the DRBD kernel module.
>>  stdout:
>>  stdout: USAGE: drbdsetup command {arguments} [options]
>>  stdout:
>>  stdout: Commands:
>>  stdout:     primary - Change the role of a node in a resource to
>primary.
>>  stdout:     secondary - Change the role of a node in a resource to
>secondary.
>>  stdout:     attach - Attach a lower-level device to an existing
>replicated device.
>>  stdout:     disk-options - Change the disk options of an attached
>lower-level device.
>>  stdout:     detach - Detach the lower-level device of a replicated
>device.
>>  stdout:     connect - Attempt to (re)establish a replication link to
>a peer host.
>>  stdout:     new-peer - Make a peer host known to a resource.
>>  stdout:     del-peer - Remove a connection to a peer host.
>>  stdout:     new-path - Add a path (endpoint address pair) where a
>peer host should be
>>  stdout:         reachable.
>>  stdout:     del-path - Remove a path (endpoint address pair) from a
>connection to a peer
>>  stdout:         host.
>>  stdout:     net-options - Change the network options of a
>connection.
>>  stdout:     disconnect - Unconnect from a peer host.
>>  stdout:     resize - Reexamine the lower-level device sizes to
>resize a replicated
>>  stdout:         device.
>>  stdout:     resource-options - Change the resource options of an
>existing resource.
>>  stdout:     peer-device-options - Change peer-device options.
>>  stdout:     new-current-uuid - Generate a new current UUID.
>>  stdout:     invalidate - Replace the local data of a volume with
>that of a peer.
>>  stdout:     invalidate-remote - Replace a peer's data of a volume
>with the local data.
>>  stdout:     pause-sync - Stop resynchronizing between a local and a
>peer device.
>>  stdout:     resume-sync - Allow resynchronization to resume on a
>replicated device.
>>  stdout:     suspend-io - Suspend I/O on a replicated device.
>>  stdout:     resume-io - Resume I/O on a replicated device.
>>  stdout:     outdate - Mark the data on a lower-level device as
>outdated.
>>  stdout:     verify - Verify the data on a lower-level device against
>a peer device.
>>  stdout:     down - Take a resource down.
>>  stdout:     role - Show the current role of a resource.
>>  stdout:     cstate - Show the current state of a connection.
>>  stdout:     dstate - Show the current disk state of a lower-level
>device.
>>  stdout:     show-gi - Show the data generation identifiers for a
>device on a particular
>>  stdout:         connection, with explanations.
>>  stdout:     get-gi - Show the data generation identifiers for a
>device on a particular
>>  stdout:         connection.
>>  stdout:     show - Show the current configuration of a resource, or
>of all resources.
>>  stdout:     status - Show the state of a resource, or of all
>resources.
>>  stdout:     check-resize - Remember the current size of a
>lower-level device.
>>  stdout:     events2 - Show the current state and all state changes
>of a resource, or of
>>  stdout:         all resources.
>>  stdout:     wait-sync-volume - Wait until resync finished on a
>volume.
>>  stdout:     wait-sync-connection - Wait until resync finished on all
>volumes of a
>>  stdout:         connection.
>>  stdout:     wait-sync-resource - Wait until resync finished on all
>volumes.
>>  stdout:     wait-connect-volume - Wait until a device on a peer is
>visible.
>>  stdout:     wait-connect-connection - Wait until all peer volumes of
>connection are
>>  stdout:         visible.
>>  stdout:     wait-connect-resource - Wait until all connections are
>establised.
>>  stdout:     new-resource - Create a new resource.
>>  stdout:     new-minor - Create a new replicated device within a
>resource.
>>  stdout:     del-minor - Remove a replicated device.
>>  stdout:     del-resource - Remove a resource.
>>  stdout:     forget-peer - Completely remove any reference to a
>unconnected peer from
>>  stdout:         meta-data.
>>  stdout:
>>  stdout: Use 'drbdsetup help command' for command-specific help.
>>  stdout:
>>  stdout:
>>  stdout: invalid command
>>  stdout:
>>  stdout:
>>  stdout:
>>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 318: USAGE::
>command not found
>>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 320: Commands::
>command not found
>>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 321: primary:
>command not found
>>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 322: secondary:
>command not found
>>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 323: attach:
>command not found
>>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 324: disk-options:
>command not found
>>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 325: detach:
>command not found
>>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: eval: line 326: syntax
>error near unexpected token `('
>>  stderr: /usr/lib/ocf/resource.d/linbit/drbd: eval: line 326: `   
>connect - Attempt to (re)establish a replication link to a peer host.'
>>  stderr: new-minor r0 0 0: sysfs node
>'/sys/devices/virtual/block/drbd0' (already? still?) exists
>>  stderr: r0: Failure: (161) Minor or volume exists already (delete it
>first)
>>  stderr: Command 'drbdsetup new-minor r0 0 0' terminated with exit
>code 10
>>  stderr: Aug 14 16:47:38 ERROR: r0: Called drbdadm -c /etc/drbd.conf
>-S new-minor r0
>>  stderr: Aug 14 16:47:38 ERROR: r0: Exit code 10
>>  stderr: Aug 14 16:47:38 ERROR: r0: Command output:
>
>Gerry

From nwahl at redhat.com  Sat Aug 15 21:25:07 2020
From: nwahl at redhat.com (Reid Wahl)
Date: Sat, 15 Aug 2020 18:25:07 -0700
Subject: [ClusterLabs] Antw: [EXT] Stonith failing
In-Reply-To: <1118615335.1895.1597410544148@www>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
Message-ID: <CAPiuu98r7S7b86L5G8kM+Q0cCXPTXSEw16Sr6xci6yysrz5Mxg@mail.gmail.com>

On Fri, Aug 14, 2020 at 6:10 AM Gabriele Bulfon <gbulfon at sonicle.com> wrote:

> Thanks to all your suggestions, I now have the systems with stonith
> configured on ipmi.
>
> Two questions:
> - how can I simulate a stonith situation to check that everything is ok?
>

You can run `stonith_admin -B <node>` to tell Pacemaker to reboot the node
using the configured stonith devices. If you want to test a network
failure, you can have iptables block inbound and outbound traffic on the
heartbeat IP address on one node.


> - considering that I have both nodes with stonith against the other node,
> once the two nodes can communicate, how can I be sure the two nodes will
> not try to stonith each other?
>

The simplest option is to add a delay attribute (e.g., delay=10) to one of
the stonith devices. That way, if both nodes want to fence each other, the
node whose stonith device has a delay configured will wait for the delay to
expire before executing the reboot action.

Alternatively, you can set up corosync-qdevice, using a separate system
running qnetd server as a quorum arbitrator.


> :)
> Thanks!
> Gabriele
>
>
>
> *Sonicle S.r.l. *: http://www.sonicle.com
> *Music: *http://www.gabrielebulfon.com
> *Quantum Mechanics : *http://www.cdbaby.com/cd/gabrielebulfon
>
> ------------------------------
>
>
> *Da:* Gabriele Bulfon <gbulfon at sonicle.com>
> *A:* Cluster Labs - All topics related to open-source clustering welcomed
> <users at clusterlabs.org>
> *Data:* 29 luglio 2020 14.22.42 CEST
> *Oggetto:* Re: [ClusterLabs] Antw: [EXT] Stonith failing
>
>
>
> It is a ZFS based illumos system.
> I don't think SBD is an option.
> Is there a reliable ZFS based stonith?
>
> Gabriele
>
>
>
> *Sonicle S.r.l. *: http://www.sonicle.com
> *Music: *http://www.gabrielebulfon.com
> *Quantum Mechanics : *http://www.cdbaby.com/cd/gabrielebulfon
>
> ------------------------------
>
>
> *Da:* Andrei Borzenkov <arvidjaar at gmail.com>
> *A:* Cluster Labs - All topics related to open-source clustering welcomed
> <users at clusterlabs.org>
> *Data:* 29 luglio 2020 9.46.09 CEST
> *Oggetto:* Re: [ClusterLabs] Antw: [EXT] Stonith failing
>
>
>
>
> On Wed, Jul 29, 2020 at 9:01 AM Gabriele Bulfon <gbulfon at sonicle.com>
> wrote:
>
>> That one was taken from a specific implementation on Solaris 11.
>> The situation is a dual node server with shared storage controller: both
>> nodes see the same disks concurrently.
>> Here we must be sure that the two nodes are not going to import/mount the
>> same zpool at the same time, or we will encounter data corruption:
>>
>
> ssh based "stonith" cannot guarantee it.
>
>
>> node 1 will be perferred for pool 1, node 2 for pool 2, only in case one
>> of the node goes down or is taken offline the resources should be first
>> free by the leaving node and taken by the other node.
>>
>> Would you suggest one of the available stonith in this case?
>>
>>
>
> IPMI, managed PDU, SBD ...
> In practice, the only stonith method that works in case of complete node
> outage including any power supply is SBD.
>
> _______________________________________________
> Manage your subscription:https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>
> _______________________________________________
> Manage your subscription:https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>


-- 
Regards,

Reid Wahl, RHCA
Software Maintenance Engineer, Red Hat
CEE - Platform Support Delivery - ClusterHA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200815/d98d1f64/attachment.htm>

From arvidjaar at gmail.com  Sun Aug 16 05:40:05 2020
From: arvidjaar at gmail.com (Andrei Borzenkov)
Date: Sun, 16 Aug 2020 12:40:05 +0300
Subject: [ClusterLabs] Antw: [EXT] Stonith failing
In-Reply-To: <CAPiuu98r7S7b86L5G8kM+Q0cCXPTXSEw16Sr6xci6yysrz5Mxg@mail.gmail.com>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
 <CAPiuu98r7S7b86L5G8kM+Q0cCXPTXSEw16Sr6xci6yysrz5Mxg@mail.gmail.com>
Message-ID: <032dd57d-957f-3e64-c42c-d03ded631a27@gmail.com>

16.08.2020 04:25, Reid Wahl ?????:
> 
> 
>> - considering that I have both nodes with stonith against the other node,
>> once the two nodes can communicate, how can I be sure the two nodes will
>> not try to stonith each other?
>>
> 
> The simplest option is to add a delay attribute (e.g., delay=10) to one of
> the stonith devices. That way, if both nodes want to fence each other, the
> node whose stonith device has a delay configured will wait for the delay to
> expire before executing the reboot action.
> 

Current pacemaker (2.0.4) also supports priority-fencing-delay option
that computes delay based on which resources are active on specific
node, so favoring node with "more important" resources.

> Alternatively, you can set up corosync-qdevice, using a separate system
> running qnetd server as a quorum arbitrator.
> 

Any solution that is based on node suicide is prone to complete cluster
loss. In particular, in two node cluster with qdevice surviving node
will commit suicide is qnetd is not accessible.

As long as external stonith is reasonably reliable it is much preferred
to any solution based on quorum (unless you have very specific
requirements and can tolerate running remaining nodes in "frozen" mode
to limit unavailability).

And before someone jumps in - SBD falls into "solution based on suicide"
as well.

From gerry at infinityit.ie  Sat Aug 15 12:02:05 2020
From: gerry at infinityit.ie (Gerry Kernan)
Date: Sat, 15 Aug 2020 16:02:05 +0000
Subject: [ClusterLabs] DRBD resource not starting
In-Reply-To: <E2F890BF-D894-401A-9D5D-41D783AE0F70@yahoo.com>
References: <DB6P190MB0535C4F2F0FE9FCB8C9500F2BC400@DB6P190MB0535.EURP190.PROD.OUTLOOK.COM>,
 <E2F890BF-D894-401A-9D5D-41D783AE0F70@yahoo.com>
Message-ID: <6550F8AA-40D1-41A3-B568-131E293C9C80@infinityit.ie>

Hi 

Got it sorted 
Removed drbd-pacemaker and reinstalled drbd90-utilise
All ok now 

Gerry 

Sent from my iPhone

> On 15 Aug 2020, at 16:18, Strahil Nikolov <hunter86_bg at yahoo.com> wrote:
> 
> ?And how did you define the drbd resource?
> 
> Best Regards,
> Strahil Nikolov
> 
> ?? 14 ?????? 2020 ?. 18:48:32 GMT+03:00, Gerry Kernan <gerry at infinityit.ie> ??????:
>> Hi
>> Im trying to add a drbd resource to pacemaker cluster on centos 7
>> 
>> 
>> But getting this error on pcs status
>> drbd_r0        (ocf::linbit:drbd):      ORPHANED FAILED (blocked)[ vps1
>> vps2 ]
>> 
>> 
>> if I try and the resource from cli I get out out below
>> 
>> [root at VPS1 drbd-utils]# pcs resource debug-start rep-r0
>> Operation start for rep-r0:0 (ocf:linbit:drbd) returned: 'unknown
>> error' (1)
>>> stdout: drbdsetup - Configure the DRBD kernel module.
>>> stdout:
>>> stdout: USAGE: drbdsetup command {arguments} [options]
>>> stdout:
>>> stdout: Commands:
>>> stdout:     primary - Change the role of a node in a resource to
>> primary.
>>> stdout:     secondary - Change the role of a node in a resource to
>> secondary.
>>> stdout:     attach - Attach a lower-level device to an existing
>> replicated device.
>>> stdout:     disk-options - Change the disk options of an attached
>> lower-level device.
>>> stdout:     detach - Detach the lower-level device of a replicated
>> device.
>>> stdout:     connect - Attempt to (re)establish a replication link to
>> a peer host.
>>> stdout:     new-peer - Make a peer host known to a resource.
>>> stdout:     del-peer - Remove a connection to a peer host.
>>> stdout:     new-path - Add a path (endpoint address pair) where a
>> peer host should be
>>> stdout:         reachable.
>>> stdout:     del-path - Remove a path (endpoint address pair) from a
>> connection to a peer
>>> stdout:         host.
>>> stdout:     net-options - Change the network options of a
>> connection.
>>> stdout:     disconnect - Unconnect from a peer host.
>>> stdout:     resize - Reexamine the lower-level device sizes to
>> resize a replicated
>>> stdout:         device.
>>> stdout:     resource-options - Change the resource options of an
>> existing resource.
>>> stdout:     peer-device-options - Change peer-device options.
>>> stdout:     new-current-uuid - Generate a new current UUID.
>>> stdout:     invalidate - Replace the local data of a volume with
>> that of a peer.
>>> stdout:     invalidate-remote - Replace a peer's data of a volume
>> with the local data.
>>> stdout:     pause-sync - Stop resynchronizing between a local and a
>> peer device.
>>> stdout:     resume-sync - Allow resynchronization to resume on a
>> replicated device.
>>> stdout:     suspend-io - Suspend I/O on a replicated device.
>>> stdout:     resume-io - Resume I/O on a replicated device.
>>> stdout:     outdate - Mark the data on a lower-level device as
>> outdated.
>>> stdout:     verify - Verify the data on a lower-level device against
>> a peer device.
>>> stdout:     down - Take a resource down.
>>> stdout:     role - Show the current role of a resource.
>>> stdout:     cstate - Show the current state of a connection.
>>> stdout:     dstate - Show the current disk state of a lower-level
>> device.
>>> stdout:     show-gi - Show the data generation identifiers for a
>> device on a particular
>>> stdout:         connection, with explanations.
>>> stdout:     get-gi - Show the data generation identifiers for a
>> device on a particular
>>> stdout:         connection.
>>> stdout:     show - Show the current configuration of a resource, or
>> of all resources.
>>> stdout:     status - Show the state of a resource, or of all
>> resources.
>>> stdout:     check-resize - Remember the current size of a
>> lower-level device.
>>> stdout:     events2 - Show the current state and all state changes
>> of a resource, or of
>>> stdout:         all resources.
>>> stdout:     wait-sync-volume - Wait until resync finished on a
>> volume.
>>> stdout:     wait-sync-connection - Wait until resync finished on all
>> volumes of a
>>> stdout:         connection.
>>> stdout:     wait-sync-resource - Wait until resync finished on all
>> volumes.
>>> stdout:     wait-connect-volume - Wait until a device on a peer is
>> visible.
>>> stdout:     wait-connect-connection - Wait until all peer volumes of
>> connection are
>>> stdout:         visible.
>>> stdout:     wait-connect-resource - Wait until all connections are
>> establised.
>>> stdout:     new-resource - Create a new resource.
>>> stdout:     new-minor - Create a new replicated device within a
>> resource.
>>> stdout:     del-minor - Remove a replicated device.
>>> stdout:     del-resource - Remove a resource.
>>> stdout:     forget-peer - Completely remove any reference to a
>> unconnected peer from
>>> stdout:         meta-data.
>>> stdout:
>>> stdout: Use 'drbdsetup help command' for command-specific help.
>>> stdout:
>>> stdout:
>>> stdout: invalid command
>>> stdout:
>>> stdout:
>>> stdout:
>>> stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 318: USAGE::
>> command not found
>>> stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 320: Commands::
>> command not found
>>> stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 321: primary:
>> command not found
>>> stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 322: secondary:
>> command not found
>>> stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 323: attach:
>> command not found
>>> stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 324: disk-options:
>> command not found
>>> stderr: /usr/lib/ocf/resource.d/linbit/drbd: line 325: detach:
>> command not found
>>> stderr: /usr/lib/ocf/resource.d/linbit/drbd: eval: line 326: syntax
>> error near unexpected token `('
>>> stderr: /usr/lib/ocf/resource.d/linbit/drbd: eval: line 326: `   
>> connect - Attempt to (re)establish a replication link to a peer host.'
>>> stderr: new-minor r0 0 0: sysfs node
>> '/sys/devices/virtual/block/drbd0' (already? still?) exists
>>> stderr: r0: Failure: (161) Minor or volume exists already (delete it
>> first)
>>> stderr: Command 'drbdsetup new-minor r0 0 0' terminated with exit
>> code 10
>>> stderr: Aug 14 16:47:38 ERROR: r0: Called drbdadm -c /etc/drbd.conf
>> -S new-minor r0
>>> stderr: Aug 14 16:47:38 ERROR: r0: Exit code 10
>>> stderr: Aug 14 16:47:38 ERROR: r0: Command output:
>> 
>> Gerry

From kwenning at redhat.com  Mon Aug 17 03:06:12 2020
From: kwenning at redhat.com (Klaus Wenninger)
Date: Mon, 17 Aug 2020 09:06:12 +0200
Subject: [ClusterLabs] Antw: [EXT] Stonith failing
In-Reply-To: <032dd57d-957f-3e64-c42c-d03ded631a27@gmail.com>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
 <CAPiuu98r7S7b86L5G8kM+Q0cCXPTXSEw16Sr6xci6yysrz5Mxg@mail.gmail.com>
 <032dd57d-957f-3e64-c42c-d03ded631a27@gmail.com>
Message-ID: <96e70278-7fab-d86d-4df8-2f1c8b4291a0@redhat.com>

On 8/16/20 11:40 AM, Andrei Borzenkov wrote:
> 16.08.2020 04:25, Reid Wahl ?????:
>>
>>> - considering that I have both nodes with stonith against the other node,
>>> once the two nodes can communicate, how can I be sure the two nodes will
>>> not try to stonith each other?
>>>
>> The simplest option is to add a delay attribute (e.g., delay=10) to one of
>> the stonith devices. That way, if both nodes want to fence each other, the
>> node whose stonith device has a delay configured will wait for the delay to
>> expire before executing the reboot action.
If your fence-agent supports a delay attribute you can of course use
that. As this isn't available with every fence-agent or is looking
differently depending on the fence-agent we've introduced
pcmk_delay_max & pcmk_delay_base. These are applied prior
to actually calling the fence-agent and thus are always available and
always look the same. The delay is gonna be some random time
between pcmk_delay_base and pcmk_delay_max.
This takes us to another approach how you can reduce chances
of a fatal fence-race. Assuming that the reason why the fence-race
is triggered is detected around the same time when just adding a
random time you will very likely prevent them killing each other.
This is especially interesting when there is no clear / easy way
to determine which of the nodes is more important at this time.
>>
> Current pacemaker (2.0.4) also supports priority-fencing-delay option
> that computes delay based on which resources are active on specific
> node, so favoring node with "more important" resources.
>
>> Alternatively, you can set up corosync-qdevice, using a separate system
>> running qnetd server as a quorum arbitrator.
>>
> Any solution that is based on node suicide is prone to complete cluster
> loss. In particular, in two node cluster with qdevice surviving node
> will commit suicide is qnetd is not accessible.
I don't think that what Reid suggested was going for nodes
that loose quorum to commit suicide right away.
You can use quorum simply as a means of preventing fence-races
otherwise inherent to 2-node-clusters.
>
> As long as external stonith is reasonably reliable it is much preferred
> to any solution based on quorum (unless you have very specific
> requirements and can tolerate running remaining nodes in "frozen" mode
> to limit unavailability).
Well we can name the predominant scenario why one might not want to depend
on fencing-devices like ipmi: If you want to cover a scenario where the
nodes don't
just loose corosync connectivity but as well access from one node to the
fencing
device of the other is interrupted you probably won't get around an
approach that
involves some kind of arbitrator.
>
> And before someone jumps in - SBD falls into "solution based on suicide"
> as well.
Got your point without that hint ;-)

Klaus
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/


From arvidjaar at gmail.com  Mon Aug 17 03:19:33 2020
From: arvidjaar at gmail.com (Andrei Borzenkov)
Date: Mon, 17 Aug 2020 10:19:33 +0300
Subject: [ClusterLabs] Antw: [EXT] Stonith failing
In-Reply-To: <96e70278-7fab-d86d-4df8-2f1c8b4291a0@redhat.com>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
 <CAPiuu98r7S7b86L5G8kM+Q0cCXPTXSEw16Sr6xci6yysrz5Mxg@mail.gmail.com>
 <032dd57d-957f-3e64-c42c-d03ded631a27@gmail.com>
 <96e70278-7fab-d86d-4df8-2f1c8b4291a0@redhat.com>
Message-ID: <b0ca7344-488f-18c5-f84e-4b5fb8cdbb67@gmail.com>

17.08.2020 10:06, Klaus Wenninger ?????:
>>
>>> Alternatively, you can set up corosync-qdevice, using a separate system
>>> running qnetd server as a quorum arbitrator.
>>>
>> Any solution that is based on node suicide is prone to complete cluster
>> loss. In particular, in two node cluster with qdevice surviving node
>> will commit suicide is qnetd is not accessible.
> I don't think that what Reid suggested was going for nodes
> that loose quorum to commit suicide right away.
> You can use quorum simply as a means of preventing fence-races
> otherwise inherent to 2-node-clusters.

Can you please show the configuration example how to do it? Sorry, but I
do not understand how is it possible.

>>
>> As long as external stonith is reasonably reliable it is much preferred
>> to any solution based on quorum (unless you have very specific
>> requirements and can tolerate running remaining nodes in "frozen" mode
>> to limit unavailability).
> Well we can name the predominant scenario why one might not want to depend
> on fencing-devices like ipmi: If you want to cover a scenario where the
> nodes don't
> just loose corosync connectivity but as well access from one node to the
> fencing
> device of the other is interrupted you probably won't get around an
> approach that
> involves some kind of arbitrator.

Sure. Which is why I said "reasonably reliable". Still even in this case
one must understand all pros and cons to decide which risk is more
important to mitigate.


From kwenning at redhat.com  Mon Aug 17 03:25:30 2020
From: kwenning at redhat.com (Klaus Wenninger)
Date: Mon, 17 Aug 2020 09:25:30 +0200
Subject: [ClusterLabs] Antw: [EXT] Stonith failing
In-Reply-To: <b0ca7344-488f-18c5-f84e-4b5fb8cdbb67@gmail.com>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
 <CAPiuu98r7S7b86L5G8kM+Q0cCXPTXSEw16Sr6xci6yysrz5Mxg@mail.gmail.com>
 <032dd57d-957f-3e64-c42c-d03ded631a27@gmail.com>
 <96e70278-7fab-d86d-4df8-2f1c8b4291a0@redhat.com>
 <b0ca7344-488f-18c5-f84e-4b5fb8cdbb67@gmail.com>
Message-ID: <b60aa6ab-c596-36d8-019f-a7ac6ec32995@redhat.com>

On 8/17/20 9:19 AM, Andrei Borzenkov wrote:
> 17.08.2020 10:06, Klaus Wenninger ?????:
>>>> Alternatively, you can set up corosync-qdevice, using a separate system
>>>> running qnetd server as a quorum arbitrator.
>>>>
>>> Any solution that is based on node suicide is prone to complete cluster
>>> loss. In particular, in two node cluster with qdevice surviving node
>>> will commit suicide is qnetd is not accessible.
>> I don't think that what Reid suggested was going for nodes
>> that loose quorum to commit suicide right away.
>> You can use quorum simply as a means of preventing fence-races
>> otherwise inherent to 2-node-clusters.
> Can you please show the configuration example how to do it? Sorry, but I
> do not understand how is it possible.
Simply don't set the 2-node-flag. So just one of the nodes will have
quorum and just one of them will attempt fencing.
>
>>> As long as external stonith is reasonably reliable it is much preferred
>>> to any solution based on quorum (unless you have very specific
>>> requirements and can tolerate running remaining nodes in "frozen" mode
>>> to limit unavailability).
>> Well we can name the predominant scenario why one might not want to depend
>> on fencing-devices like ipmi: If you want to cover a scenario where the
>> nodes don't
>> just loose corosync connectivity but as well access from one node to the
>> fencing
>> device of the other is interrupted you probably won't get around an
>> approach that
>> involves some kind of arbitrator.
> Sure. Which is why I said "reasonably reliable". Still even in this case
> one must understand all pros and cons to decide which risk is more
> important to mitigate.
>
Exactly! Which is why I tried to give some flesh to the idea to foster
this kind of understanding. You always have to be aware of the failure
scenarios you want to cover and what it might cost you elsewhere
to cover one specific scenario.

Klaus


From jfriesse at redhat.com  Mon Aug 17 03:38:49 2020
From: jfriesse at redhat.com (Jan Friesse)
Date: Mon, 17 Aug 2020 09:38:49 +0200
Subject: [ClusterLabs] Alerts for qdevice/qnetd/booth
In-Reply-To: <CA+ZsWTUvCPU53PP+WZV45c2Jk01n+08bkany4qYD=t4Qz4yEfA@mail.gmail.com>
References: <CA+ZsWTXN8srS-G-ONPvMiXk=8YqDdwgCMjn-3C+n7D85N5ASBg@mail.gmail.com>
 <942629ca-1035-32b4-2025-1ddd970e1b1a@redhat.com>
 <CA+ZsWTVEiEJmgk+7Uydq+GM_TwO+CUkBqbKyF2g6BDW76omz6w@mail.gmail.com>
 <4ad3e321-4204-859f-cc2b-18f67e1faf9e@redhat.com>
 <CA+ZsWTUvCPU53PP+WZV45c2Jk01n+08bkany4qYD=t4Qz4yEfA@mail.gmail.com>
Message-ID: <afdb699f-a3f0-eb67-3b71-fe1d5fc39df4@redhat.com>

> Thanks Honza. I have raised these on both upstream projects.

Thanks

> I will leave upto implementer how best this can be done, considering the
> technical limitations you mentioned.
> 
> https://github.com/corosync/corosync-qdevice/issues/13
> https://github.com/ClusterLabs/booth/issues/99
> 
> Thanks,
> Rohit
> 
> On Thu, Aug 13, 2020 at 1:03 PM Jan Friesse <jfriesse at redhat.com> wrote:
> 
>> Hi Rohit,
>>
>>> Hi Honza,
>>> Thanks for your reply. Please find the attached image below:
>>>
>>> [image: image.png]
>>>
>>> Yes, I am talking about pacemaker alerts only.
>>>
>>> Please find my suggestions/requirements below:
>>>
>>> *Booth:*
>>> 1. Node5 booth-arbitrator should be able to give event when any of the
>>> booth node joins or leaves. booth-ip can be passed in event.
>>
>> This is not how booth works. Ticket leader (so site booth, never
>> arbitrator) executes election and get replies from other
>> sites/arbitrator. Follower executes election when leader hasn't for
>> configured timeout.
>>
>> What I want to say is, that there is no "membership" - as in (for
>> example) corosync fashion.
>>
>> The best we could get is the rough estimation based on election
>> request/replies.
>>
>>> 2. Event when booth-arbitrator is up successfully and has started
>>> monitoring the booth nodes.
>>
>> This is basically start of service. I think it's doable with small
>> change in unit file (something like
>>
>> https://northernlightlabs.se/2014-07-05/systemd-status-mail-on-unit-failure.html
>> )
>>
>>> 2. Geo site booth should be able to give event when its booth peers
>>> joins/leaves. For example, Geo site1 gives an event when node5
>>> booth-arbitrator joins/leaves OR site2 booth joins/leaves.  booth-ip can
>> be
>>> passed in event.
>>> 3. On ticket movements (revoke/grant), every booth node(Site1/2 and
>> node5)
>>> should give events.
>>
>> That would be doable
>>
>>>
>>> Note: pacemaker alerts works in a cluster. Since, arbitrator is a
>>> non-cluster node, not sure how exactly it will work there. But this is
>> good
>>> to have feature.
>>>
>>> *Qnetd/Qdevice:*
>>> This is similar to above.
>>> 1. Node5 qnetd should be able to raise an event when any of the cluster
>>> node joins/leaves the quorum.
>>
>> Doable
>>
>>> 2. Event when qnetd is up successfully and has started monitoring the
>>> cluster nodes
>>
>> Qnetd itself is not monitoring qdevice nodes (it doesn't have list of
>> nodes). It monitors node status after node joins (= it would be possible
>> to trigger event on leave). So that may be enough.
>>
>>> 3. Cluster node should be able to give event when any of the quorum node
>>> leaves/joins.
>>
>> You mean qdevice should be able to trigger event when connected to qnetd?
>>
>>>
>>> If you see on high level, then these are kind of node/resource events wrt
>>> booth and qnetd/qdevice.
>>
>> Yeah
>>
>>>
>>> As of today wrt booth/qnetd, I don't see any provision where any of the
>>> nodes gives any event when its peer leaves/joins. This makes it difficult
>>> to know whether geo sites nodes can see booth-arbitrator or not. This is
>>
>> Got it. That's exactly what would be really problematic to implement,
>> because of no "membership" in booth. It would be, however, possible to
>> implement message when ticket was granted/rejected and have a list of
>> other booths replies and what was their votes.
>>
>>> true the other way around also where booth-arbitrator cannot see geo
>> booth
>>> sites.
>>> I am not sure how others are doing it in today's deployment, but I see
>> need
>>> of monitoring of every other booth/qnet node. So that on basis of event,
>>> appropriate alarms can be raised and action can be taken accordingly.
>>>
>>> Please let me know if you agree on the usecases. I'll raise
>> feature-request
>>
>> I can agree on usecases, but (especially with booth) there are technical
>> problems on realizing them.
>>
>>> on the pacemaker upstream project accordingly.
>>
>> Please use booth (https://github.com/ClusterLabs/booth) and qdevice
>> (https://github.com/corosync/corosync-qdevice) upstream rather than
>> pacemaker, because these requests has really nothing to do with pcmk.
>>
>> Regards,
>>     honza
>>
>>>
>>> Thanks,
>>> Rohit
>>>
>>> On Wed, Aug 12, 2020 at 8:58 PM Jan Friesse <jfriesse at redhat.com> wrote:
>>>
>>>> Hi Rohit,
>>>>
>>>> Rohit Saini napsal(a):
>>>>> Hi Team,
>>>>>
>>>>> Question-1:
>>>>> Similar to pcs alerts, do we have something similar for qdevice/qnetd?
>>>> This
>>>>
>>>> You mean pacemaker alerts right?
>>>>
>>>>> is to detect asynchronously if any of the member is
>>>> unreachable/joined/left
>>>>> and if that member is qdevice or qnetd.
>>>>
>>>> Nope but actually shouldn't be that hard to implement. What exactly
>>>> would you like to see there?
>>>>
>>>>>
>>>>> Question-2:
>>>>> Same above question for booth nodes and arbitrator. Is there any way to
>>>>> receive events from booth daemon?
>>>>
>>>> Not directly (again, shouldn't be that hard to implement). But pacemaker
>>>> alerts should be triggered when service changes state because of ticket
>>>> grant/reject, isn't it?
>>>>
>>>>>
>>>>> My main objective is to see if these daemons give events related to
>>>>> their internal state transitions  and raise some alarms accordingly.
>> For
>>>>> example, boothd arbitrator is unreachable, ticket moved from x to y,
>> etc.
>>>>
>>>> I don't think "boothd arbitrator is unreachable" alert is really doable.
>>>> Ticket moved from x to y would be probably two alerts - 1. ticket
>>>> rejected on X and 2. granted on Y.
>>>>
>>>> Would you mind to elaborate a bit more on events you would like to see
>>>> and potentially open issue for upstream project (or, if you have a RH
>>>> subscription try to contact GSS, so I get more time to work on this
>> issue).
>>>>
>>>> Regards,
>>>>      Honza
>>>>
>>>>>
>>>>> Thanks,
>>>>> Rohit
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Manage your subscription:
>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>
>>>>> ClusterLabs home: https://www.clusterlabs.org/
>>>>>
>>>>
>>>>
>>>
>>
>>
> 


From kwenning at redhat.com  Mon Aug 17 03:52:02 2020
From: kwenning at redhat.com (Klaus Wenninger)
Date: Mon, 17 Aug 2020 09:52:02 +0200
Subject: [ClusterLabs] Clear Pending Fencing Action
In-Reply-To: <CAPiuu9-cppfbxmJ3qMHrOzQ-z9zXbqTf_HvVdYAd4HKkgR3GKw@mail.gmail.com>
References: <20200803032623.75154600275@iwtm.local>
 <CAPiuu9-cppfbxmJ3qMHrOzQ-z9zXbqTf_HvVdYAd4HKkgR3GKw@mail.gmail.com>
Message-ID: <0f8d1c4f-850f-0c1c-368d-2f668af75ebe@redhat.com>

On 8/3/20 7:04 AM, Reid Wahl wrote:
> Hi, ????. `stonith_admin --cleanup` doesn't get rid of pending
> actions, only failed ones. You might be hitting
> https://bugs.clusterlabs.org/show_bug.cgi?id=5401.
>
> I believe a simultaneous reboot of both nodes will clear the pending
> actions. I don't recall whether there's any other way to clear them.
Even simultaneous rebooting might be some kind of a challenge.
When a node is coming up it will request the history from the
running nodes. Thus a simultaneous reboot might not be
simultaneous enough so that the nodes aren't still able to
pass this list from one to another.
To be on the safe side you would have to shut down all nodes
and fire them up again.

If it is not the bug stated above and we are talking about a
pending fence-action that was going on on node that itself
just got fenced (and is still down) pacemaker coming up on
that node will remove (fail) the pending fence-action.

Just for completeness:
'stonith_admin --cleanup' cleans everything that is 'just'
history (failed & successful) whereas weather an attempt is
still pending does have an effect on how fencing is working.
As nobody would expect a history-cleanup to influence
behavior, not touching pending actions is a safety measure
and not a bug.

Klaus


>
> On Sun, Aug 2, 2020 at 8:26 PM ???? ??????? <elias at po-mayak.ru
> <mailto:elias at po-mayak.ru>> wrote:
>
>     Hello!
>
>     ?
>
>     After troubleshooting 2-Node cluster, crm_mon deprecated actions
>     are displayed in ?Pending Fencing Action:? list.
>
>     How can I delete them.
>
>     ?stonith_admin --cleanup --history=*? does not delete it.
>
>     ?
>
>     ?
>
>     ? ?????????,
>     ???? ???????
>     elias at po-mayak.ru
>
>     ?
>
>     _______________________________________________
>     Manage your subscription:
>     https://lists.clusterlabs.org/mailman/listinfo/users
>
>     ClusterLabs home: https://www.clusterlabs.org/
>
>
>
> -- 
> Regards,
>
> Reid Wahl, RHCA
> Software Maintenance Engineer, Red Hat
> CEE - Platform Support Delivery - ClusterHA
>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200817/58a5a531/attachment.htm>

From kwenning at redhat.com  Mon Aug 17 04:09:32 2020
From: kwenning at redhat.com (Klaus Wenninger)
Date: Mon, 17 Aug 2020 10:09:32 +0200
Subject: [ClusterLabs] Coming in Pacemaker 2.0.5: on-fail=demote /
 no-quorum-policy=demote
In-Reply-To: <cb172f00a8ed1abdaa1b9cdc4c01c1a5c7aaf7c1.camel@redhat.com>
References: <cb172f00a8ed1abdaa1b9cdc4c01c1a5c7aaf7c1.camel@redhat.com>
Message-ID: <e94f0259-1912-3461-9639-9322aba2da92@redhat.com>

On 8/10/20 6:47 PM, Ken Gaillot wrote:
> Hi all,
>
> Looking ahead to the Pacemaker 2.0.5 release expected at the end of
> this year, here is a new feature already in the master branch.
>
> When configuring resource operations, Pacemaker lets you set an "on-
> fail" policy to specify whether to restart the resource, fence the
> node, etc., if the operation fails. With 2.0.5, a new possible value
> will be "demote", which will mean "demote this resource but do not
> fully restart it".
>
> "Demote" will be a valid value only for promote actions, and for
> recurring monitors with "role" set to "Master".
>
> Once the resource is demoted, it will be eligible for promotion again,
> so if the promotion scores have not changed, a promote on the same node
> may be attempted. If this is not desired, the agent can change the
> promotion scores either in the failed monitor or the demote.
>
> The intended use case is an application where a successful demote 
> assures a well-functioning service, and a full restart would be
> unnecessarily heavyweight. A large database might be an example.
>
> Similarly, Pacemaker offers the cluster-wide "no-quorum-policy" option
> to specify what happens to resources when quorum is lost (the default
> being to stop them). With 2.0.5, "demote" will be a possible value here
> as well, and will mean "demote all promotable resources and stop all
> other resources".
When using the new "no-quorum-policy" together with SBD please be
sure to use an SBD version that has
https://github.com/ClusterLabs/sbd/pull/111.

Klaus
>
> The intended use case is an application that cannot cause any harm
> after being demoted, and may be useful in a demoted role even if there
> is no quorum. A database that operates read-only when demoted and
> doesn't depend on any non-promotable resources might be an example.
>
> Happy clustering :)


From fabbione at fabbione.net  Mon Aug 17 05:33:05 2020
From: fabbione at fabbione.net (Fabio M. Di Nitto)
Date: Mon, 17 Aug 2020 11:33:05 +0200
Subject: [ClusterLabs] kronosnet v1.19 released
Message-ID: <a470cac1-d95b-3b6c-482d-7eaaf3648b31@fabbione.net>

All,

We are pleased to announce the general availability of kronosnet v1.19

kronosnet (or knet for short) is the new underlying network protocol for 
Linux HA components (corosync), that features the ability to use 
multiple links between nodes, active/active and active/passive link 
failover policies, automatic link recovery, FIPS compliant encryption 
(nss and/or openssl), automatic PMTUd and in general better performance 
compared to the old network protocol.

Highlights in this release:

* Add native support for openssl 3.0 (drop API COMPAT macros).
* Code cleanup of public APIs. Lots of lines of code moved around, no
   functional changes.
* Removed kronosnetd unsupported code completely
* Removed unused poc-code from the source tree
* Make sure to initialize epoll events structures

Known issues in this release:

* None

The source tarballs can be downloaded here:

https://www.kronosnet.org/releases/

Upstream resources and contacts:

https://kronosnet.org/
https://github.com/kronosnet/kronosnet/
https://ci.kronosnet.org/
https://trello.com/kronosnet (TODO list and activities tracking)
https://goo.gl/9ZvkLS (google shared drive with presentations and diagrams)
IRC: #kronosnet on Freenode
https://lists.kronosnet.org/mailman/listinfo/users
https://lists.kronosnet.org/mailman/listinfo/devel
https://lists.kronosnet.org/mailman/listinfo/commits

Cheers,
The knet developer team


From fabbione at fabbione.net  Mon Aug 17 05:35:13 2020
From: fabbione at fabbione.net (Fabio M. Di Nitto)
Date: Mon, 17 Aug 2020 11:35:13 +0200
Subject: [ClusterLabs] kronosnet v1.x series and future support / development
Message-ID: <02f339d5-f6e5-28e0-e5d9-68e1ac910bd9@fabbione.net>

All,

kronosnet (or knet for short) is the new underlying network protocol for 
Linux HA components (corosync), that features the ability to use 
multiple links between nodes, active/active and active/passive link 
failover policies, automatic link recovery, FIPS compliant encryption 
(nss and/or openssl), automatic PMTUd and in general better performance 
compared to the old network protocol.

After several weeks / months without any major bug reported, starting 
with v1.19 release, we are going to lock down the 1.x series to only 2 
kind of changes:

* Bug fixes
* Onwire compatibility changes to allow rolling upgrades with the v2.x
   series (if necessary at all)

Upstream will continue to support v1.x for at least 12 months after v2.x 
will be released (date unknown, no really, we didn't even start the 
development).

If you have any amazing ideas on what v2.x should include, please file 
issues here:

https://github.com/kronosnet/kronosnet/issues

Or check the current TODO list here:

https://trello.com/kronosnet

Cheers,
The knet developer team

From kgaillot at redhat.com  Mon Aug 17 10:54:21 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Mon, 17 Aug 2020 09:54:21 -0500
Subject: [ClusterLabs] why is node fenced ?
In-Reply-To: <1954017432.49729402.1597400271575.JavaMail.zimbra@helmholtz-muenchen.de>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
 <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>
 <f411d804d82fd34b7ae0974fcd9f17821a115299.camel@redhat.com>
 <1954017432.49729402.1597400271575.JavaMail.zimbra@helmholtz-muenchen.de>
Message-ID: <426bbd8063885706b0e8fdd3dab81be7e0c9f25d.camel@redhat.com>

On Fri, 2020-08-14 at 12:17 +0200, Lentes, Bernd wrote:
> 
> ----- On Aug 10, 2020, at 11:59 PM, kgaillot kgaillot at redhat.com
> wrote:
> > The most recent transition is aborted, but since all its actions
> > are
> > complete, the only effect is to trigger a new transition.
> > 
> > We should probably rephrase the log message. In fact, the whole
> > "transition" terminology is kind of obscure. It's hard to come up
> > with
> > something better though.
> > 
> 
> Hi Ken,
> 
> i don't get it. How can s.th. be aborted which is already completed ?

I agree the wording is confusing :)

>From the code's point of view, the actions in the transition are
complete, but the transition itself (as an abstract entity) remains
current until the next one starts. However that's academic and
meaningless from a user's point of view, so the log messages should be
reworded.

> Bernd
> Helmholtz Zentrum M?nchen
> 
> Helmholtz Zentrum Muenchen
> Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
> Ingolstaedter Landstr. 1
> 85764 Neuherberg
> www.helmholtz-muenchen.de
> Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
> Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin
> Guenther
> Registergericht: Amtsgericht Muenchen HRB 6466
> USt-IdNr: DE 129521671
-- 
Ken Gaillot <kgaillot at redhat.com>


From kgaillot at redhat.com  Mon Aug 17 11:09:02 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Mon, 17 Aug 2020 10:09:02 -0500
Subject: [ClusterLabs] why is node fenced ?
In-Reply-To: <1230152204.50050895.1597430263067.JavaMail.zimbra@helmholtz-muenchen.de>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
 <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>
 <1230152204.50050895.1597430263067.JavaMail.zimbra@helmholtz-muenchen.de>
Message-ID: <84a98a74897f0b5523f44b23a415fcde5bff0de6.camel@redhat.com>

On Fri, 2020-08-14 at 20:37 +0200, Lentes, Bernd wrote:
> ----- On Aug 9, 2020, at 10:17 PM, Bernd Lentes 
> bernd.lentes at helmholtz-muenchen.de wrote:
> 
> 
> > > So this appears to be the problem. From these logs I would guess
> > > the
> > > successful stop on ha-idg-1 did not get written to the CIB for
> > > some
> > > reason. I'd look at the pe input from this transition on ha-idg-2 
> > > to
> > > confirm that.
> > > 
> > > Without the DC knowing about the stop, it tries to schedule a new
> > > one,
> > > but the node is shutting down so it can't do it, which means it
> > > has to
> > > be fenced.
> 
> I checked all relevant pe-files in this time period.
> This is what i found out (i just write the important entries):
> 
> ha-idg-1:~/why-fenced/ha-idg-1/pengine # crm_simulate -S -x pe-input-
> 3116 -G transition-3116.xml -D transition-3116.dot
> Current cluster status:
>  ...
>  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Started ha-idg-1
> Transition Summary:
>  ...
> * Migrate    vm_nextcloud           ( ha-idg-1 -> ha-idg-2 )
> Executing cluster transition:
>  * Resource action: vm_nextcloud    migrate_from on ha-idg-2 <=======
> migrate vm_nextcloud
>  * Resource action: vm_nextcloud    stop on ha-idg-1 
>  * Pseudo action:   vm_nextcloud_start_0
> Revised cluster status:
> Node ha-idg-1 (1084777482): standby
> Online: [ ha-idg-2 ]
> vm_nextcloud   (ocf::heartbeat:VirtualDomain): Started ha-idg-2
> 
> 
> ha-idg-1:~/why-fenced/ha-idg-1/pengine # crm_simulate -S -x pe-error-
> 48 -G transition-4514.xml -D transition-4514.dot
> Current cluster status:
> Node ha-idg-1 (1084777482): standby
> Online: [ ha-idg-2 ]
> ...
>  vm_nextcloud   (ocf::heartbeat:VirtualDomain): FAILED[ ha-idg-2 ha-
> idg-1 ] <====== migration failed
> Transition Summary:
> ..
>  * Recover    vm_nextcloud            (             ha-idg-2 )
> Executing cluster transition:
>  * Resource action: vm_nextcloud    stop on ha-idg-2
>  * Resource action: vm_nextcloud    stop on ha-idg-1 
>  * Resource action: vm_nextcloud    start on ha-idg-2
>  * Resource action: vm_nextcloud    monitor=30000 on ha-idg-2
> Revised cluster status:
>  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Started ha-idg-2
> 
> ha-idg-1:~/why-fenced/ha-idg-1/pengine # crm_simulate -S -x pe-input-
> 3117 -G transition-3117.xml -D transition-3117.dot
> Current cluster status:
> Node ha-idg-1 (1084777482): standby
> Online: [ ha-idg-2 ]
>  vm_nextcloud   (ocf::heartbeat:VirtualDomain): FAILED ha-idg-2
> <====== start on ha-idg-2 failed
> Transition Summary:
>  * Stop       vm_nextcloud     ( ha-idg-2 )   due to node
> availability <==== stop vm_nextcloud (what means due to node
> availability ?)

"Due to node availability" means no node is allowed to run the
resource, so it has to be stopped.

> Executing cluster transition:
>  * Resource action: vm_nextcloud    stop on ha-idg-2
> Revised cluster status:
>  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped
> 
> ha-idg-1:~/why-fenced/ha-idg-1/pengine # crm_simulate -S -x pe-input-
> 3118 -G transition-4516.xml -D transition-4516.dot
> Current cluster status:
> Node ha-idg-1 (1084777482): standby
> Online: [ ha-idg-2 ]
>  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped
> <============== vm_nextcloud is stopped
> Transition Summary:
>  * Shutdown ha-idg-1
> Executing cluster transition:
>  * Resource action: vm_nextcloud    stop on ha-idg-1 <==== why stop ?
> It is already stopped

I'm not sure, I'd have to see the pe input.

> Revised cluster status:
>  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped
> 
> ha-idg-1:~/why-fenced/ha-idg-2/pengine # crm_simulate -S -x pe-input-
> 3545 -G transition-0.xml -D transition-0.dot
> Current cluster status:
> Node ha-idg-1 (1084777482): pending
> Online: [ ha-idg-2 ]
>  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped <======
> vm_nextcloud is stopped
> Transition Summary:
> 
> Executing cluster transition:
> Using the original execution date of: 2020-07-20 15:05:33Z
> Revised cluster status:
> vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped
> 
> ha-idg-1:~/why-fenced/ha-idg-2/pengine # crm_simulate -S -x pe-warn-
> 749 -G transition-1.xml -D transition-1.dot
> Current cluster status:
> Node ha-idg-1 (1084777482): OFFLINE (standby)
> Online: [ ha-idg-2 ]
>  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped <=======
> vm_nextcloud is stopped
> Transition Summary:
>  * Fence (Off) ha-idg-1 'resource actions are unrunnable'
> Executing cluster transition:
>  * Fencing ha-idg-1 (Off)
>  * Pseudo action:   vm_nextcloud_stop_0 <======= why stop ? It is
> already stopped ?
> Revised cluster status:
> Node ha-idg-1 (1084777482): OFFLINE (standby)
> Online: [ ha-idg-2 ]
>  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped
> 
> I don't understand why the cluster tries to stop a resource which is
> already stopped.
> 
> Bernd
> Helmholtz Zentrum M?nchen
> 
> Helmholtz Zentrum Muenchen
> Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
> Ingolstaedter Landstr. 1
> 85764 Neuherberg
> www.helmholtz-muenchen.de
> Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
> Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin
> Guenther
> Registergericht: Amtsgericht Muenchen HRB 6466
> USt-IdNr: DE 129521671
> 
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> ClusterLabs home: https://www.clusterlabs.org/
-- 
Ken Gaillot <kgaillot at redhat.com>


From kgaillot at redhat.com  Mon Aug 17 11:19:45 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Mon, 17 Aug 2020 10:19:45 -0500
Subject: [ClusterLabs] Antw: [EXT] Stonith failing
In-Reply-To: <1118615335.1895.1597410544148@www>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
Message-ID: <73d6ecf113098a3154a2e7db2e2a59557272024a.camel@redhat.com>

On Fri, 2020-08-14 at 15:09 +0200, Gabriele Bulfon wrote:
> Thanks to all your suggestions, I now have the systems with stonith
> configured on ipmi.

A word of caution: if the IPMI is on-board -- i.e. it shares the same
power supply as the computer -- power becomes a single point of
failure. If the node loses power, the other node can't fence because
the IPMI is also down, and the cluster can't recover.

Some on-board IPMI controllers can share an Ethernet port with the main
computer, which would be a similar situation.

It's best to have a backup fencing method when using IPMI as the
primary fencing method. An example would be an intelligent power switch
or sbd.

> Two questions:
> - how can I simulate a stonith situation to check that everything is
> ok?
> - considering that I have both nodes with stonith against the other
> node, once the two nodes can communicate, how can I be sure the two
> nodes will not try to stonith each other?
>  
> :)
> Thanks!
> Gabriele
> 
>  
>  
> Sonicle S.r.l. : http://www.sonicle.com
> Music: http://www.gabrielebulfon.com
> Quantum Mechanics : http://www.cdbaby.com/cd/gabrielebulfon
> 
> 
> 
> Da: Gabriele Bulfon <gbulfon at sonicle.com>
> A: Cluster Labs - All topics related to open-source clustering
> welcomed <users at clusterlabs.org>
> Data: 29 luglio 2020 14.22.42 CEST
> Oggetto: Re: [ClusterLabs] Antw: [EXT] Stonith failing
> 
> 
> >  
> > It is a ZFS based illumos system.
> > I don't think SBD is an option.
> > Is there a reliable ZFS based stonith?
> >  
> > Gabriele
> > 
> >  
> >  
> > Sonicle S.r.l. : http://www.sonicle.com
> > Music: http://www.gabrielebulfon.com
> > Quantum Mechanics : http://www.cdbaby.com/cd/gabrielebulfon
> > 
> > 
> > 
> > Da: Andrei Borzenkov <arvidjaar at gmail.com>
> > A: Cluster Labs - All topics related to open-source clustering
> > welcomed <users at clusterlabs.org>
> > Data: 29 luglio 2020 9.46.09 CEST
> > Oggetto: Re: [ClusterLabs] Antw: [EXT] Stonith failing
> > 
> > 
> > >  
> > > 
> > > On Wed, Jul 29, 2020 at 9:01 AM Gabriele Bulfon <
> > > gbulfon at sonicle.com> wrote:
> > > > That one was taken from a specific implementation on Solaris
> > > > 11.
> > > > The situation is a dual node server with shared storage
> > > > controller: both nodes see the same disks concurrently.
> > > > Here we must be sure that the two nodes are not going to
> > > > import/mount the same zpool at the same time, or we will
> > > > encounter data corruption:
> > > > 
> > > 
> > >  
> > > ssh based "stonith" cannot guarantee it.
> > >  
> > > > node 1 will be perferred for pool 1, node 2 for pool 2, only in
> > > > case one of the node goes down or is taken offline the
> > > > resources should be first free by the leaving node and taken by
> > > > the other node.
> > > >  
> > > > Would you suggest one of the available stonith in this case?
> > > >  
> > > > 
> > > 
> > >  
> > > IPMI, managed PDU, SBD ...
> > > In practice, the only stonith method that works in case of
> > > complete node outage including any power supply is SBD.
-- 
Ken Gaillot <kgaillot at redhat.com>


From kadlecsik.jozsef at wigner.hu  Mon Aug 17 06:12:36 2020
From: kadlecsik.jozsef at wigner.hu (=?UTF-8?Q?Kadlecsik_J=C3=B3zsef?=)
Date: Mon, 17 Aug 2020 12:12:36 +0200 (CEST)
Subject: [ClusterLabs] node utilization attributes are lost during upgrade
Message-ID: <alpine.DEB.2.23.453.2008171134050.25087@blackhole.kfki.hu>

Hello,

At upgrading a corosync/pacemaker/libvirt/KVM cluster from Debian stretch 
to buster, all the node utilization attributes were erased from the 
configuration. However, the same attributes were kept at the VirtualDomain 
resources. This resulted that all resources with utilization attributes 
were stopped.

The documentation says: "You can name utilization attributes according to 
your preferences and define as many name/value pairs as your configuration 
needs.", so one assumes utilization attributes are kept during upgrades, 
for nodes and resources as well.

The corosync incompatibility made the upgrade more stressful anyway and 
the stopping of the resources came out of the blue. The resources could 
not be started of course - and there were no log warning/error messages 
that the resources are not started because the utilization constrains 
could not be satisfied. Pacemaker logs a lot (from admin point of view it 
is too much), but in this case there was no indication why the resources 
could not be started (or we were unable to find it in the logs?). So we 
wasted a lot of time with debugging the VirtualDomain agent.

Currently we run the cluster with the placement-strategy set to default.

In my opinion node attributes should be kept and preserved during an 
upgrade. Also, it should be logged when a resource must be stopped/cannot 
be started because the utilization constrains cannot be satisfied.

Best regards,
Jozsef
--
E-mail : kadlecsik.jozsef at wigner.hu
PGP key: https://wigner.hu/~kadlec/pgp_public_key.txt
Address: Wigner Research Centre for Physics
         H-1525 Budapest 114, POB. 49, Hungary

From sameerdhiman at gmail.com  Mon Aug 17 08:40:22 2020
From: sameerdhiman at gmail.com (Sameer Dhiman)
Date: Mon, 17 Aug 2020 18:10:22 +0530
Subject: [ClusterLabs] Beginner Question about VirtualDomain
Message-ID: <CANNzTaALh5XDn6HMmEwA3KMoef2Kc0if9MdSeUTXtCEbMW7VhA@mail.gmail.com>

Hi,

I am a beginner using pacemaker and corosync. I am trying to set up
a cluster of HA KVM guests as described by Alteeve wiki (CentOS-6) but in
CentOS-8.2. My R&D  setup is described below

Physical Host running CentOS-8.2 with Nested Virtualization
2 x CentOS-8.2 guest machines as Cluster Node 1 and 2.
WinXP as a HA guest.

1. drbd --> dlm --> lvmlockd --> LVM-activate --> gfs2 (guest machine
definitions)
2. drbd --> dlm --> lvmlockd --> LVM-activate --> raw-lv (guest machine HDD)

Question(s):
1. How to prevent guest startup until gfs2 and raw-lv are available? In
CentOS-6 Alteeve used autostart=0 in the <vm > tag. Is there any similar
option in pacemaker because I did not find it in the documentation?

2. Suppose, If I configure constraint order gfs2 and raw-lv then guest
machine. Stopping the guest machine would also stop the complete service
tree so how can I prevent this?

-- 
Sameer Dhiman
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200817/27502ef1/attachment.htm>

From kgaillot at redhat.com  Mon Aug 17 16:38:09 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Mon, 17 Aug 2020 15:38:09 -0500
Subject: [ClusterLabs] node utilization attributes are lost during
 upgrade
In-Reply-To: <alpine.DEB.2.23.453.2008171134050.25087@blackhole.kfki.hu>
References: <alpine.DEB.2.23.453.2008171134050.25087@blackhole.kfki.hu>
Message-ID: <08e1ec020aef4f25f2eadc8294135d2f611a2b76.camel@redhat.com>

On Mon, 2020-08-17 at 12:12 +0200, Kadlecsik J?zsef wrote:
> Hello,
> 
> At upgrading a corosync/pacemaker/libvirt/KVM cluster from Debian
> stretch 
> to buster, all the node utilization attributes were erased from the 
> configuration. However, the same attributes were kept at the
> VirtualDomain 
> resources. This resulted that all resources with utilization
> attributes 
> were stopped.

Ouch :(

There are two types of node attributes, transient and permanent.
Transient attributes last only until pacemaker is next stopped on the
node, while permanent attributes persist between reboots/restarts.

If you configured the utilization attributes with crm_attribute -z/
--utilization, it will default to permanent, but it's possible to
override that with -l/--lifetime reboot (or equivalently, -t/--type
status).

Permanent node attributes should definitely not be erased in an
upgrade.

> 
> The documentation says: "You can name utilization attributes
> according to 
> your preferences and define as many name/value pairs as your
> configuration 
> needs.", so one assumes utilization attributes are kept during
> upgrades, 
> for nodes and resources as well.
> 
> The corosync incompatibility made the upgrade more stressful anyway
> and 
> the stopping of the resources came out of the blue. The resources
> could 
> not be started of course - and there were no log warning/error
> messages 
> that the resources are not started because the utilization
> constrains 
> could not be satisfied. Pacemaker logs a lot (from admin point of
> view it 
> is too much), but in this case there was no indication why the
> resources 
> could not be started (or we were unable to find it in the logs?). So
> we 
> wasted a lot of time with debugging the VirtualDomain agent.
> 
> Currently we run the cluster with the placement-strategy set to
> default.
> 
> In my opinion node attributes should be kept and preserved during an 
> upgrade. Also, it should be logged when a resource must be
> stopped/cannot 
> be started because the utilization constrains cannot be satisfied.
> 
> Best regards,
> Jozsef
> --
> E-mail : kadlecsik.jozsef at wigner.hu
> PGP key: https://wigner.hu/~kadlec/pgp_public_key.txt
> Address: Wigner Research Centre for Physics
>          H-1525 Budapest 114, POB. 49, Hungary
-- 
Ken Gaillot <kgaillot at redhat.com>


From jgdr at dalibo.com  Mon Aug 17 16:39:19 2020
From: jgdr at dalibo.com (Jehan-Guillaume de Rorthais)
Date: Mon, 17 Aug 2020 22:39:19 +0200
Subject: [ClusterLabs] Antw: [EXT] Stonith failing
In-Reply-To: <73d6ecf113098a3154a2e7db2e2a59557272024a.camel@redhat.com>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www>
 <1118615335.1895.1597410544148@www>
 <73d6ecf113098a3154a2e7db2e2a59557272024a.camel@redhat.com>
Message-ID: <20200817223919.6187c711@firost>

On Mon, 17 Aug 2020 10:19:45 -0500
Ken Gaillot <kgaillot at redhat.com> wrote:

> On Fri, 2020-08-14 at 15:09 +0200, Gabriele Bulfon wrote:
> > Thanks to all your suggestions, I now have the systems with stonith
> > configured on ipmi.  
> 
> A word of caution: if the IPMI is on-board -- i.e. it shares the same
> power supply as the computer -- power becomes a single point of
> failure. If the node loses power, the other node can't fence because
> the IPMI is also down, and the cluster can't recover.
> 
> Some on-board IPMI controllers can share an Ethernet port with the main
> computer, which would be a similar situation.
> 
> It's best to have a backup fencing method when using IPMI as the
> primary fencing method. An example would be an intelligent power switch
> or sbd.

How SBD would be useful in this scenario? Poison pill will not be swallowed by
the dead node... Is it just to wait for the watchdog timeout?

Regards,

From kgaillot at redhat.com  Mon Aug 17 17:07:52 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Mon, 17 Aug 2020 16:07:52 -0500
Subject: [ClusterLabs] Antw: [EXT] Stonith failing
In-Reply-To: <20200817223919.6187c711@firost>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
 <73d6ecf113098a3154a2e7db2e2a59557272024a.camel@redhat.com>
 <20200817223919.6187c711@firost>
Message-ID: <8a2c89afdc005dc102e35df222056d08e08c21af.camel@redhat.com>

On Mon, 2020-08-17 at 22:39 +0200, Jehan-Guillaume de Rorthais wrote:
> On Mon, 17 Aug 2020 10:19:45 -0500
> Ken Gaillot <kgaillot at redhat.com> wrote:
> 
> > On Fri, 2020-08-14 at 15:09 +0200, Gabriele Bulfon wrote:
> > > Thanks to all your suggestions, I now have the systems with
> > > stonith
> > > configured on ipmi.  
> > 
> > A word of caution: if the IPMI is on-board -- i.e. it shares the
> > same
> > power supply as the computer -- power becomes a single point of
> > failure. If the node loses power, the other node can't fence
> > because
> > the IPMI is also down, and the cluster can't recover.
> > 
> > Some on-board IPMI controllers can share an Ethernet port with the
> > main
> > computer, which would be a similar situation.
> > 
> > It's best to have a backup fencing method when using IPMI as the
> > primary fencing method. An example would be an intelligent power
> > switch
> > or sbd.
> 
> How SBD would be useful in this scenario? Poison pill will not be
> swallowed by
> the dead node... Is it just to wait for the watchdog timeout?

Right, I meant watchdog-only SBD. Although now that I think about it,
I'm not sure of the details of if/how that would work. Klaus Wenninger
might have some insight.
-- 
Ken Gaillot <kgaillot at redhat.com>


From arvidjaar at gmail.com  Tue Aug 18 01:49:48 2020
From: arvidjaar at gmail.com (Andrei Borzenkov)
Date: Tue, 18 Aug 2020 08:49:48 +0300
Subject: [ClusterLabs] Antw: [EXT] Stonith failing
In-Reply-To: <20200817223919.6187c711@firost>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
 <73d6ecf113098a3154a2e7db2e2a59557272024a.camel@redhat.com>
 <20200817223919.6187c711@firost>
Message-ID: <cd796b78-ca9d-7fac-610c-5fa98fee6cea@gmail.com>

17.08.2020 23:39, Jehan-Guillaume de Rorthais ?????:
> On Mon, 17 Aug 2020 10:19:45 -0500
> Ken Gaillot <kgaillot at redhat.com> wrote:
> 
>> On Fri, 2020-08-14 at 15:09 +0200, Gabriele Bulfon wrote:
>>> Thanks to all your suggestions, I now have the systems with stonith
>>> configured on ipmi.  
>>
>> A word of caution: if the IPMI is on-board -- i.e. it shares the same
>> power supply as the computer -- power becomes a single point of
>> failure. If the node loses power, the other node can't fence because
>> the IPMI is also down, and the cluster can't recover.
>>
>> Some on-board IPMI controllers can share an Ethernet port with the main
>> computer, which would be a similar situation.
>>
>> It's best to have a backup fencing method when using IPMI as the
>> primary fencing method. An example would be an intelligent power switch
>> or sbd.
> 
> How SBD would be useful in this scenario? Poison pill will not be swallowed by
> the dead node... Is it just to wait for the watchdog timeout?
> 

Node is expected to commit suicide if SBD lost access to shared block
device. So either node swallowed poison pill and died or node died
because it realized it was impossible to see poison pill or node was
dead already. After watchdog timeout (twice watchdog timeout for safety)
we assume node is dead.

From kwenning at redhat.com  Tue Aug 18 02:21:50 2020
From: kwenning at redhat.com (Klaus Wenninger)
Date: Tue, 18 Aug 2020 08:21:50 +0200
Subject: [ClusterLabs] Antw: [EXT] Stonith failing
In-Reply-To: <cd796b78-ca9d-7fac-610c-5fa98fee6cea@gmail.com>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
 <73d6ecf113098a3154a2e7db2e2a59557272024a.camel@redhat.com>
 <20200817223919.6187c711@firost>
 <cd796b78-ca9d-7fac-610c-5fa98fee6cea@gmail.com>
Message-ID: <9fdd714d-ccb7-9cfa-8952-23d0e5336695@redhat.com>

On 8/18/20 7:49 AM, Andrei Borzenkov wrote:
> 17.08.2020 23:39, Jehan-Guillaume de Rorthais ?????:
>> On Mon, 17 Aug 2020 10:19:45 -0500
>> Ken Gaillot <kgaillot at redhat.com> wrote:
>>
>>> On Fri, 2020-08-14 at 15:09 +0200, Gabriele Bulfon wrote:
>>>> Thanks to all your suggestions, I now have the systems with stonith
>>>> configured on ipmi.  
>>> A word of caution: if the IPMI is on-board -- i.e. it shares the same
>>> power supply as the computer -- power becomes a single point of
>>> failure. If the node loses power, the other node can't fence because
>>> the IPMI is also down, and the cluster can't recover.
>>>
>>> Some on-board IPMI controllers can share an Ethernet port with the main
>>> computer, which would be a similar situation.
>>>
>>> It's best to have a backup fencing method when using IPMI as the
>>> primary fencing method. An example would be an intelligent power switch
>>> or sbd.
>> How SBD would be useful in this scenario? Poison pill will not be swallowed by
>> the dead node... Is it just to wait for the watchdog timeout?
>>
> Node is expected to commit suicide if SBD lost access to shared block
> device. So either node swallowed poison pill and died or node died
> because it realized it was impossible to see poison pill or node was
> dead already. After watchdog timeout (twice watchdog timeout for safety)
> we assume node is dead.
Yes, like this a suicide via watchdog will be triggered if there are
issues with thedisk. This is why it is important to have a reliable
watchdog with SBD even whenusing poison pill. As this alone would
make a single shared disk a SPOF, runningwith pacemaker integration
(default) a node with SBD will survive despite ofloosing the disk
when it has quorum and pacemaker looks healthy. As corosync-quorum
in 2-node-mode obviously won't be fit for this purpose SBD will switch
to checking for presence of both nodes if 2-node-flag is set.

Sorry for the lengthy explanation but the full picture is required
to understand whyit is sufficiently reliable and useful if configured
correctly.

Klaus


From Ulrich.Windl at rz.uni-regensburg.de  Tue Aug 18 03:04:44 2020
From: Ulrich.Windl at rz.uni-regensburg.de (Ulrich Windl)
Date: Tue, 18 Aug 2020 09:04:44 +0200
Subject: [ClusterLabs] Antw: [EXT] Re:  why is node fenced ?
In-Reply-To: <426bbd8063885706b0e8fdd3dab81be7e0c9f25d.camel@redhat.com>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
 <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>
 <f411d804d82fd34b7ae0974fcd9f17821a115299.camel@redhat.com>
 <1954017432.49729402.1597400271575.JavaMail.zimbra@helmholtz-muenchen.de>
 <426bbd8063885706b0e8fdd3dab81be7e0c9f25d.camel@redhat.com>
Message-ID: <5F3B7D8C020000A10003AACF@gwsmtp.uni-regensburg.de>

>>> Ken Gaillot <kgaillot at redhat.com> schrieb am 17.08.2020 um 16:54 in
Nachricht
<426bbd8063885706b0e8fdd3dab81be7e0c9f25d.camel at redhat.com>:
> On Fri, 2020-08-14 at 12:17 +0200, Lentes, Bernd wrote:
>> 
>> ----- On Aug 10, 2020, at 11:59 PM, kgaillot kgaillot at redhat.com 
>> wrote:
>> > The most recent transition is aborted, but since all its actions
>> > are
>> > complete, the only effect is to trigger a new transition.
>> > 
>> > We should probably rephrase the log message. In fact, the whole
>> > "transition" terminology is kind of obscure. It's hard to come up
>> > with
>> > something better though.
>> > 
>> 
>> Hi Ken,
>> 
>> i don't get it. How can s.th. be aborted which is already completed ?
> 
> I agree the wording is confusing :)
> 
> From the code's point of view, the actions in the transition are
> complete, but the transition itself (as an abstract entity) remains
> current until the next one starts. However that's academic and

Hi!

So when the "transition" had completed, it became a "state" ;-)
Aborting a "state" seems to be a "transition" started in my view.

> meaningless from a user's point of view, so the log messages should be
> reworded.

Another thing that always had confused me in the past was that concept of
"synapse"...
Good wording in error messages is a valuable resource!

Regards,
Ulrich

> 
>> Bernd
>> Helmholtz Zentrum M?nchen
>> 
>> Helmholtz Zentrum Muenchen
>> Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
>> Ingolstaedter Landstr. 1
>> 85764 Neuherberg
>> www.helmholtz-muenchen.de 
>> Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
>> Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin
>> Guenther
>> Registergericht: Amtsgericht Muenchen HRB 6466
>> USt-IdNr: DE 129521671
> -- 
> Ken Gaillot <kgaillot at redhat.com>
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




From Ulrich.Windl at rz.uni-regensburg.de  Tue Aug 18 03:10:07 2020
From: Ulrich.Windl at rz.uni-regensburg.de (Ulrich Windl)
Date: Tue, 18 Aug 2020 09:10:07 +0200
Subject: [ClusterLabs] Antw: Re:  Antw: [EXT] Stonith failing
In-Reply-To: <73d6ecf113098a3154a2e7db2e2a59557272024a.camel@redhat.com>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
 <73d6ecf113098a3154a2e7db2e2a59557272024a.camel@redhat.com>
Message-ID: <5F3B7ECF020000A10003AAD3@gwsmtp.uni-regensburg.de>

>>> Ken Gaillot <kgaillot at redhat.com> schrieb am 17.08.2020 um 17:19 in
Nachricht
<73d6ecf113098a3154a2e7db2e2a59557272024a.camel at redhat.com>:
> On Fri, 2020?08?14 at 15:09 +0200, Gabriele Bulfon wrote:
>> Thanks to all your suggestions, I now have the systems with stonith
>> configured on ipmi.
> 
> A word of caution: if the IPMI is on?board ?? i.e. it shares the same
> power supply as the computer ?? power becomes a single point of
> failure. If the node loses power, the other node can't fence because
> the IPMI is also down, and the cluster can't recover.

This may not always be true: We had servers with three(!) power supplies and a
BMC (what today is called "light-out management"). You could "power down" the
server, while the BMC was still operational (and thus could "power up" the
server again).
With standard PC architecture these days things seem to be a bit more
compicated (meaning "primitive")...

> 
> Some on?board IPMI controllers can share an Ethernet port with the main
> computer, which would be a similar situation.
> 
> It's best to have a backup fencing method when using IPMI as the
> primary fencing method. An example would be an intelligent power switch
> or sbd.
> 
>> Two questions:
>> ? how can I simulate a stonith situation to check that everything is
>> ok?
>> ? considering that I have both nodes with stonith against the other
>> node, once the two nodes can communicate, how can I be sure the two
>> nodes will not try to stonith each other?
>>  
>> :)
>> Thanks!
>> Gabriele
>> 
>>  
>>  
>> Sonicle S.r.l. : http://www.sonicle.com 
>> Music: http://www.gabrielebulfon.com 
>> Quantum Mechanics : http://www.cdbaby.com/cd/gabrielebulfon 
>> 
>> 
>> 
>> Da: Gabriele Bulfon <gbulfon at sonicle.com>
>> A: Cluster Labs ? All topics related to open?source clustering
>> welcomed <users at clusterlabs.org>
>> Data: 29 luglio 2020 14.22.42 CEST
>> Oggetto: Re: [ClusterLabs] Antw: [EXT] Stonith failing
>> 
>> 
>> >  
>> > It is a ZFS based illumos system.
>> > I don't think SBD is an option.
>> > Is there a reliable ZFS based stonith?
>> >  
>> > Gabriele
>> > 
>> >  
>> >  
>> > Sonicle S.r.l. : http://www.sonicle.com 
>> > Music: http://www.gabrielebulfon.com 
>> > Quantum Mechanics : http://www.cdbaby.com/cd/gabrielebulfon 
>> > 
>> > 
>> > 
>> > Da: Andrei Borzenkov <arvidjaar at gmail.com>
>> > A: Cluster Labs ? All topics related to open?source clustering
>> > welcomed <users at clusterlabs.org>
>> > Data: 29 luglio 2020 9.46.09 CEST
>> > Oggetto: Re: [ClusterLabs] Antw: [EXT] Stonith failing
>> > 
>> > 
>> > >  
>> > > 
>> > > On Wed, Jul 29, 2020 at 9:01 AM Gabriele Bulfon <
>> > > gbulfon at sonicle.com> wrote:
>> > > > That one was taken from a specific implementation on Solaris
>> > > > 11.
>> > > > The situation is a dual node server with shared storage
>> > > > controller: both nodes see the same disks concurrently.
>> > > > Here we must be sure that the two nodes are not going to
>> > > > import/mount the same zpool at the same time, or we will
>> > > > encounter data corruption:
>> > > > 
>> > > 
>> > >  
>> > > ssh based "stonith" cannot guarantee it.
>> > >  
>> > > > node 1 will be perferred for pool 1, node 2 for pool 2, only in
>> > > > case one of the node goes down or is taken offline the
>> > > > resources should be first free by the leaving node and taken by
>> > > > the other node.
>> > > >  
>> > > > Would you suggest one of the available stonith in this case?
>> > > >  
>> > > > 
>> > > 
>> > >  
>> > > IPMI, managed PDU, SBD ...
>> > > In practice, the only stonith method that works in case of
>> > > complete node outage including any power supply is SBD.
> ?? 
> Ken Gaillot <kgaillot at redhat.com>
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




From Ulrich.Windl at rz.uni-regensburg.de  Tue Aug 18 03:21:07 2020
From: Ulrich.Windl at rz.uni-regensburg.de (Ulrich Windl)
Date: Tue, 18 Aug 2020 09:21:07 +0200
Subject: [ClusterLabs] Antw: [EXT] node utilization attributes are lost
 during upgrade
In-Reply-To: <alpine.DEB.2.23.453.2008171134050.25087@blackhole.kfki.hu>
References: <alpine.DEB.2.23.453.2008171134050.25087@blackhole.kfki.hu>
Message-ID: <5F3B8163020000A10003AADB@gwsmtp.uni-regensburg.de>

>>> Kadlecsik J?zsef <kadlecsik.jozsef at wigner.hu> schrieb am 17.08.2020 um
12:12 in
Nachricht <alpine.DEB.2.23.453.2008171134050.25087 at blackhole.kfki.hu>:
> Hello,
> 
> At upgrading a corosync/pacemaker/libvirt/KVM cluster from Debian stretch 
> to buster, all the node utilization attributes were erased from the 
> configuration. However, the same attributes were kept at the VirtualDomain 
> resources. This resulted that all resources with utilization attributes 
> were stopped.
> 
> The documentation says: "You can name utilization attributes according to 
> your preferences and define as many name/value pairs as your configuration 
> needs.", so one assumes utilization attributes are kept during upgrades, 
> for nodes and resources as well.

Now that you mention it, I think we had it in the past with SLES, too.

> 
> The corosync incompatibility made the upgrade more stressful anyway and 
> the stopping of the resources came out of the blue. The resources could 
> not be started of course ? and there were no log warning/error messages 
> that the resources are not started because the utilization constrains 
> could not be satisfied. Pacemaker logs a lot (from admin point of view it 
> is too much), but in this case there was no indication why the resources 
> could not be started (or we were unable to find it in the logs?). So we 
> wasted a lot of time with debugging the VirtualDomain agent.

Also true: It's not very obvious when resources are not started due to
utilization constraints.

> 
> Currently we run the cluster with the placement?strategy set to default.
> 
> In my opinion node attributes should be kept and preserved during an 
> upgrade. Also, it should be logged when a resource must be stopped/cannot 
> be started because the utilization constrains cannot be satisfied.

+1

> 
> Best regards,
> Jozsef
> ??
> E?mail : kadlecsik.jozsef at wigner.hu 
> PGP key: https://wigner.hu/~kadlec/pgp_public_key.txt 
> Address: Wigner Research Centre for Physics
>          H?1525 Budapest 114, POB. 49, Hungary
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




From arvidjaar at gmail.com  Tue Aug 18 03:24:01 2020
From: arvidjaar at gmail.com (Andrei Borzenkov)
Date: Tue, 18 Aug 2020 10:24:01 +0300
Subject: [ClusterLabs] Antw: Re: Antw: [EXT] Stonith failing
In-Reply-To: <5F3B7ECF020000A10003AAD3@gwsmtp.uni-regensburg.de>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
 <73d6ecf113098a3154a2e7db2e2a59557272024a.camel@redhat.com>
 <5F3B7ECF020000A10003AAD3@gwsmtp.uni-regensburg.de>
Message-ID: <83aba38d-c9ea-1dff-e53b-14a9e0623d9d@gmail.com>

18.08.2020 10:10, Ulrich Windl ?????:
>>>> Ken Gaillot <kgaillot at redhat.com> schrieb am 17.08.2020 um 17:19 in
> Nachricht
> <73d6ecf113098a3154a2e7db2e2a59557272024a.camel at redhat.com>:
>> On Fri, 2020?08?14 at 15:09 +0200, Gabriele Bulfon wrote:
>>> Thanks to all your suggestions, I now have the systems with stonith
>>> configured on ipmi.
>>
>> A word of caution: if the IPMI is on?board ?? i.e. it shares the same
>> power supply as the computer ?? power becomes a single point of
>> failure. If the node loses power, the other node can't fence because
>> the IPMI is also down, and the cluster can't recover.
> 
> This may not always be true: We had servers with three(!) power supplies and a
> BMC (what today is called "light-out management"). You could "power down" the
> server, while the BMC was still operational (and thus could "power up" the
> server again).
> With standard PC architecture these days things seem to be a bit more
> compicated (meaning "primitive")...
> 

BMC is powered by standby voltage. If AC input to all of your power
supplies is cut off, there is no standby voltage anymore. Just try to
unplug all power cables and see if BMC is still accessible.

From Ulrich.Windl at rz.uni-regensburg.de  Tue Aug 18 03:35:28 2020
From: Ulrich.Windl at rz.uni-regensburg.de (Ulrich Windl)
Date: Tue, 18 Aug 2020 09:35:28 +0200
Subject: [ClusterLabs] Antw: Re:  Antw: Re: Antw: [EXT] Stonith failing
In-Reply-To: <83aba38d-c9ea-1dff-e53b-14a9e0623d9d@gmail.com>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
 <73d6ecf113098a3154a2e7db2e2a59557272024a.camel@redhat.com>
 <5F3B7ECF020000A10003AAD3@gwsmtp.uni-regensburg.de>
 <83aba38d-c9ea-1dff-e53b-14a9e0623d9d@gmail.com>
Message-ID: <5F3B84C0020000A10003AAE4@gwsmtp.uni-regensburg.de>

>>> Andrei Borzenkov <arvidjaar at gmail.com> schrieb am 18.08.2020 um 09:24 in
Nachricht <83aba38d-c9ea-1dff-e53b-14a9e0623d9d at gmail.com>:
> 18.08.2020 10:10, Ulrich Windl ?????:
>>>>> Ken Gaillot <kgaillot at redhat.com> schrieb am 17.08.2020 um 17:19 in
>> Nachricht
>> <73d6ecf113098a3154a2e7db2e2a59557272024a.camel at redhat.com>:
>>> On Fri, 2020?08?14 at 15:09 +0200, Gabriele Bulfon wrote:
>>>> Thanks to all your suggestions, I now have the systems with stonith
>>>> configured on ipmi.
>>>
>>> A word of caution: if the IPMI is on?board ?? i.e. it shares the same
>>> power supply as the computer ?? power becomes a single point of
>>> failure. If the node loses power, the other node can't fence because
>>> the IPMI is also down, and the cluster can't recover.
>> 
>> This may not always be true: We had servers with three(!) power supplies
and 
> a
>> BMC (what today is called "light-out management"). You could "power down" 
> the
>> server, while the BMC was still operational (and thus could "power up" the
>> server again).
>> With standard PC architecture these days things seem to be a bit more
>> compicated (meaning "primitive")...
>> 
> 
> BMC is powered by standby voltage. If AC input to all of your power
> supplies is cut off, there is no standby voltage anymore. Just try to
> unplug all power cables and see if BMC is still accessible.

Of course! What I tried to point out is: With a proper BMC, you DON'T need to
cut off the server power.

Regards,
Ulrich

> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




From arvidjaar at gmail.com  Tue Aug 18 03:56:27 2020
From: arvidjaar at gmail.com (Andrei Borzenkov)
Date: Tue, 18 Aug 2020 10:56:27 +0300
Subject: [ClusterLabs] Antw: Re: Antw: Re: Antw: [EXT] Stonith failing
In-Reply-To: <5F3B84C0020000A10003AAE4@gwsmtp.uni-regensburg.de>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
 <73d6ecf113098a3154a2e7db2e2a59557272024a.camel@redhat.com>
 <5F3B7ECF020000A10003AAD3@gwsmtp.uni-regensburg.de>
 <83aba38d-c9ea-1dff-e53b-14a9e0623d9d@gmail.com>
 <5F3B84C0020000A10003AAE4@gwsmtp.uni-regensburg.de>
Message-ID: <7ac5e8da-2f4f-2327-8a1f-8b007f5d7f9c@gmail.com>

18.08.2020 10:35, Ulrich Windl ?????:
>>>> Andrei Borzenkov <arvidjaar at gmail.com> schrieb am 18.08.2020 um 09:24 in
> Nachricht <83aba38d-c9ea-1dff-e53b-14a9e0623d9d at gmail.com>:
>> 18.08.2020 10:10, Ulrich Windl ?????:
>>>>>> Ken Gaillot <kgaillot at redhat.com> schrieb am 17.08.2020 um 17:19 in
>>> Nachricht
>>> <73d6ecf113098a3154a2e7db2e2a59557272024a.camel at redhat.com>:
>>>> On Fri, 2020?08?14 at 15:09 +0200, Gabriele Bulfon wrote:
>>>>> Thanks to all your suggestions, I now have the systems with stonith
>>>>> configured on ipmi.
>>>>
>>>> A word of caution: if the IPMI is on?board ?? i.e. it shares the same
>>>> power supply as the computer ?? power becomes a single point of
>>>> failure. If the node loses power, the other node can't fence because
>>>> the IPMI is also down, and the cluster can't recover.
>>>
>>> This may not always be true: We had servers with three(!) power supplies
> and 
>> a
>>> BMC (what today is called "light-out management"). You could "power down" 
>> the
>>> server, while the BMC was still operational (and thus could "power up" the
>>> server again).
>>> With standard PC architecture these days things seem to be a bit more
>>> compicated (meaning "primitive")...
>>>
>>
>> BMC is powered by standby voltage. If AC input to all of your power
>> supplies is cut off, there is no standby voltage anymore. Just try to
>> unplug all power cables and see if BMC is still accessible.
> 
> Of course! What I tried to point out is: With a proper BMC, you DON'T need to
> cut off the server power.
> 

You seem to completely misunderstand the problem - if external power is
cut off, it is impossible to stonith node via IPMI because BMC is not
accessible.

From jgdr at dalibo.com  Tue Aug 18 06:07:08 2020
From: jgdr at dalibo.com (Jehan-Guillaume de Rorthais)
Date: Tue, 18 Aug 2020 12:07:08 +0200
Subject: [ClusterLabs] Antw: [EXT] Stonith failing
In-Reply-To: <9fdd714d-ccb7-9cfa-8952-23d0e5336695@redhat.com>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www>
 <1118615335.1895.1597410544148@www>
 <73d6ecf113098a3154a2e7db2e2a59557272024a.camel@redhat.com>
 <20200817223919.6187c711@firost>
 <cd796b78-ca9d-7fac-610c-5fa98fee6cea@gmail.com>
 <9fdd714d-ccb7-9cfa-8952-23d0e5336695@redhat.com>
Message-ID: <20200818120708.5b192b54@firost>

On Tue, 18 Aug 2020 08:21:50 +0200
Klaus Wenninger <kwenning at redhat.com> wrote:

> On 8/18/20 7:49 AM, Andrei Borzenkov wrote:
> > 17.08.2020 23:39, Jehan-Guillaume de Rorthais ?????:  
> >> On Mon, 17 Aug 2020 10:19:45 -0500
> >> Ken Gaillot <kgaillot at redhat.com> wrote:
> >>  
> >>> On Fri, 2020-08-14 at 15:09 +0200, Gabriele Bulfon wrote:  
> >>>> Thanks to all your suggestions, I now have the systems with stonith
> >>>> configured on ipmi.    
> >>> A word of caution: if the IPMI is on-board -- i.e. it shares the same
> >>> power supply as the computer -- power becomes a single point of
> >>> failure. If the node loses power, the other node can't fence because
> >>> the IPMI is also down, and the cluster can't recover.
> >>>
> >>> Some on-board IPMI controllers can share an Ethernet port with the main
> >>> computer, which would be a similar situation.
> >>>
> >>> It's best to have a backup fencing method when using IPMI as the
> >>> primary fencing method. An example would be an intelligent power switch
> >>> or sbd.  
> >> How SBD would be useful in this scenario? Poison pill will not be
> >> swallowed by the dead node... Is it just to wait for the watchdog timeout?
> >>  
> > Node is expected to commit suicide if SBD lost access to shared block
> > device. So either node swallowed poison pill and died or node died
> > because it realized it was impossible to see poison pill or node was
> > dead already. After watchdog timeout (twice watchdog timeout for safety)
> > we assume node is dead.  
> Yes, like this a suicide via watchdog will be triggered if there are
> issues with thedisk. This is why it is important to have a reliable
> watchdog with SBD even whenusing poison pill. As this alone would
> make a single shared disk a SPOF, runningwith pacemaker integration
> (default) a node with SBD will survive despite ofloosing the disk
> when it has quorum and pacemaker looks healthy. As corosync-quorum
> in 2-node-mode obviously won't be fit for this purpose SBD will switch
> to checking for presence of both nodes if 2-node-flag is set.
> 
> Sorry for the lengthy explanation but the full picture is required
> to understand whyit is sufficiently reliable and useful if configured

Thank you Andrei and Klaus for the explanation.

Regards,

From kadlecsik.jozsef at wigner.hu  Tue Aug 18 08:35:27 2020
From: kadlecsik.jozsef at wigner.hu (=?UTF-8?Q?Kadlecsik_J=C3=B3zsef?=)
Date: Tue, 18 Aug 2020 14:35:27 +0200 (CEST)
Subject: [ClusterLabs] node utilization attributes are lost during
 upgrade
In-Reply-To: <08e1ec020aef4f25f2eadc8294135d2f611a2b76.camel@redhat.com>
References: <alpine.DEB.2.23.453.2008171134050.25087@blackhole.kfki.hu>
 <08e1ec020aef4f25f2eadc8294135d2f611a2b76.camel@redhat.com>
Message-ID: <alpine.DEB.2.23.453.2008181409330.25087@blackhole.kfki.hu>

Hi,

On Mon, 17 Aug 2020, Ken Gaillot wrote:

> On Mon, 2020-08-17 at 12:12 +0200, Kadlecsik J?zsef wrote:
> > 
> > At upgrading a corosync/pacemaker/libvirt/KVM cluster from Debian 
> > stretch to buster, all the node utilization attributes were erased 
> > from the configuration. However, the same attributes were kept at the 
> > VirtualDomain resources. This resulted that all resources with 
> > utilization attributes were stopped.
> 
> Ouch :(
> 
> There are two types of node attributes, transient and permanent. 
> Transient attributes last only until pacemaker is next stopped on the 
> node, while permanent attributes persist between reboots/restarts.
> 
> If you configured the utilization attributes with crm_attribute -z/ 
> --utilization, it will default to permanent, but it's possible to 
> override that with -l/--lifetime reboot (or equivalently, -t/--type 
> status).

The attributes were defined by "crm configure edit", simply stating:

node 1084762113: atlas0 \
        utilization hv_memory=192 cpu=32 \
        attributes standby=off
...
node 1084762119: atlas6 \
        utilization hv_memory=192 cpu=32 \

But I believe now that corosync caused the problem, because the nodes had 
been renumbered:

node 3232245761: atlas0
...
node 3232245767: atlas6

The upgrade process was:

for each node do
    set the "hold" mark on the corosync package
    put the node standby
    wait for the resources to be migrated off
    upgrade from stretch to buster
    reboot
    put the node online
    wait for the resources to be migrated (back)
done

Up to this point all resources were running fine.

In order to upgrade corosync, we followed the next steps:

enable maintenance mode
stop pacemaker and corosync on all nodes
for each node do
    delete the hold mark and upgrade corosync
    install new config file (nodeid not specified)
    restart corosync, start pacemaker
done

We could see that all resources were running unmanaged. When disabling the 
maintenance mode, then those were stopped.

So I think corosync renumbered the nodes and I suspect the reason for that 
was that "clear_node_high_bit: yes" was not specified in the new config 
file. It means it was an admin error then.

Best regards,
Jozsef
--
E-mail : kadlecsik.jozsef at wigner.hu
PGP key: https://wigner.hu/~kadlec/pgp_public_key.txt
Address: Wigner Research Centre for Physics
         H-1525 Budapest 114, POB. 49, Hungary

From kgaillot at redhat.com  Tue Aug 18 10:02:33 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Tue, 18 Aug 2020 09:02:33 -0500
Subject: [ClusterLabs] Antw: [EXT] Stonith failing
In-Reply-To: <9fdd714d-ccb7-9cfa-8952-23d0e5336695@redhat.com>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
 <73d6ecf113098a3154a2e7db2e2a59557272024a.camel@redhat.com>
 <20200817223919.6187c711@firost>
 <cd796b78-ca9d-7fac-610c-5fa98fee6cea@gmail.com>
 <9fdd714d-ccb7-9cfa-8952-23d0e5336695@redhat.com>
Message-ID: <b94c4adfdd71e5633eea81ce064f1aa2aa0b9966.camel@redhat.com>

On Tue, 2020-08-18 at 08:21 +0200, Klaus Wenninger wrote:
> On 8/18/20 7:49 AM, Andrei Borzenkov wrote:
> > 17.08.2020 23:39, Jehan-Guillaume de Rorthais ?????:
> > > On Mon, 17 Aug 2020 10:19:45 -0500
> > > Ken Gaillot <kgaillot at redhat.com> wrote:
> > > 
> > > > On Fri, 2020-08-14 at 15:09 +0200, Gabriele Bulfon wrote:
> > > > > Thanks to all your suggestions, I now have the systems with
> > > > > stonith
> > > > > configured on ipmi.  
> > > > 
> > > > A word of caution: if the IPMI is on-board -- i.e. it shares
> > > > the same
> > > > power supply as the computer -- power becomes a single point of
> > > > failure. If the node loses power, the other node can't fence
> > > > because
> > > > the IPMI is also down, and the cluster can't recover.
> > > > 
> > > > Some on-board IPMI controllers can share an Ethernet port with
> > > > the main
> > > > computer, which would be a similar situation.
> > > > 
> > > > It's best to have a backup fencing method when using IPMI as
> > > > the
> > > > primary fencing method. An example would be an intelligent
> > > > power switch
> > > > or sbd.
> > > 
> > > How SBD would be useful in this scenario? Poison pill will not be
> > > swallowed by
> > > the dead node... Is it just to wait for the watchdog timeout?
> > > 
> > 
> > Node is expected to commit suicide if SBD lost access to shared
> > block
> > device. So either node swallowed poison pill and died or node died
> > because it realized it was impossible to see poison pill or node
> > was
> > dead already. After watchdog timeout (twice watchdog timeout for
> > safety)
> > we assume node is dead.
> 
> Yes, like this a suicide via watchdog will be triggered if there are
> issues with thedisk. This is why it is important to have a reliable
> watchdog with SBD even whenusing poison pill. As this alone would
> make a single shared disk a SPOF, runningwith pacemaker integration
> (default) a node with SBD will survive despite ofloosing the disk
> when it has quorum and pacemaker looks healthy. As corosync-quorum
> in 2-node-mode obviously won't be fit for this purpose SBD will
> switch
> to checking for presence of both nodes if 2-node-flag is set.
> 
> Sorry for the lengthy explanation but the full picture is required
> to understand whyit is sufficiently reliable and useful if configured
> correctly.
> 
> Klaus

What I'm not sure about is how watchdog-only sbd would behave as a
fail-back method for a regular fence device. Will the cluster wait for
the sbd timeout no matter what, or only if the regular fencing fails,
or ...?
-- 
Ken Gaillot <kgaillot at redhat.com>


From kgaillot at redhat.com  Tue Aug 18 10:15:49 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Tue, 18 Aug 2020 09:15:49 -0500
Subject: [ClusterLabs] node utilization attributes are lost during
 upgrade
In-Reply-To: <alpine.DEB.2.23.453.2008181409330.25087@blackhole.kfki.hu>
References: <alpine.DEB.2.23.453.2008171134050.25087@blackhole.kfki.hu>
 <08e1ec020aef4f25f2eadc8294135d2f611a2b76.camel@redhat.com>
 <alpine.DEB.2.23.453.2008181409330.25087@blackhole.kfki.hu>
Message-ID: <04b9e334c828e1e1a718643a75e60d6e40bd4eb0.camel@redhat.com>

On Tue, 2020-08-18 at 14:35 +0200, Kadlecsik J?zsef wrote:
> Hi,
> 
> On Mon, 17 Aug 2020, Ken Gaillot wrote:
> 
> > On Mon, 2020-08-17 at 12:12 +0200, Kadlecsik J?zsef wrote:
> > > 
> > > At upgrading a corosync/pacemaker/libvirt/KVM cluster from
> > > Debian 
> > > stretch to buster, all the node utilization attributes were
> > > erased 
> > > from the configuration. However, the same attributes were kept at
> > > the 
> > > VirtualDomain resources. This resulted that all resources with 
> > > utilization attributes were stopped.
> > 
> > Ouch :(
> > 
> > There are two types of node attributes, transient and permanent. 
> > Transient attributes last only until pacemaker is next stopped on
> > the 
> > node, while permanent attributes persist between reboots/restarts.
> > 
> > If you configured the utilization attributes with crm_attribute
> > -z/ 
> > --utilization, it will default to permanent, but it's possible to 
> > override that with -l/--lifetime reboot (or equivalently, -t/
> > --type 
> > status).
> 
> The attributes were defined by "crm configure edit", simply stating:
> 
> node 1084762113: atlas0 \
>         utilization hv_memory=192 cpu=32 \
>         attributes standby=off
> ...
> node 1084762119: atlas6 \
>         utilization hv_memory=192 cpu=32 \
> 
> But I believe now that corosync caused the problem, because the nodes
> had 
> been renumbered:

Ah yes, that would do it. Pacemaker would consider them different nodes
with the same names. The "other" node's attributes would not apply to
the "new" node.

The upgrade procedure would be similar except that you would start
corosync by itself after each upgrade. After all nodes were upgraded,
you would modify the CIB on one node (while pacemaker is not running)
with:

  CIB_file=/var/lib/pacemaker/cib/cib.xml cibadmin --modify --scope=nodes -X '...'

where '...' is a <node> XML entry from the CIB with the "id" value
changed to the new ID, and repeat that for each node. Then, start
pacemaker on that node and wait for it to come up, then start pacemaker
on the other nodes.

> 
> node 3232245761: atlas0
> ...
> node 3232245767: atlas6
> 
> The upgrade process was:
> 
> for each node do
>     set the "hold" mark on the corosync package
>     put the node standby
>     wait for the resources to be migrated off
>     upgrade from stretch to buster
>     reboot
>     put the node online
>     wait for the resources to be migrated (back)
> done
> 
> Up to this point all resources were running fine.
> 
> In order to upgrade corosync, we followed the next steps:
> 
> enable maintenance mode
> stop pacemaker and corosync on all nodes
> for each node do
>     delete the hold mark and upgrade corosync
>     install new config file (nodeid not specified)
>     restart corosync, start pacemaker
> done
> 
> We could see that all resources were running unmanaged. When
> disabling the 
> maintenance mode, then those were stopped.
> 
> So I think corosync renumbered the nodes and I suspect the reason for
> that 
> was that "clear_node_high_bit: yes" was not specified in the new
> config 
> file. It means it was an admin error then.
> 
> Best regards,
> Jozsef
> --
> E-mail : kadlecsik.jozsef at wigner.hu
> PGP key: https://wigner.hu/~kadlec/pgp_public_key.txt
> Address: Wigner Research Centre for Physics
>          H-1525 Budapest 114, POB. 49, Hungary
-- 
Ken Gaillot <kgaillot at redhat.com>


From bernd.lentes at helmholtz-muenchen.de  Tue Aug 18 10:47:54 2020
From: bernd.lentes at helmholtz-muenchen.de (Lentes, Bernd)
Date: Tue, 18 Aug 2020 16:47:54 +0200 (CEST)
Subject: [ClusterLabs] why is node fenced ?
In-Reply-To: <84a98a74897f0b5523f44b23a415fcde5bff0de6.camel@redhat.com>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
 <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>
 <1230152204.50050895.1597430263067.JavaMail.zimbra@helmholtz-muenchen.de>
 <84a98a74897f0b5523f44b23a415fcde5bff0de6.camel@redhat.com>
Message-ID: <453537998.54408252.1597762074665.JavaMail.zimbra@helmholtz-muenchen.de>



----- On Aug 17, 2020, at 5:09 PM, kgaillot kgaillot at redhat.com wrote:


>> I checked all relevant pe-files in this time period.
>> This is what i found out (i just write the important entries):
 
>> Executing cluster transition:
>>  * Resource action: vm_nextcloud    stop on ha-idg-2
>> Revised cluster status:
>>  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped
>> 
>> ha-idg-1:~/why-fenced/ha-idg-1/pengine # crm_simulate -S -x pe-input-
>> 3118 -G transition-4516.xml -D transition-4516.dot
>> Current cluster status:
>> Node ha-idg-1 (1084777482): standby
>> Online: [ ha-idg-2 ]
>>  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped
>> <============== vm_nextcloud is stopped
>> Transition Summary:
>>  * Shutdown ha-idg-1
>> Executing cluster transition:
>>  * Resource action: vm_nextcloud    stop on ha-idg-1 <==== why stop ?
>> It is already stopped
> 
> I'm not sure, I'd have to see the pe input.

You find it here: https://hmgubox2.helmholtz-muenchen.de/index.php/s/WJGtodMZ9k7rN29

>>  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped <=======
>> vm_nextcloud is stopped
>> Transition Summary:
>>  * Fence (Off) ha-idg-1 'resource actions are unrunnable'
>> Executing cluster transition:
>>  * Fencing ha-idg-1 (Off)
>>  * Pseudo action:   vm_nextcloud_stop_0 <======= why stop ? It is
>> already stopped ?
>> Revised cluster status:
>> Node ha-idg-1 (1084777482): OFFLINE (standby)
>> Online: [ ha-idg-2 ]
>>  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped
>> 
>> I don't understand why the cluster tries to stop a resource which is
>> already stopped.

Bernd
Helmholtz Zentrum M?nchen

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin Guenther
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671



From kgaillot at redhat.com  Tue Aug 18 13:30:34 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Tue, 18 Aug 2020 12:30:34 -0500
Subject: [ClusterLabs] why is node fenced ?
In-Reply-To: <453537998.54408252.1597762074665.JavaMail.zimbra@helmholtz-muenchen.de>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
 <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>
 <1230152204.50050895.1597430263067.JavaMail.zimbra@helmholtz-muenchen.de>
 <84a98a74897f0b5523f44b23a415fcde5bff0de6.camel@redhat.com>
 <453537998.54408252.1597762074665.JavaMail.zimbra@helmholtz-muenchen.de>
Message-ID: <104df2890058a7948e93251043ba4fb43ed28446.camel@redhat.com>

On Tue, 2020-08-18 at 16:47 +0200, Lentes, Bernd wrote:
> 
> ----- On Aug 17, 2020, at 5:09 PM, kgaillot kgaillot at redhat.com
> wrote:
> 
> 
> > > I checked all relevant pe-files in this time period.
> > > This is what i found out (i just write the important entries):
> 
>  
> > > Executing cluster transition:
> > >  * Resource action: vm_nextcloud    stop on ha-idg-2
> > > Revised cluster status:
> > >  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped
> > > 
> > > ha-idg-1:~/why-fenced/ha-idg-1/pengine # crm_simulate -S -x pe-
> > > input-
> > > 3118 -G transition-4516.xml -D transition-4516.dot
> > > Current cluster status:
> > > Node ha-idg-1 (1084777482): standby
> > > Online: [ ha-idg-2 ]
> > >  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped
> > > <============== vm_nextcloud is stopped
> > > Transition Summary:
> > >  * Shutdown ha-idg-1
> > > Executing cluster transition:
> > >  * Resource action: vm_nextcloud    stop on ha-idg-1 <==== why
> > > stop ?
> > > It is already stopped
> > 
> > I'm not sure, I'd have to see the pe input.
> 
> You find it here: 
> https://hmgubox2.helmholtz-muenchen.de/index.php/s/WJGtodMZ9k7rN29

This appears to be a scheduler bug.

The scheduler considers a migration to be "dangling" if it has a record
of a failed migrate_to on the source node, but no migrate_from on the
target node (and no migrate_from or start on the source node, which
would indicate a later full restart or reverse migration).

In this case, any migrate_from on the target has since been superseded
by a failed start and a successful stop, so there is no longer a record
of it. Therefore the migration is considered dangling, which requires a
full stop on the source node.

However in this case we already have a successful stop on the source
node after the failed migrate_to, and I believe that should be
sufficient to consider it no longer dangling.

> > >  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped <=======
> > > vm_nextcloud is stopped
> > > Transition Summary:
> > >  * Fence (Off) ha-idg-1 'resource actions are unrunnable'
> > > Executing cluster transition:
> > >  * Fencing ha-idg-1 (Off)
> > >  * Pseudo action:   vm_nextcloud_stop_0 <======= why stop ? It is
> > > already stopped ?
> > > Revised cluster status:
> > > Node ha-idg-1 (1084777482): OFFLINE (standby)
> > > Online: [ ha-idg-2 ]
> > >  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped
> > > 
> > > I don't understand why the cluster tries to stop a resource which
> > > is
> > > already stopped.
> 
> Bernd
> Helmholtz Zentrum M?nchen
> 
> Helmholtz Zentrum Muenchen
> Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
> Ingolstaedter Landstr. 1
> 85764 Neuherberg
> www.helmholtz-muenchen.de
> Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
> Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin
> Guenther
> Registergericht: Amtsgericht Muenchen HRB 6466
> USt-IdNr: DE 129521671
-- 
Ken Gaillot <kgaillot at redhat.com>


From arvidjaar at gmail.com  Tue Aug 18 15:07:44 2020
From: arvidjaar at gmail.com (Andrei Borzenkov)
Date: Tue, 18 Aug 2020 22:07:44 +0300
Subject: [ClusterLabs] Antw: [EXT] Stonith failing
In-Reply-To: <b94c4adfdd71e5633eea81ce064f1aa2aa0b9966.camel@redhat.com>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
 <73d6ecf113098a3154a2e7db2e2a59557272024a.camel@redhat.com>
 <20200817223919.6187c711@firost>
 <cd796b78-ca9d-7fac-610c-5fa98fee6cea@gmail.com>
 <9fdd714d-ccb7-9cfa-8952-23d0e5336695@redhat.com>
 <b94c4adfdd71e5633eea81ce064f1aa2aa0b9966.camel@redhat.com>
Message-ID: <e5d787fc-cc44-2900-93e3-a432091dc07e@gmail.com>

18.08.2020 17:02, Ken Gaillot ?????:
> On Tue, 2020-08-18 at 08:21 +0200, Klaus Wenninger wrote:
>> On 8/18/20 7:49 AM, Andrei Borzenkov wrote:
>>> 17.08.2020 23:39, Jehan-Guillaume de Rorthais ?????:
>>>> On Mon, 17 Aug 2020 10:19:45 -0500
>>>> Ken Gaillot <kgaillot at redhat.com> wrote:
>>>>
>>>>> On Fri, 2020-08-14 at 15:09 +0200, Gabriele Bulfon wrote:
>>>>>> Thanks to all your suggestions, I now have the systems with
>>>>>> stonith
>>>>>> configured on ipmi.  
>>>>>
>>>>> A word of caution: if the IPMI is on-board -- i.e. it shares
>>>>> the same
>>>>> power supply as the computer -- power becomes a single point of
>>>>> failure. If the node loses power, the other node can't fence
>>>>> because
>>>>> the IPMI is also down, and the cluster can't recover.
>>>>>
>>>>> Some on-board IPMI controllers can share an Ethernet port with
>>>>> the main
>>>>> computer, which would be a similar situation.
>>>>>
>>>>> It's best to have a backup fencing method when using IPMI as
>>>>> the
>>>>> primary fencing method. An example would be an intelligent
>>>>> power switch
>>>>> or sbd.
>>>>
>>>> How SBD would be useful in this scenario? Poison pill will not be
>>>> swallowed by
>>>> the dead node... Is it just to wait for the watchdog timeout?
>>>>
>>>
>>> Node is expected to commit suicide if SBD lost access to shared
>>> block
>>> device. So either node swallowed poison pill and died or node died
>>> because it realized it was impossible to see poison pill or node
>>> was
>>> dead already. After watchdog timeout (twice watchdog timeout for
>>> safety)
>>> we assume node is dead.
>>
>> Yes, like this a suicide via watchdog will be triggered if there are
>> issues with thedisk. This is why it is important to have a reliable
>> watchdog with SBD even whenusing poison pill. As this alone would
>> make a single shared disk a SPOF, runningwith pacemaker integration
>> (default) a node with SBD will survive despite ofloosing the disk
>> when it has quorum and pacemaker looks healthy. As corosync-quorum
>> in 2-node-mode obviously won't be fit for this purpose SBD will
>> switch
>> to checking for presence of both nodes if 2-node-flag is set.
>>
>> Sorry for the lengthy explanation but the full picture is required
>> to understand whyit is sufficiently reliable and useful if configured
>> correctly.
>>
>> Klaus
> 
> What I'm not sure about is how watchdog-only sbd would behave as a
> fail-back method for a regular fence device. Will the cluster wait for
> the sbd timeout no matter what, or only if the regular fencing fails,
> or ...?
> 

Diskless SBD implicitly creates fencing device ("watchdog"), timeout
starts only when this device is selected for fencing. This device
appears to be completely invisible to normal stonith_admin operation, I
do not know how to query for it. In my testing explicit stonith resource
was always called first and only if it failed was "watchdog" self
fencing attempted. I tried to set negative priority for CIB stonith
resource but it did not change anything.

From hunter86_bg at yahoo.com  Tue Aug 18 15:45:29 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Tue, 18 Aug 2020 22:45:29 +0300
Subject: [ClusterLabs] node utilization attributes are lost during
 upgrade
In-Reply-To: <04b9e334c828e1e1a718643a75e60d6e40bd4eb0.camel@redhat.com>
References: <alpine.DEB.2.23.453.2008171134050.25087@blackhole.kfki.hu>
 <08e1ec020aef4f25f2eadc8294135d2f611a2b76.camel@redhat.com>
 <alpine.DEB.2.23.453.2008181409330.25087@blackhole.kfki.hu>
 <04b9e334c828e1e1a718643a75e60d6e40bd4eb0.camel@redhat.com>
Message-ID: <BA72CAA0-12E7-4E8E-8FF5-57B0745EB66D@yahoo.com>

Won't it be easier if:
- set a node in standby
- stop a node
- remove the node
- add again with the new hostname

Best Regards,
Strahil Nikolov

?? 18 ?????? 2020 ?. 17:15:49 GMT+03:00, Ken Gaillot <kgaillot at redhat.com> ??????:
>On Tue, 2020-08-18 at 14:35 +0200, Kadlecsik J?zsef wrote:
>> Hi,
>> 
>> On Mon, 17 Aug 2020, Ken Gaillot wrote:
>> 
>> > On Mon, 2020-08-17 at 12:12 +0200, Kadlecsik J?zsef wrote:
>> > > 
>> > > At upgrading a corosync/pacemaker/libvirt/KVM cluster from
>> > > Debian 
>> > > stretch to buster, all the node utilization attributes were
>> > > erased 
>> > > from the configuration. However, the same attributes were kept at
>> > > the 
>> > > VirtualDomain resources. This resulted that all resources with 
>> > > utilization attributes were stopped.
>> > 
>> > Ouch :(
>> > 
>> > There are two types of node attributes, transient and permanent. 
>> > Transient attributes last only until pacemaker is next stopped on
>> > the 
>> > node, while permanent attributes persist between reboots/restarts.
>> > 
>> > If you configured the utilization attributes with crm_attribute
>> > -z/ 
>> > --utilization, it will default to permanent, but it's possible to 
>> > override that with -l/--lifetime reboot (or equivalently, -t/
>> > --type 
>> > status).
>> 
>> The attributes were defined by "crm configure edit", simply stating:
>> 
>> node 1084762113: atlas0 \
>>         utilization hv_memory=192 cpu=32 \
>>         attributes standby=off
>> ...
>> node 1084762119: atlas6 \
>>         utilization hv_memory=192 cpu=32 \
>> 
>> But I believe now that corosync caused the problem, because the nodes
>> had 
>> been renumbered:
>
>Ah yes, that would do it. Pacemaker would consider them different nodes
>with the same names. The "other" node's attributes would not apply to
>the "new" node.
>
>The upgrade procedure would be similar except that you would start
>corosync by itself after each upgrade. After all nodes were upgraded,
>you would modify the CIB on one node (while pacemaker is not running)
>with:
>
>CIB_file=/var/lib/pacemaker/cib/cib.xml cibadmin --modify --scope=nodes
>-X '...'
>
>where '...' is a <node> XML entry from the CIB with the "id" value
>changed to the new ID, and repeat that for each node. Then, start
>pacemaker on that node and wait for it to come up, then start pacemaker
>on the other nodes.
>
>> 
>> node 3232245761: atlas0
>> ...
>> node 3232245767: atlas6
>> 
>> The upgrade process was:
>> 
>> for each node do
>>     set the "hold" mark on the corosync package
>>     put the node standby
>>     wait for the resources to be migrated off
>>     upgrade from stretch to buster
>>     reboot
>>     put the node online
>>     wait for the resources to be migrated (back)
>> done
>> 
>> Up to this point all resources were running fine.
>> 
>> In order to upgrade corosync, we followed the next steps:
>> 
>> enable maintenance mode
>> stop pacemaker and corosync on all nodes
>> for each node do
>>     delete the hold mark and upgrade corosync
>>     install new config file (nodeid not specified)
>>     restart corosync, start pacemaker
>> done
>> 
>> We could see that all resources were running unmanaged. When
>> disabling the 
>> maintenance mode, then those were stopped.
>> 
>> So I think corosync renumbered the nodes and I suspect the reason for
>> that 
>> was that "clear_node_high_bit: yes" was not specified in the new
>> config 
>> file. It means it was an admin error then.
>> 
>> Best regards,
>> Jozsef
>> --
>> E-mail : kadlecsik.jozsef at wigner.hu
>> PGP key: https://wigner.hu/~kadlec/pgp_public_key.txt
>> Address: Wigner Research Centre for Physics
>>          H-1525 Budapest 114, POB. 49, Hungary
>-- 
>Ken Gaillot <kgaillot at redhat.com>
>
>_______________________________________________
>Manage your subscription:
>https://lists.clusterlabs.org/mailman/listinfo/users
>
>ClusterLabs home: https://www.clusterlabs.org/

From kwenning at redhat.com  Tue Aug 18 15:49:13 2020
From: kwenning at redhat.com (Klaus Wenninger)
Date: Tue, 18 Aug 2020 21:49:13 +0200
Subject: [ClusterLabs] Antw: [EXT] Stonith failing
In-Reply-To: <e5d787fc-cc44-2900-93e3-a432091dc07e@gmail.com>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
 <73d6ecf113098a3154a2e7db2e2a59557272024a.camel@redhat.com>
 <20200817223919.6187c711@firost>
 <cd796b78-ca9d-7fac-610c-5fa98fee6cea@gmail.com>
 <9fdd714d-ccb7-9cfa-8952-23d0e5336695@redhat.com>
 <b94c4adfdd71e5633eea81ce064f1aa2aa0b9966.camel@redhat.com>
 <e5d787fc-cc44-2900-93e3-a432091dc07e@gmail.com>
Message-ID: <a350fa7b-3fa1-903f-1121-72f7562c9316@redhat.com>

On 8/18/20 9:07 PM, Andrei Borzenkov wrote:
> 18.08.2020 17:02, Ken Gaillot ?????:
>> On Tue, 2020-08-18 at 08:21 +0200, Klaus Wenninger wrote:
>>> On 8/18/20 7:49 AM, Andrei Borzenkov wrote:
>>>> 17.08.2020 23:39, Jehan-Guillaume de Rorthais ?????:
>>>>> On Mon, 17 Aug 2020 10:19:45 -0500
>>>>> Ken Gaillot <kgaillot at redhat.com> wrote:
>>>>>
>>>>>> On Fri, 2020-08-14 at 15:09 +0200, Gabriele Bulfon wrote:
>>>>>>> Thanks to all your suggestions, I now have the systems with
>>>>>>> stonith
>>>>>>> configured on ipmi.  
>>>>>> A word of caution: if the IPMI is on-board -- i.e. it shares
>>>>>> the same
>>>>>> power supply as the computer -- power becomes a single point of
>>>>>> failure. If the node loses power, the other node can't fence
>>>>>> because
>>>>>> the IPMI is also down, and the cluster can't recover.
>>>>>>
>>>>>> Some on-board IPMI controllers can share an Ethernet port with
>>>>>> the main
>>>>>> computer, which would be a similar situation.
>>>>>>
>>>>>> It's best to have a backup fencing method when using IPMI as
>>>>>> the
>>>>>> primary fencing method. An example would be an intelligent
>>>>>> power switch
>>>>>> or sbd.
>>>>> How SBD would be useful in this scenario? Poison pill will not be
>>>>> swallowed by
>>>>> the dead node... Is it just to wait for the watchdog timeout?
>>>>>
>>>> Node is expected to commit suicide if SBD lost access to shared
>>>> block
>>>> device. So either node swallowed poison pill and died or node died
>>>> because it realized it was impossible to see poison pill or node
>>>> was
>>>> dead already. After watchdog timeout (twice watchdog timeout for
>>>> safety)
>>>> we assume node is dead.
>>> Yes, like this a suicide via watchdog will be triggered if there are
>>> issues with thedisk. This is why it is important to have a reliable
>>> watchdog with SBD even whenusing poison pill. As this alone would
>>> make a single shared disk a SPOF, runningwith pacemaker integration
>>> (default) a node with SBD will survive despite ofloosing the disk
>>> when it has quorum and pacemaker looks healthy. As corosync-quorum
>>> in 2-node-mode obviously won't be fit for this purpose SBD will
>>> switch
>>> to checking for presence of both nodes if 2-node-flag is set.
>>>
>>> Sorry for the lengthy explanation but the full picture is required
>>> to understand whyit is sufficiently reliable and useful if configured
>>> correctly.
>>>
>>> Klaus
>> What I'm not sure about is how watchdog-only sbd would behave as a
>> fail-back method for a regular fence device. Will the cluster wait for
>> the sbd timeout no matter what, or only if the regular fencing fails,
>> or ...?
>>
> Diskless SBD implicitly creates fencing device ("watchdog"), timeout
> starts only when this device is selected for fencing. This device
> appears to be completely invisible to normal stonith_admin operation, I
> do not know how to query for it. In my testing explicit stonith resource
> was always called first and only if it failed was "watchdog" self
> fencing attempted. I tried to set negative priority for CIB stonith
> resource but it did not change anything.
>
This matches with what I remember from going through the code ...
like with lowest prio but not at all if there is a topology defined ...
which probably should be overhauled ...
If interested there is a branch about having just certain nodes
watchdog-fenced on my pacemaker-clone that makes the watchdog
device visible.


Klaus


From kgaillot at redhat.com  Tue Aug 18 16:02:07 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Tue, 18 Aug 2020 15:02:07 -0500
Subject: [ClusterLabs] node utilization attributes are lost during
 upgrade
In-Reply-To: <BA72CAA0-12E7-4E8E-8FF5-57B0745EB66D@yahoo.com>
References: <alpine.DEB.2.23.453.2008171134050.25087@blackhole.kfki.hu>
 <08e1ec020aef4f25f2eadc8294135d2f611a2b76.camel@redhat.com>
 <alpine.DEB.2.23.453.2008181409330.25087@blackhole.kfki.hu>
 <04b9e334c828e1e1a718643a75e60d6e40bd4eb0.camel@redhat.com>
 <BA72CAA0-12E7-4E8E-8FF5-57B0745EB66D@yahoo.com>
Message-ID: <ad1c9495402f6818a462c7ee6661319577422732.camel@redhat.com>

On Tue, 2020-08-18 at 22:45 +0300, Strahil Nikolov wrote:
> Won't it be easier if:
> - set a node in standby
> - stop a node
> - remove the node
> - add again with the new hostname

The hostname stays the same, but corosync is changing the numeric node
ID as part of the upgrade. If they remove the node, they'll lose its
utilization attributes, which is what they want to keep.

Looking at it again, I'm guessing there are no explicit node IDs in
corosync.conf, and corosync is choosing the IDs. In that case the
easiest approach would be to explicitly set the original node IDs in
corosync.conf before the upgrade, so they don't change.

> 
> Best Regards,
> Strahil Nikolov
> 
> ?? 18 ?????? 2020 ?. 17:15:49 GMT+03:00, Ken Gaillot <
> kgaillot at redhat.com> ??????:
> > On Tue, 2020-08-18 at 14:35 +0200, Kadlecsik J?zsef wrote:
> > > Hi,
> > > 
> > > On Mon, 17 Aug 2020, Ken Gaillot wrote:
> > > 
> > > > On Mon, 2020-08-17 at 12:12 +0200, Kadlecsik J?zsef wrote:
> > > > > 
> > > > > At upgrading a corosync/pacemaker/libvirt/KVM cluster from
> > > > > Debian 
> > > > > stretch to buster, all the node utilization attributes were
> > > > > erased 
> > > > > from the configuration. However, the same attributes were
> > > > > kept at
> > > > > the 
> > > > > VirtualDomain resources. This resulted that all resources
> > > > > with 
> > > > > utilization attributes were stopped.
> > > > 
> > > > Ouch :(
> > > > 
> > > > There are two types of node attributes, transient and
> > > > permanent. 
> > > > Transient attributes last only until pacemaker is next stopped
> > > > on
> > > > the 
> > > > node, while permanent attributes persist between
> > > > reboots/restarts.
> > > > 
> > > > If you configured the utilization attributes with crm_attribute
> > > > -z/ 
> > > > --utilization, it will default to permanent, but it's possible
> > > > to 
> > > > override that with -l/--lifetime reboot (or equivalently, -t/
> > > > --type 
> > > > status).
> > > 
> > > The attributes were defined by "crm configure edit", simply
> > > stating:
> > > 
> > > node 1084762113: atlas0 \
> > >         utilization hv_memory=192 cpu=32 \
> > >         attributes standby=off
> > > ...
> > > node 1084762119: atlas6 \
> > >         utilization hv_memory=192 cpu=32 \
> > > 
> > > But I believe now that corosync caused the problem, because the
> > > nodes
> > > had 
> > > been renumbered:
> > 
> > Ah yes, that would do it. Pacemaker would consider them different
> > nodes
> > with the same names. The "other" node's attributes would not apply
> > to
> > the "new" node.
> > 
> > The upgrade procedure would be similar except that you would start
> > corosync by itself after each upgrade. After all nodes were
> > upgraded,
> > you would modify the CIB on one node (while pacemaker is not
> > running)
> > with:
> > 
> > CIB_file=/var/lib/pacemaker/cib/cib.xml cibadmin --modify --
> > scope=nodes
> > -X '...'
> > 
> > where '...' is a <node> XML entry from the CIB with the "id" value
> > changed to the new ID, and repeat that for each node. Then, start
> > pacemaker on that node and wait for it to come up, then start
> > pacemaker
> > on the other nodes.
> > 
> > > 
> > > node 3232245761: atlas0
> > > ...
> > > node 3232245767: atlas6
> > > 
> > > The upgrade process was:
> > > 
> > > for each node do
> > >     set the "hold" mark on the corosync package
> > >     put the node standby
> > >     wait for the resources to be migrated off
> > >     upgrade from stretch to buster
> > >     reboot
> > >     put the node online
> > >     wait for the resources to be migrated (back)
> > > done
> > > 
> > > Up to this point all resources were running fine.
> > > 
> > > In order to upgrade corosync, we followed the next steps:
> > > 
> > > enable maintenance mode
> > > stop pacemaker and corosync on all nodes
> > > for each node do
> > >     delete the hold mark and upgrade corosync
> > >     install new config file (nodeid not specified)
> > >     restart corosync, start pacemaker
> > > done
> > > 
> > > We could see that all resources were running unmanaged. When
> > > disabling the 
> > > maintenance mode, then those were stopped.
> > > 
> > > So I think corosync renumbered the nodes and I suspect the reason
> > > for
> > > that 
> > > was that "clear_node_high_bit: yes" was not specified in the new
> > > config 
> > > file. It means it was an admin error then.
> > > 
> > > Best regards,
> > > Jozsef
> > > --
> > > E-mail : kadlecsik.jozsef at wigner.hu
> > > PGP key: https://wigner.hu/~kadlec/pgp_public_key.txt
> > > Address: Wigner Research Centre for Physics
> > >          H-1525 Budapest 114, POB. 49, Hungary
> > 
> > -- 
> > Ken Gaillot <kgaillot at redhat.com>
> > 
> > _______________________________________________
> > Manage your subscription:
> > https://lists.clusterlabs.org/mailman/listinfo/users
> > 
> > ClusterLabs home: https://www.clusterlabs.org/
> 
> 
-- 
Ken Gaillot <kgaillot at redhat.com>


From lists at alteeve.ca  Wed Aug 19 01:10:08 2020
From: lists at alteeve.ca (Digimer)
Date: Wed, 19 Aug 2020 01:10:08 -0400
Subject: [ClusterLabs] Beginner Question about VirtualDomain
In-Reply-To: <CANNzTaALh5XDn6HMmEwA3KMoef2Kc0if9MdSeUTXtCEbMW7VhA@mail.gmail.com>
References: <CANNzTaALh5XDn6HMmEwA3KMoef2Kc0if9MdSeUTXtCEbMW7VhA@mail.gmail.com>
Message-ID: <a082d515-3de5-a742-0cdc-537718a242ed@alteeve.ca>

On 2020-08-17 8:40 a.m., Sameer Dhiman wrote:
> Hi,
> 
> I am a beginner using pacemaker and corosync. I am trying to set up
> a?cluster of HA KVM guests as described by Alteeve wiki (CentOS-6) but
> in CentOS-8.2. My R&D? setup is described below
> 
> Physical Host running CentOS-8.2 with Nested Virtualization
> 2 x CentOS-8.2 guest machines as Cluster Node 1 and 2.
> WinXP as a HA guest.
> 
> 1. drbd --> dlm --> lvmlockd --> LVM-activate --> gfs2 (guest machine
> definitions)
> 2. drbd --> dlm --> lvmlockd --> LVM-activate --> raw-lv (guest machine HDD)
> 
> Question(s):
> 1. How to prevent guest startup until gfs2 and raw-lv are available? In
> CentOS-6 Alteeve used autostart=0 in the <vm > tag. Is there any similar
> option in pacemaker because I did not find it in the documentation?
> 
> 2. Suppose, If I configure constraint order gfs2 and raw-lv then guest
> machine. Stopping the guest machine would also stop the complete service
> tree so how can I prevent this?
> 
> -- 
> Sameer Dhiman

Hi Sameer,

  I'm the author of that wiki. It's quite out of date, as you noted, and
we're actively developing a new release for EL8. Though, it would be
ready until near the end if the year.

  There are a few changes we've made that you might want to consider;

1. We never were too happy with DLM, and so we've reworked things to no
longer need it. So we use normal LVM backing DRBD resources. One
resource per VM, on volume per virtual disk backed by an LV. Our tools
will automate this, but you can easily enough manually create them if
your environment is fairly stable.

2. To get around GFS2, we create a
/mnt/shared/{provision,definitions,files,archive} directory (note
/shared -> /mnt/shared to be more LFS friendly). We'll again automate
management of files in Striker, but you can copy the files manually and
rsync out changes as needed (again, if your environment doesn't change
much).

3. We changed DRBD from v8.4 to 9.0, and this meant a few things had to
change. We will integrate support for short-throw DR hosts (async "third
node" in DRBD that is outside pacemaker). We run the resources to only
allow a single primary normally and enable auto-promote. For
live-migration, we temporarily enable live migration, promote the
target, migrate, demote the old host and disable dual-primary. This
makes it safer as it's far less likely that someone could accidentally
start a VM on the passive node (not that it ever happened as our tools
prevented it, but it was _possible_, so we wanted to improve that).

That handle #3, we've written our own custom RA (ocf:alteeve:server
[1]). This RA is smart enough to watch/wait for things to become ready
before starting. It also handles the DRBD stuff I mentioned, and the
virsh call to do the migration. So it means the pacemaker config is
extremely simple. Note though it depends on the rest of our tools so it
won't work outside the Anvil!. That said, if you wanted to use it before
we release Anvil! M3, you could probably adapt it easily enough.

If you have any questions, please let me know and I'll help as best I can.

Cheers,

digimer

(Note: during development, this code base is kept outside of
Clusterlabs. We'll move it in when it reaches beta).
1. https://github.com/digimer/anvil/blob/master/ocf/alteeve/server

-- 
Digimer
Papers and Projects: https://alteeve.com/w/
"I am, somehow, less interested in the weight and convolutions of
Einstein?s brain than in the near certainty that people of equal talent
have lived and died in cotton fields and sweatshops." - Stephen Jay Gould

From arvidjaar at gmail.com  Wed Aug 19 02:44:02 2020
From: arvidjaar at gmail.com (Andrei Borzenkov)
Date: Wed, 19 Aug 2020 09:44:02 +0300
Subject: [ClusterLabs] Antw: [EXT] Stonith failing
In-Reply-To: <a350fa7b-3fa1-903f-1121-72f7562c9316@redhat.com>
References: <1330096936.11468.1595926619455@www>
 <5F1FF427020000A10003A47F@gwsmtp.uni-regensburg.de>
 <CAPiuu9-NHfnn9UbW5xak+PP0yDjgV0Wa-geU60cQVUm-tjX+_A@mail.gmail.com>
 <1599253492.11645.1595932012959@www>
 <0557F18C-6718-43B6-9CA0-8FE7AE2A8785@yahoo.com>
 <603366395.379.1596002482554@www>
 <CAA91j0V=8RZE48SfvJaPJMfu1M2wvzcqgUnNevsv_n=yOEPdfg@mail.gmail.com>
 <68622439.459.1596025362188@www> <1118615335.1895.1597410544148@www>
 <73d6ecf113098a3154a2e7db2e2a59557272024a.camel@redhat.com>
 <20200817223919.6187c711@firost>
 <cd796b78-ca9d-7fac-610c-5fa98fee6cea@gmail.com>
 <9fdd714d-ccb7-9cfa-8952-23d0e5336695@redhat.com>
 <b94c4adfdd71e5633eea81ce064f1aa2aa0b9966.camel@redhat.com>
 <e5d787fc-cc44-2900-93e3-a432091dc07e@gmail.com>
 <a350fa7b-3fa1-903f-1121-72f7562c9316@redhat.com>
Message-ID: <e7072867-c694-f248-da20-03535f41eb93@gmail.com>

18.08.2020 22:49, Klaus Wenninger ?????:
>>> What I'm not sure about is how watchdog-only sbd would behave as a
>>> fail-back method for a regular fence device. Will the cluster wait for
>>> the sbd timeout no matter what, or only if the regular fencing fails,
>>> or ...?
>>>
>> Diskless SBD implicitly creates fencing device ("watchdog"), timeout
>> starts only when this device is selected for fencing. This device
>> appears to be completely invisible to normal stonith_admin operation, I
>> do not know how to query for it. In my testing explicit stonith resource
>> was always called first and only if it failed was "watchdog" self
>> fencing attempted. I tried to set negative priority for CIB stonith
>> resource but it did not change anything.
>>
> This matches with what I remember from going through the code ...
> like with lowest prio but not at all if there is a topology defined ...
> which probably should be overhauled ...
> If interested there is a branch about having just certain nodes
> watchdog-fenced on my pacemaker-clone that makes the watchdog
> device visible.
> 

Actually I was wrong. fenced indeed looks like it creates "watchdog"
device, but it happens too early - at this time fenced still does not
have value of watchdog timeout so it always skips this device. That
explains why device is "invisible" - it does not exist :)

Otherwise what I said applies - watchdog fencing is used as fallback if
all other devices failed and timeout starts when pacemaker decided to
use watchdog fencing.

From kgaillot at redhat.com  Wed Aug 19 10:04:11 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Wed, 19 Aug 2020 09:04:11 -0500
Subject: [ClusterLabs] why is node fenced ?
In-Reply-To: <104df2890058a7948e93251043ba4fb43ed28446.camel@redhat.com>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
 <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>
 <1230152204.50050895.1597430263067.JavaMail.zimbra@helmholtz-muenchen.de>
 <84a98a74897f0b5523f44b23a415fcde5bff0de6.camel@redhat.com>
 <453537998.54408252.1597762074665.JavaMail.zimbra@helmholtz-muenchen.de>
 <104df2890058a7948e93251043ba4fb43ed28446.camel@redhat.com>
Message-ID: <61ea6f4310eac7382e19ba655dcb09cac7348b54.camel@redhat.com>

On Tue, 2020-08-18 at 12:30 -0500, Ken Gaillot wrote:
> On Tue, 2020-08-18 at 16:47 +0200, Lentes, Bernd wrote:
> > 
> > ----- On Aug 17, 2020, at 5:09 PM, kgaillot kgaillot at redhat.com
> > wrote:
> > 
> > 
> > > > I checked all relevant pe-files in this time period.
> > > > This is what i found out (i just write the important entries):
> > 
> >  
> > > > Executing cluster transition:
> > > >  * Resource action: vm_nextcloud    stop on ha-idg-2
> > > > Revised cluster status:
> > > >  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped
> > > > 
> > > > ha-idg-1:~/why-fenced/ha-idg-1/pengine # crm_simulate -S -x pe-
> > > > input-
> > > > 3118 -G transition-4516.xml -D transition-4516.dot
> > > > Current cluster status:
> > > > Node ha-idg-1 (1084777482): standby
> > > > Online: [ ha-idg-2 ]
> > > >  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped
> > > > <============== vm_nextcloud is stopped
> > > > Transition Summary:
> > > >  * Shutdown ha-idg-1
> > > > Executing cluster transition:
> > > >  * Resource action: vm_nextcloud    stop on ha-idg-1 <==== why
> > > > stop ?
> > > > It is already stopped
> > > 
> > > I'm not sure, I'd have to see the pe input.
> > 
> > You find it here: 
> > https://hmgubox2.helmholtz-muenchen.de/index.php/s/WJGtodMZ9k7rN29
> 
> This appears to be a scheduler bug.

Fix is in master branch and will land in 2.0.5 expected at end of the
year

https://github.com/ClusterLabs/pacemaker/pull/2146

> The scheduler considers a migration to be "dangling" if it has a
> record
> of a failed migrate_to on the source node, but no migrate_from on the
> target node (and no migrate_from or start on the source node, which
> would indicate a later full restart or reverse migration).
> 
> In this case, any migrate_from on the target has since been
> superseded
> by a failed start and a successful stop, so there is no longer a
> record
> of it. Therefore the migration is considered dangling, which requires
> a
> full stop on the source node.
> 
> However in this case we already have a successful stop on the source
> node after the failed migrate_to, and I believe that should be
> sufficient to consider it no longer dangling.
> 
> > > >  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped
> > > > <=======
> > > > vm_nextcloud is stopped
> > > > Transition Summary:
> > > >  * Fence (Off) ha-idg-1 'resource actions are unrunnable'
> > > > Executing cluster transition:
> > > >  * Fencing ha-idg-1 (Off)
> > > >  * Pseudo action:   vm_nextcloud_stop_0 <======= why stop ? It
> > > > is
> > > > already stopped ?
> > > > Revised cluster status:
> > > > Node ha-idg-1 (1084777482): OFFLINE (standby)
> > > > Online: [ ha-idg-2 ]
> > > >  vm_nextcloud   (ocf::heartbeat:VirtualDomain): Stopped
> > > > 
> > > > I don't understand why the cluster tries to stop a resource
> > > > which
> > > > is
> > > > already stopped.
> > 
> > Bernd
> > Helmholtz Zentrum M?nchen
> > 
> > Helmholtz Zentrum Muenchen
> > Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
> > Ingolstaedter Landstr. 1
> > 85764 Neuherberg
> > www.helmholtz-muenchen.de
> > Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
> > Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep,
> > Kerstin
> > Guenther
> > Registergericht: Amtsgericht Muenchen HRB 6466
> > USt-IdNr: DE 129521671
-- 
Ken Gaillot <kgaillot at redhat.com>


From bernd.lentes at helmholtz-muenchen.de  Wed Aug 19 10:22:23 2020
From: bernd.lentes at helmholtz-muenchen.de (Lentes, Bernd)
Date: Wed, 19 Aug 2020 16:22:23 +0200 (CEST)
Subject: [ClusterLabs] why is node fenced ?
In-Reply-To: <104df2890058a7948e93251043ba4fb43ed28446.camel@redhat.com>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
 <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>
 <1230152204.50050895.1597430263067.JavaMail.zimbra@helmholtz-muenchen.de>
 <84a98a74897f0b5523f44b23a415fcde5bff0de6.camel@redhat.com>
 <453537998.54408252.1597762074665.JavaMail.zimbra@helmholtz-muenchen.de>
 <104df2890058a7948e93251043ba4fb43ed28446.camel@redhat.com>
Message-ID: <268993835.55008340.1597846943328.JavaMail.zimbra@helmholtz-muenchen.de>


----- On Aug 18, 2020, at 7:30 PM, kgaillot kgaillot at redhat.com wrote:


>> > I'm not sure, I'd have to see the pe input.
>> 
>> You find it here:
>> https://hmgubox2.helmholtz-muenchen.de/index.php/s/WJGtodMZ9k7rN29
> 
> This appears to be a scheduler bug.
> 
> The scheduler considers a migration to be "dangling" if it has a record
> of a failed migrate_to on the source node, but no migrate_from on the
> target node (and no migrate_from or start on the source node, which
> would indicate a later full restart or reverse migration).
> 
> In this case, any migrate_from on the target has since been superseded
> by a failed start and a successful stop, so there is no longer a record
> of it. Therefore the migration is considered dangling, which requires a
> full stop on the source node.
> 
> However in this case we already have a successful stop on the source
> node after the failed migrate_to, and I believe that should be
> sufficient to consider it no longer dangling.
> 

Thanks for your explananation Ken.
For me a Fence i don't understand is the worst that can happen to a HA cluster.

Bernd
Helmholtz Zentrum M?nchen

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin Guenther
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671



From bernd.lentes at helmholtz-muenchen.de  Wed Aug 19 10:29:32 2020
From: bernd.lentes at helmholtz-muenchen.de (Lentes, Bernd)
Date: Wed, 19 Aug 2020 16:29:32 +0200 (CEST)
Subject: [ClusterLabs] why is node fenced ?
In-Reply-To: <61ea6f4310eac7382e19ba655dcb09cac7348b54.camel@redhat.com>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
 <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>
 <1230152204.50050895.1597430263067.JavaMail.zimbra@helmholtz-muenchen.de>
 <84a98a74897f0b5523f44b23a415fcde5bff0de6.camel@redhat.com>
 <453537998.54408252.1597762074665.JavaMail.zimbra@helmholtz-muenchen.de>
 <104df2890058a7948e93251043ba4fb43ed28446.camel@redhat.com>
 <61ea6f4310eac7382e19ba655dcb09cac7348b54.camel@redhat.com>
Message-ID: <2116742364.55012219.1597847372185.JavaMail.zimbra@helmholtz-muenchen.de>


----- On Aug 19, 2020, at 4:04 PM, kgaillot kgaillot at redhat.com wrote:
>> This appears to be a scheduler bug.
> 
> Fix is in master branch and will land in 2.0.5 expected at end of the
> year
> 
> https://github.com/ClusterLabs/pacemaker/pull/2146

A principal question:
I have SLES 12 and i'm using the pacemaker version provided with the distribution.
If this fix is backported depends on Suse.

If i install und update pacemaker manually (not the version provided by Suse),
i loose my support from them, but have always the most recent code and fixes.

If i stay with the version from Suse i have support from them, but maybe not all fixes and not the most recent code.

What is your approach ?
Recommendations ?

Thanks.

Bernd
Helmholtz Zentrum M?nchen

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin Guenther
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671



From hunter86_bg at yahoo.com  Wed Aug 19 13:41:01 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Wed, 19 Aug 2020 20:41:01 +0300
Subject: [ClusterLabs] why is node fenced ?
In-Reply-To: <2116742364.55012219.1597847372185.JavaMail.zimbra@helmholtz-muenchen.de>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
 <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>
 <1230152204.50050895.1597430263067.JavaMail.zimbra@helmholtz-muenchen.de>
 <84a98a74897f0b5523f44b23a415fcde5bff0de6.camel@redhat.com>
 <453537998.54408252.1597762074665.JavaMail.zimbra@helmholtz-muenchen.de>
 <104df2890058a7948e93251043ba4fb43ed28446.camel@redhat.com>
 <61ea6f4310eac7382e19ba655dcb09cac7348b54.camel@redhat.com>
 <2116742364.55012219.1597847372185.JavaMail.zimbra@helmholtz-muenchen.de>
Message-ID: <488C2994-6696-4B26-B414-D581747B8BA0@yahoo.com>

Hi Bernd,

As SLES 12 is in a such a support phase, I guess SUSE will provide fixes only for SLES 15.
It will be best if you open them a case and ask them about that.

Best Regards,
Strahil Nikolov

?? 19 ?????? 2020 ?. 17:29:32 GMT+03:00, "Lentes, Bernd" <bernd.lentes at helmholtz-muenchen.de> ??????:
>
>----- On Aug 19, 2020, at 4:04 PM, kgaillot kgaillot at redhat.com wrote:
>>> This appears to be a scheduler bug.
>> 
>> Fix is in master branch and will land in 2.0.5 expected at end of the
>> year
>> 
>> https://github.com/ClusterLabs/pacemaker/pull/2146
>
>A principal question:
>I have SLES 12 and i'm using the pacemaker version provided with the
>distribution.
>If this fix is backported depends on Suse.
>
>If i install und update pacemaker manually (not the version provided by
>Suse),
>i loose my support from them, but have always the most recent code and
>fixes.
>
>If i stay with the version from Suse i have support from them, but
>maybe not all fixes and not the most recent code.
>
>What is your approach ?
>Recommendations ?
>
>Thanks.
>
>Bernd
>Helmholtz Zentrum M?nchen
>
>Helmholtz Zentrum Muenchen
>Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
>Ingolstaedter Landstr. 1
>85764 Neuherberg
>www.helmholtz-muenchen.de
>Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
>Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin
>Guenther
>Registergericht: Amtsgericht Muenchen HRB 6466
>USt-IdNr: DE 129521671
>
>
>_______________________________________________
>Manage your subscription:
>https://lists.clusterlabs.org/mailman/listinfo/users
>
>ClusterLabs home: https://www.clusterlabs.org/

From kgaillot at redhat.com  Wed Aug 19 13:46:34 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Wed, 19 Aug 2020 12:46:34 -0500
Subject: [ClusterLabs] why is node fenced ?
In-Reply-To: <2116742364.55012219.1597847372185.JavaMail.zimbra@helmholtz-muenchen.de>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
 <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>
 <1230152204.50050895.1597430263067.JavaMail.zimbra@helmholtz-muenchen.de>
 <84a98a74897f0b5523f44b23a415fcde5bff0de6.camel@redhat.com>
 <453537998.54408252.1597762074665.JavaMail.zimbra@helmholtz-muenchen.de>
 <104df2890058a7948e93251043ba4fb43ed28446.camel@redhat.com>
 <61ea6f4310eac7382e19ba655dcb09cac7348b54.camel@redhat.com>
 <2116742364.55012219.1597847372185.JavaMail.zimbra@helmholtz-muenchen.de>
Message-ID: <9c91355be4a33a5e505e7e37eda2784c065289f3.camel@redhat.com>

On Wed, 2020-08-19 at 16:29 +0200, Lentes, Bernd wrote:
> ----- On Aug 19, 2020, at 4:04 PM, kgaillot kgaillot at redhat.com
> wrote:
> > > This appears to be a scheduler bug.
> > 
> > Fix is in master branch and will land in 2.0.5 expected at end of
> > the
> > year
> > 
> > https://github.com/ClusterLabs/pacemaker/pull/2146
> 
> A principal question:
> I have SLES 12 and i'm using the pacemaker version provided with the
> distribution.
> If this fix is backported depends on Suse.
> 
> If i install und update pacemaker manually (not the version provided
> by Suse),
> i loose my support from them, but have always the most recent code
> and fixes.
> 
> If i stay with the version from Suse i have support from them, but
> maybe not all fixes and not the most recent code.
> 
> What is your approach ?
> Recommendations ?

I'd recommend sticking with the supported version, and filing a bug
report with the distro asking for a specific fix to be backported when
you have the need.

Regarding the upstream project, running a release is fine, but I
wouldn't recommend running the master branch in production. Important
details of new features can change before release, and with frequent
development it's more likely to have regressions.

> 
> Thanks.
> 
> Bernd
> Helmholtz Zentrum M?nchen
> 
> Helmholtz Zentrum Muenchen
> Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
> Ingolstaedter Landstr. 1
> 85764 Neuherberg
> www.helmholtz-muenchen.de
> Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
> Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin
> Guenther
> Registergericht: Amtsgericht Muenchen HRB 6466
> USt-IdNr: DE 129521671
-- 
Ken Gaillot <kgaillot at redhat.com>


From Ulrich.Windl at rz.uni-regensburg.de  Thu Aug 20 01:56:32 2020
From: Ulrich.Windl at rz.uni-regensburg.de (Ulrich Windl)
Date: Thu, 20 Aug 2020 07:56:32 +0200
Subject: [ClusterLabs] anothe rcompletion bug in crm shell
 (2.1.2+git132.gbc9fde0 of SLES11)
Message-ID: <5F3E1090020000A10003AB41@gwsmtp.uni-regensburg.de>

Hi!

After having booted a cluster node yesterday, I checked the logs today (as the resources were expected to be rebalanced in the evening, controlled by rules).
When trying to check the utilization interactively using crm shell, I realized that completion is wrong:

When typing "utilization show <TAB>" in "crm(live)node#", I see a list of cluster resources, but not the list of utilization attributes. (Here it's easy as all utilization attributes start with "utl_", while primitives and closes start with "prm_" or "cln_", respectively.

This is for SLES11 SP4 without LTSS (Long-term support). However this bug is fixed in crmsh-4.1.0+git.1585823743.3acb5567 of SLES12 SP5 already.

Regards,
Ulrich




From Ulrich.Windl at rz.uni-regensburg.de  Thu Aug 20 02:10:56 2020
From: Ulrich.Windl at rz.uni-regensburg.de (Ulrich Windl)
Date: Thu, 20 Aug 2020 08:10:56 +0200
Subject: [ClusterLabs] Antw: [EXT] anothe rcompletion bug in crm shell
 (2.1.2+git132.gbc9fde0 of SLES11)
In-Reply-To: <5F3E1090020000A10003AB41@gwsmtp.uni-regensburg.de>
References: <5F3E1090020000A10003AB41@gwsmtp.uni-regensburg.de>
Message-ID: <5F3E13F0020000A10003AB45@gwsmtp.uni-regensburg.de>

Hi again!

While talking about it, I can imagine two useful extensions to "crm node
utilization":

1) Allow leaving out the attribute name, showing all utilization atributes
(much like "crm configure show" shows all resources)
2) Allow "*" as valid host name (to fulfill the syntax requirement of having
an argument at the position, expanding to all hosts

crm node utilization node|* show [attribute]

So "crm node utilization node1 show " would show all utilization attributes
for node1.
So "crm node utilization * show attr" would show all utilization
attribute"attr"  for all nodes.
So "crm node utilization *" would show all utilization attributes for all
nodes.

Maybe repositioning the arguments would make it even easier:
Instead of "crm utilization <node> <command> <attribute>" using "crm
utilization <command> [<node> [<attribute>]]" would make the syntax even more
straight-forward:
"crm node utilization show" would show all utilization attributes for all
nodes,
"crm node utilization show node" would show all utilization attributes for
"node", and
"crm node utilization show node attr" would show attribute "attr" for node
"node".
The probably less common case would be "crm node utilization show * attr" to
show all attribute "attr" for all nodes...

Maybe "[<attribute>]" could even be "[<attribzte>...]" (a list of attributes
to show)

Maybe even more clever syntax exists... ;-)

Regards,
Ulrich

>>> "Ulrich Windl" <Ulrich.Windl at rz.uni-regensburg.de> schrieb am 20.08.2020
um
07:56 in Nachricht <5F3E1090020000A10003AB41 at gwsmtp.uni-regensburg.de>:
> Hi!
> 
> After having booted a cluster node yesterday, I checked the logs today (as 
> the resources were expected to be rebalanced in the evening, controlled by 
> rules).
> When trying to check the utilization interactively using crm shell, I 
> realized that completion is wrong:
> 
> When typing "utilization show <TAB>" in "crm(live)node#", I see a list of 
> cluster resources, but not the list of utilization attributes. (Here it's 
> easy as all utilization attributes start with "utl_", while primitives and 
> closes start with "prm_" or "cln_", respectively.
> 
> This is for SLES11 SP4 without LTSS (Long?term support). However this bug is

> fixed in crmsh?4.1.0+git.1585823743.3acb5567 of SLES12 SP5 already.
> 
> Regards,
> Ulrich
> 
> 
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




From Ulrich.Windl at rz.uni-regensburg.de  Thu Aug 20 02:24:02 2020
From: Ulrich.Windl at rz.uni-regensburg.de (Ulrich Windl)
Date: Thu, 20 Aug 2020 08:24:02 +0200
Subject: [ClusterLabs] Antw: Antw: [EXT] anothe rcompletion bug in crm shell
 (2.1.2+git132.gbc9fde0 of SLES11)
In-Reply-To: <5F3E13F0020000A10003AB45@gwsmtp.uni-regensburg.de>
References: <5F3E1090020000A10003AB41@gwsmtp.uni-regensburg.de>
 <5F3E13F0020000A10003AB45@gwsmtp.uni-regensburg.de>
Message-ID: <5F3E1702020000A10003AB49@gwsmtp.uni-regensburg.de>

Hi another time!

I apologize for not having used crm shell a lot recently. After having sent
the last message, I realized that the node attributes displayed by the command
are the static ("original") values, not the dynamic (current) ones. For example
"ptest -LU" would show:
Utilization information:
Original: n01 capacity: utl_cpu=200 utl_ram=1240
Original: n01 capacity: utl_cpu=200 utl_ram=1240
Original: n03 capacity: utl_cpu=200 utl_ram=1240
...
Remaining: n01 capacity: utl_cpu=140 utl_ram=748
Remaining: n02 capacity: utl_cpu=100 utl_ram=728
Remaining: n03 capacity: utl_cpu=110 utl_ram=584

Still after reading the man poage of crm shell, I could not identify the
command to output the current values of the utilization atributes.

Browsing the output of "cibadmin -Q" I realized that the "current" utilization
values don't seem to be part of the CIB, not even the status section... 8-(

Regards,
Ulrich


>>> "Ulrich Windl" <Ulrich.Windl at rz.uni-regensburg.de> schrieb am 20.08.2020
um
08:10 in Nachricht <5F3E13F0020000A10003AB45 at gwsmtp.uni-regensburg.de>:
> Hi again!
> 
> While talking about it, I can imagine two useful extensions to "crm node
> utilization":
> 
> 1) Allow leaving out the attribute name, showing all utilization atributes
> (much like "crm configure show" shows all resources)
> 2) Allow "*" as valid host name (to fulfill the syntax requirement of
having
> an argument at the position, expanding to all hosts
> 
> crm node utilization node|* show [attribute]
> 
> So "crm node utilization node1 show " would show all utilization attributes
> for node1.
> So "crm node utilization * show attr" would show all utilization
> attribute"attr"  for all nodes.
> So "crm node utilization *" would show all utilization attributes for all
> nodes.
> 
> Maybe repositioning the arguments would make it even easier:
> Instead of "crm utilization <node> <command> <attribute>" using "crm
> utilization <command> [<node> [<attribute>]]" would make the syntax even
more
> straight-forward:
> "crm node utilization show" would show all utilization attributes for all
> nodes,
> "crm node utilization show node" would show all utilization attributes for
> "node", and
> "crm node utilization show node attr" would show attribute "attr" for node
> "node".
> The probably less common case would be "crm node utilization show * attr"
to
> show all attribute "attr" for all nodes...
> 
> Maybe "[<attribute>]" could even be "[<attribzte>...]" (a list of
attributes
> to show)
> 
> Maybe even more clever syntax exists... ;-)
> 
> Regards,
> Ulrich
> 
>>>> "Ulrich Windl" <Ulrich.Windl at rz.uni-regensburg.de> schrieb am 20.08.2020
> um
> 07:56 in Nachricht <5F3E1090020000A10003AB41 at gwsmtp.uni-regensburg.de>:
>> Hi!
>> 
>> After having booted a cluster node yesterday, I checked the logs today (as

>> the resources were expected to be rebalanced in the evening, controlled by

>> rules).
>> When trying to check the utilization interactively using crm shell, I 
>> realized that completion is wrong:
>> 
>> When typing "utilization show <TAB>" in "crm(live)node#", I see a list of 
>> cluster resources, but not the list of utilization attributes. (Here it's 
>> easy as all utilization attributes start with "utl_", while primitives and

>> closes start with "prm_" or "cln_", respectively.
>> 
>> This is for SLES11 SP4 without LTSS (Long?term support). However this bug
is
> 
>> fixed in crmsh?4.1.0+git.1585823743.3acb5567 of SLES12 SP5 already.
>> 
>> Regards,
>> Ulrich
>> 
>> 
>> 
>> _______________________________________________
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>> 
>> ClusterLabs home: https://www.clusterlabs.org/ 
> 
> 
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




From Ulrich.Windl at rz.uni-regensburg.de  Thu Aug 20 02:54:40 2020
From: Ulrich.Windl at rz.uni-regensburg.de (Ulrich Windl)
Date: Thu, 20 Aug 2020 08:54:40 +0200
Subject: [ClusterLabs] Antw: [EXT] Re:  why is node fenced ?
In-Reply-To: <2116742364.55012219.1597847372185.JavaMail.zimbra@helmholtz-muenchen.de>
References: <1894379294.27456141.1596036406000.JavaMail.zimbra@helmholtz-muenchen.de>
 <197754c0efa9654e23c0d3b25d7d1ac8a7e2f919.camel@redhat.com>
 <989597611.38099136.1597004244408.JavaMail.zimbra@helmholtz-muenchen.de>
 <1230152204.50050895.1597430263067.JavaMail.zimbra@helmholtz-muenchen.de>
 <84a98a74897f0b5523f44b23a415fcde5bff0de6.camel@redhat.com>
 <453537998.54408252.1597762074665.JavaMail.zimbra@helmholtz-muenchen.de>
 <104df2890058a7948e93251043ba4fb43ed28446.camel@redhat.com>
 <61ea6f4310eac7382e19ba655dcb09cac7348b54.camel@redhat.com>
 <2116742364.55012219.1597847372185.JavaMail.zimbra@helmholtz-muenchen.de>
Message-ID: <5F3E1E30020000A10003AB59@gwsmtp.uni-regensburg.de>

>>> "Lentes, Bernd" <bernd.lentes at helmholtz-muenchen.de> schrieb am 19.08.2020
um
16:29 in Nachricht
<2116742364.55012219.1597847372185.JavaMail.zimbra at helmholtz-muenchen.de>:

> ----- On Aug 19, 2020, at 4:04 PM, kgaillot kgaillot at redhat.com wrote:
>>> This appears to be a scheduler bug.
>> 
>> Fix is in master branch and will land in 2.0.5 expected at end of the
>> year
>> 
>> https://github.com/ClusterLabs/pacemaker/pull/2146 
> 
> A principal question:
> I have SLES 12 and i'm using the pacemaker version provided with the 
> distribution.
> If this fix is backported depends on Suse.
> 
> If i install und update pacemaker manually (not the version provided by 
> Suse),
> i loose my support from them, but have always the most recent code and 
> fixes.
> 
> If i stay with the version from Suse i have support from them, but maybe not

> all fixes and not the most recent code.
> 
> What is your approach ?
> Recommendations ?

Convince SUSE that it is a bug ;-)

> 
> Thanks.
> 
> Bernd
> Helmholtz Zentrum M?nchen
> 
> Helmholtz Zentrum Muenchen
> Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
> Ingolstaedter Landstr. 1
> 85764 Neuherberg
> www.helmholtz-muenchen.de 
> Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling
> Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin 
> Guenther
> Registergericht: Amtsgericht Muenchen HRB 6466
> USt-IdNr: DE 129521671
> 
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




From lists at alteeve.ca  Thu Aug 20 06:40:16 2020
From: lists at alteeve.ca (Digimer)
Date: Thu, 20 Aug 2020 06:40:16 -0400
Subject: [ClusterLabs] Format of '--lifetime' in 'pcs resource move'
Message-ID: <743e6d36-c0b8-5352-5ca9-738628e38239@alteeve.ca>

Hi all,

  Reading the pcs man page for the 'move' action, it talks about
'--lifetime' switch that appears to control when the location constraint
is removed;

====
       move   <resource   id>  [destination  node]  [--master]  [life?
       time=<lifetime>] [--wait[=n]]
              Move the resource off the node it is  currently  running
              on  by  creating  a -INFINITY location constraint to ban
              the node. If destination node is specified the  resource
              will be moved to that node by creating an INFINITY loca?
              tion constraint  to  prefer  the  destination  node.  If
              --master  is used the scope of the command is limited to
              the master role and you must use the promotable clone id
              (instead  of  the resource id). If lifetime is specified
              then the constraint will expire after that time,  other?
              wise  it  defaults to infinity and the constraint can be
              cleared manually with 'pcs resource clear' or 'pcs  con?
              straint  delete'.  If --wait is specified, pcs will wait
              up to 'n' seconds for the  resource  to  move  and  then
              return  0 on success or 1 on error. If 'n' is not speci?
              fied it defaults to 60 minutes. If you want the resource
              to preferably avoid running on some nodes but be able to
              failover to them use 'pcs constraint location avoids'.
====

I think I want to use this, as we move resources manually for various
reasons where the old host is still able to host the resource should a
node failure occur. So we'd love to immediately remove the location
constraint as soon as the move completes.

I tries using '--lifetime=60' as a test, assuming the format was
'seconds', but that was invalid. How is this switch meant to be used?

Cheers

-- 
Digimer
Papers and Projects: https://alteeve.com/w/
"I am, somehow, less interested in the weight and convolutions of
Einstein?s brain than in the near certainty that people of equal talent
have lived and died in cotton fields and sweatshops." - Stephen Jay Gould

From hunter86_bg at yahoo.com  Thu Aug 20 12:25:45 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 20 Aug 2020 19:25:45 +0300
Subject: [ClusterLabs] Format of '--lifetime' in 'pcs resource move'
In-Reply-To: <743e6d36-c0b8-5352-5ca9-738628e38239@alteeve.ca>
References: <743e6d36-c0b8-5352-5ca9-738628e38239@alteeve.ca>
Message-ID: <329B5D02-2BCB-4A2C-BC2B-CA3030E6A3B7@yahoo.com>

Have you tried ISO 8601 format.
For example: 'PT20M' 

The  ISo format  is described at:  https://manpages.debian.org/testing/crmsh/crm.8.en.html

Best Regards,
Strahil Nikolov

?? 20 ?????? 2020 ?. 13:40:16 GMT+03:00, Digimer <lists at alteeve.ca> ??????:
>Hi all,
>
>  Reading the pcs man page for the 'move' action, it talks about
>'--lifetime' switch that appears to control when the location
>constraint
>is removed;
>
>====
>       move   <resource   id>  [destination  node]  [--master]  [life?
>       time=<lifetime>] [--wait[=n]]
>              Move the resource off the node it is  currently  running
>              on  by  creating  a -INFINITY location constraint to ban
>              the node. If destination node is specified the  resource
>              will be moved to that node by creating an INFINITY loca?
>              tion constraint  to  prefer  the  destination  node.  If
>              --master  is used the scope of the command is limited to
>              the master role and you must use the promotable clone id
>              (instead  of  the resource id). If lifetime is specified
>              then the constraint will expire after that time,  other?
>              wise  it  defaults to infinity and the constraint can be
>              cleared manually with 'pcs resource clear' or 'pcs  con?
>              straint  delete'.  If --wait is specified, pcs will wait
>              up to 'n' seconds for the  resource  to  move  and  then
>              return  0 on success or 1 on error. If 'n' is not speci?
>              fied it defaults to 60 minutes. If you want the resource
>              to preferably avoid running on some nodes but be able to
>              failover to them use 'pcs constraint location avoids'.
>====
>
>I think I want to use this, as we move resources manually for various
>reasons where the old host is still able to host the resource should a
>node failure occur. So we'd love to immediately remove the location
>constraint as soon as the move completes.
>
>I tries using '--lifetime=60' as a test, assuming the format was
>'seconds', but that was invalid. How is this switch meant to be used?
>
>Cheers
>
>-- 
>Digimer
>Papers and Projects: https://alteeve.com/w/
>"I am, somehow, less interested in the weight and convolutions of
>Einstein?s brain than in the near certainty that people of equal talent
>have lived and died in cotton fields and sweatshops." - Stephen Jay
>Gould
>_______________________________________________
>Manage your subscription:
>https://lists.clusterlabs.org/mailman/listinfo/users
>
>ClusterLabs home: https://www.clusterlabs.org/

From Ulrich.Windl at rz.uni-regensburg.de  Fri Aug 21 01:56:53 2020
From: Ulrich.Windl at rz.uni-regensburg.de (Ulrich Windl)
Date: Fri, 21 Aug 2020 07:56:53 +0200
Subject: [ClusterLabs] Antw: [EXT] Re: Format of '--lifetime' in 'pcs
 resource move'
In-Reply-To: <329B5D02-2BCB-4A2C-BC2B-CA3030E6A3B7@yahoo.com>
References: <743e6d36-c0b8-5352-5ca9-738628e38239@alteeve.ca>
 <329B5D02-2BCB-4A2C-BC2B-CA3030E6A3B7@yahoo.com>
Message-ID: <5F3F6225020000A10003ABA8@gwsmtp.uni-regensburg.de>

>>> Strahil Nikolov <hunter86_bg at yahoo.com> schrieb am 20.08.2020 um 18:25 in
Nachricht <329B5D02-2BCB-4A2C-BC2B-CA3030E6A3B7 at yahoo.com>:
> Have you tried ISO 8601 format.
> For example: 'PT20M' 

And watch out not to mix Minutes wth Months ;-)

> 
> The  ISo format  is described at:  
> https://manpages.debian.org/testing/crmsh/crm.8.en.html 
> 
> Best Regards,
> Strahil Nikolov
> 
> ?? 20 ?????? 2020 ?. 13:40:16 GMT+03:00, Digimer <lists at alteeve.ca> ??????:
>>Hi all,
>>
>>  Reading the pcs man page for the 'move' action, it talks about
>>'--lifetime' switch that appears to control when the location
>>constraint
>>is removed;
>>
>>====
>>       move   <resource   id>  [destination  node]  [--master]  [life?
>>       time=<lifetime>] [--wait[=n]]
>>              Move the resource off the node it is  currently  running
>>              on  by  creating  a -INFINITY location constraint to ban
>>              the node. If destination node is specified the  resource
>>              will be moved to that node by creating an INFINITY loca?
>>              tion constraint  to  prefer  the  destination  node.  If
>>              --master  is used the scope of the command is limited to
>>              the master role and you must use the promotable clone id
>>              (instead  of  the resource id). If lifetime is specified
>>              then the constraint will expire after that time,  other?
>>              wise  it  defaults to infinity and the constraint can be
>>              cleared manually with 'pcs resource clear' or 'pcs  con?
>>              straint  delete'.  If --wait is specified, pcs will wait
>>              up to 'n' seconds for the  resource  to  move  and  then
>>              return  0 on success or 1 on error. If 'n' is not speci?
>>              fied it defaults to 60 minutes. If you want the resource
>>              to preferably avoid running on some nodes but be able to
>>              failover to them use 'pcs constraint location avoids'.
>>====
>>
>>I think I want to use this, as we move resources manually for various
>>reasons where the old host is still able to host the resource should a
>>node failure occur. So we'd love to immediately remove the location
>>constraint as soon as the move completes.
>>
>>I tries using '--lifetime=60' as a test, assuming the format was
>>'seconds', but that was invalid. How is this switch meant to be used?
>>
>>Cheers
>>
>>-- 
>>Digimer
>>Papers and Projects: https://alteeve.com/w/ 
>>"I am, somehow, less interested in the weight and convolutions of
>>Einstein?s brain than in the near certainty that people of equal talent
>>have lived and died in cotton fields and sweatshops." - Stephen Jay
>>Gould
>>_______________________________________________
>>Manage your subscription:
>>https://lists.clusterlabs.org/mailman/listinfo/users 
>>
>>ClusterLabs home: https://www.clusterlabs.org/ 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




From mark.battistella at outlook.com  Fri Aug 21 03:04:37 2020
From: mark.battistella at outlook.com (Mark Battistella)
Date: Fri, 21 Aug 2020 07:04:37 +0000
Subject: [ClusterLabs] Active-Active cluster CentOS 8
Message-ID: <MWHPR1801MB1999D37DAED79625545BCA92EC5B0@MWHPR1801MB1999.namprd18.prod.outlook.com>

Hi,

I was wondering if I could get some help when following along with the Clusters from Scratch part 9: https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/2.0/html-single/Clusters_from_Scratch/index.html#_install_cluster_filesystem_software

The first step is to install dlm which cant be found - but dlm-lib can.

Is there any update to the installation or alternative? I'd love to be able to have an active-active filesystem.

I've looked through the repo to see if there were any answers but I cant find anything

Thanks,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200821/31b28cb2/attachment.htm>

From kgaillot at redhat.com  Fri Aug 21 14:16:11 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Fri, 21 Aug 2020 13:16:11 -0500
Subject: [ClusterLabs] Coming in Pacemaker 2.0.5: better start-up/shutdown
 coordination with sbd
Message-ID: <b7e8542b0e9fbdc4ef008cea524acb76afe86e62.camel@redhat.com>

Hi all,

Looking ahead to the Pacemaker 2.0.5 release expected toward the end of
this year, we will have improvements of interest to anyone running
clusters with sbd.

Previously at start-up, if sbd was blocked from contacting Pacemaker's
CIB in a way that looked like pacemaker wasn't running (SELinux being a
good example), pacemaker would run resources without protection from
sbd. Now, if sbd is running, pacemaker will wait until sbd contacts it
before it will start any resources, so the cluster is protected in this
situation.

Additionally, sbd will now periodically contact the main pacemaker
daemon for a status report. Currently, this is just an immediate
response, but it ensures that the main pacemaker daemon is responsive
to IPC requests. This is a bit more assurance that pacemaker is not
only running, but functioning properly. In future versions, we will
have even more in-depth health checks as part of this feature.

Previously at shutdown, sbd determined a clean pacemaker shutdown by
checking whether any resources were running at shutdown. This would
lead to sbd fencing if pacemaker shut down in maintenance mode with
resources active. Now, sbd will determine clean shutdowns as part of
the status report described above, avoiding that situation.

These behaviors will be controlled by a new option in
/etc/sysconfig/sbd or /etc/default/sbd, SBD_SYNC_RESOURCE_STARTUP. This
defaults to "no" for backward compatibility when a newer sbd is used
with an older pacemaker or vice versa. Distributions may change the
value to "yes" since they can ensure both sbd and pacemaker versions
support it; users who build their own installations can set it
themselves if both versions support it.
-- 
Ken Gaillot <kgaillot at redhat.com>


From bubble at hoster-ok.com  Fri Aug 21 14:55:16 2020
From: bubble at hoster-ok.com (Vladislav Bogdanov)
Date: Fri, 21 Aug 2020 21:55:16 +0300
Subject: [ClusterLabs] Coming in Pacemaker 2.0.5: better
 start-up/shutdown coordination with sbd
In-Reply-To: <b7e8542b0e9fbdc4ef008cea524acb76afe86e62.camel@redhat.com>
References: <b7e8542b0e9fbdc4ef008cea524acb76afe86e62.camel@redhat.com>
Message-ID: <f16f1c7717b388fcd1445b74340c1372719c2d7e.camel@hoster-ok.com>

Hi,

btw, is sbd is now able to handle cib diffs internally?
Last time I tried to use it with frequently changing CIB, it became a
CPU hog - it requested full CIB copy on every change.


Fri, 21/08/2020 ? 13:16 -0500, Ken Gaillot wrote:
> Hi all,
> 
> Looking ahead to the Pacemaker 2.0.5 release expected toward the end of
> this year, we will have improvements of interest to anyone running
> clusters with sbd.
> 
> Previously at start-up, if sbd was blocked from contacting Pacemaker's
> CIB in a way that looked like pacemaker wasn't running (SELinux being a
> good example), pacemaker would run resources without protection from
> sbd. Now, if sbd is running, pacemaker will wait until sbd contacts it
> before it will start any resources, so the cluster is protected in this
> situation.
> 
> Additionally, sbd will now periodically contact the main pacemaker
> daemon for a status report. Currently, this is just an immediate
> response, but it ensures that the main pacemaker daemon is responsive
> to IPC requests. This is a bit more assurance that pacemaker is not
> only running, but functioning properly. In future versions, we will
> have even more in-depth health checks as part of this feature.
> 
> Previously at shutdown, sbd determined a clean pacemaker shutdown by
> checking whether any resources were running at shutdown. This would
> lead to sbd fencing if pacemaker shut down in maintenance mode with
> resources active. Now, sbd will determine clean shutdowns as part of
> the status report described above, avoiding that situation.
> 
> These behaviors will be controlled by a new option in
> /etc/sysconfig/sbd or /etc/default/sbd, SBD_SYNC_RESOURCE_STARTUP. This
> defaults to "no" for backward compatibility when a newer sbd is used
> with an older pacemaker or vice versa. Distributions may change the
> value to "yes" since they can ensure both sbd and pacemaker versions
> support it; users who build their own installations can set it
> themselves if both versions support it.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200821/a9888acf/attachment.htm>

From kgaillot at redhat.com  Fri Aug 21 16:35:45 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Fri, 21 Aug 2020 15:35:45 -0500
Subject: [ClusterLabs] Active-Active cluster CentOS 8
In-Reply-To: <MWHPR1801MB1999D37DAED79625545BCA92EC5B0@MWHPR1801MB1999.namprd18.prod.outlook.com>
References: <MWHPR1801MB1999D37DAED79625545BCA92EC5B0@MWHPR1801MB1999.namprd18.prod.outlook.com>
Message-ID: <33db3777fa23969b31a93b505a5aba46416f4a7c.camel@redhat.com>

On Fri, 2020-08-21 at 07:04 +0000, Mark Battistella wrote:
> Hi,
> 
> I was wondering if I could get some help when following along with
> the Clusters from Scratch part 9: 
> https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/2.0/html-single/Clusters_from_Scratch/index.html#_install_cluster_filesystem_software
> 
> The first step is to install dlm which cant be found - but dlm-lib
> can.
> 
> Is there any update to the installation or alternative? I'd love to
> be able to have an active-active filesystem.
> 
> I've looked through the repo to see if there were any answers but I
> cant find anything
> 
> Thanks,
> Mark

Looks like it's a known CentOS packaging issue:

https://bugs.centos.org/view.php?id=16939

-- 
Ken Gaillot <kgaillot at redhat.com>


From arvidjaar at gmail.com  Sat Aug 22 05:02:00 2020
From: arvidjaar at gmail.com (Andrei Borzenkov)
Date: Sat, 22 Aug 2020 12:02:00 +0300
Subject: [ClusterLabs] Coming in Pacemaker 2.0.5: better
 start-up/shutdown coordination with sbd
In-Reply-To: <b7e8542b0e9fbdc4ef008cea524acb76afe86e62.camel@redhat.com>
References: <b7e8542b0e9fbdc4ef008cea524acb76afe86e62.camel@redhat.com>
Message-ID: <f1866409-7fa4-f2be-16a4-b08d80b51307@gmail.com>

21.08.2020 21:16, Ken Gaillot ?????:
> 
> Previously at shutdown, sbd determined a clean pacemaker shutdown by
> checking whether any resources were running at shutdown. This would
> lead to sbd fencing if pacemaker shut down in maintenance mode with
> resources active.


What conditions lead to it? I tried to reproduce it, but so far I was
able to set node in maintenance mode and stop pacemaker on it without
any ill effects. This is openSUSE Tumebleweed with sbd at commit
4b617a1b8f16033ac464cf2d562f0646082c40fa.

From kwenning at redhat.com  Sun Aug 23 04:45:25 2020
From: kwenning at redhat.com (Klaus Wenninger)
Date: Sun, 23 Aug 2020 10:45:25 +0200
Subject: [ClusterLabs] Coming in Pacemaker 2.0.5: better
 start-up/shutdown coordination with sbd
In-Reply-To: <f1866409-7fa4-f2be-16a4-b08d80b51307@gmail.com>
References: <b7e8542b0e9fbdc4ef008cea524acb76afe86e62.camel@redhat.com>
 <f1866409-7fa4-f2be-16a4-b08d80b51307@gmail.com>
Message-ID: <8ad9b08b-abb4-c18b-8965-45fc4be3f2a6@redhat.com>

On 8/22/20 11:02 AM, Andrei Borzenkov wrote:
> 21.08.2020 21:16, Ken Gaillot ?????:
>> Previously at shutdown, sbd determined a clean pacemaker shutdown by
>> checking whether any resources were running at shutdown. This would
>> lead to sbd fencing if pacemaker shut down in maintenance mode with
>> resources active.
>
> What conditions lead to it? I tried to reproduce it, but so far I was
> able to set node in maintenance mode and stop pacemaker on it without
> any ill effects. This is openSUSE Tumebleweed with sbd at commit
> 4b617a1b8f16033ac464cf2d562f0646082c40fa.
You shouldn't be able to reproduce this as there was an individual
fix for this condition:

commit 824fe834c67fb7bae7feb87607381f9fa8fa2945
Author: Klaus Wenninger <klaus.wenninger at aon.at>
Date:?? Fri Jun 7 19:09:06 2019 +0200

??? Fix: sbd-pacemaker: assume graceful exit if leftovers are unmanged

Point is that sbd now doesn't need to make complicated assumptions
anymore if a pacemaker-shutdown was graceful or not as we've
introduced a channel via which pacemaker is explicitly telling sbd
that it has successfully reached a clean shutdown state.

Klaus
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/


From kwenning at redhat.com  Sun Aug 23 06:08:22 2020
From: kwenning at redhat.com (Klaus Wenninger)
Date: Sun, 23 Aug 2020 12:08:22 +0200
Subject: [ClusterLabs] Coming in Pacemaker 2.0.5: better
 start-up/shutdown coordination with sbd
In-Reply-To: <f16f1c7717b388fcd1445b74340c1372719c2d7e.camel@hoster-ok.com>
References: <b7e8542b0e9fbdc4ef008cea524acb76afe86e62.camel@redhat.com>
 <f16f1c7717b388fcd1445b74340c1372719c2d7e.camel@hoster-ok.com>
Message-ID: <ae3bf117-9f10-8e2c-edf5-2b54f13f9264@redhat.com>

On 8/21/20 8:55 PM, Vladislav Bogdanov wrote:
> Hi,
>
> btw, is sbd is now able to handle cib diffs internally?
> Last time I tried to use it with frequently changing CIB, it became a
> CPU hog - it requested full CIB copy on every change.
Actually sbd should have been able to handle cib-diffs since ever.
Are you sure it requested a full copy of the CIB with every change?
Atm it should request a full update roughly twice every watchdog-timeout
and in between just noop-pings to the cib-api - as long as imbedding the
diffs goes OK of course.
In general we need full cib-updates as otherwise loss of a cib-diff
would mean possibly missing node-state updates.
What it on top does is convert the cib to a cluster-state roughly
every second or with every 10th cib-diff. The latter might impose
some cpu-usage when cib is updating at a high rate of course and
might not be really needed.
With the new pacemakerd-API we don't need the cib-diffs anymore
for graceful-shutdown-detection. Thus easiest might be to disable
diff-handling completely when pacemakerd-API is used.
>
>
> Fri, 21/08/2020 ? 13:16 -0500, Ken Gaillot wrote:
>> Hi all,
>> Looking ahead to the Pacemaker 2.0.5 release expected toward the end of
>> this year, we will have improvements of interest to anyone running
>> clusters with sbd.
>> Previously at start-up, if sbd was blocked from contacting Pacemaker's
>> CIB in a way that looked like pacemaker wasn't running (SELinux being a
>> good example), pacemaker would run resources without protection from
>> sbd. Now, if sbd is running, pacemaker will wait until sbd contacts it
>> before it will start any resources, so the cluster is protected in this
>> situation.
>> Additionally, sbd will now periodically contact the main pacemaker
>> daemon for a status report. Currently, this is just an immediate
>> response, but it ensures that the main pacemaker daemon is responsive
>> to IPC requests. This is a bit more assurance that pacemaker is not
>> only running, but functioning properly. In future versions, we will
>> have even more in-depth health checks as part of this feature.
>> Previously at shutdown, sbd determined a clean pacemaker shutdown by
>> checking whether any resources were running at shutdown. This would
>> lead to sbd fencing if pacemaker shut down in maintenance mode with
>> resources active. Now, sbd will determine clean shutdowns as part of
>> the status report described above, avoiding that situation.
>> These behaviors will be controlled by a new option in
>> /etc/sysconfig/sbd or /etc/default/sbd, SBD_SYNC_RESOURCE_STARTUP. This
>> defaults to "no" for backward compatibility when a newer sbd is used
>> with an older pacemaker or vice versa. Distributions may change the
>> value to "yes" since they can ensure both sbd and pacemaker versions
>> support it; users who build their own installations can set it
>> themselves if both versions support it.
>
>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200823/f3c98203/attachment.htm>

From hunter86_bg at yahoo.com  Sun Aug 23 06:09:25 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Sun, 23 Aug 2020 10:09:25 +0000 (UTC)
Subject: [ClusterLabs] Active-Active cluster CentOS 8
In-Reply-To: <MWHPR1801MB1999D37DAED79625545BCA92EC5B0@MWHPR1801MB1999.namprd18.prod.outlook.com>
References: <MWHPR1801MB1999D37DAED79625545BCA92EC5B0@MWHPR1801MB1999.namprd18.prod.outlook.com>
Message-ID: <1562701513.4604011.1598177365088@mail.yahoo.com>

There is a topic about that at?https://bugs.centos.org/view.php?id=16939

Based on the comments you can obtain it from?https://koji.mbox.centos.org/koji/buildinfo?buildID=4801 , but I haven' tested it.

Best Regards,
Strahil Nikolov






? ?????, 21 ?????? 2020 ?., 18:30:31 ???????+3, Mark Battistella <mark.battistella at outlook.com> ??????: 





??

Hi,




I was wondering?if I could get some help when following along with the Clusters from Scratch part 9:?https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/2.0/html-single/Clusters_from_Scratch/index.html#_install_cluster_filesystem_software




The first step is to install dlm which cant be found - but dlm-lib can.




Is there any update to the installation or alternative? I'd love to be able to have an active-active filesystem.




I've looked through the repo to see if there were any answers but I cant find anything




Thanks,

Mark








_______________________________________________
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

From bubble at hoster-ok.com  Sun Aug 23 06:39:43 2020
From: bubble at hoster-ok.com (Vladislav Bogdanov)
Date: Sun, 23 Aug 2020 13:39:43 +0300
Subject: [ClusterLabs] Coming in Pacemaker 2.0.5: better
 start-up/shutdown coordination with sbd
In-Reply-To: <ae3bf117-9f10-8e2c-edf5-2b54f13f9264@redhat.com>
References: <b7e8542b0e9fbdc4ef008cea524acb76afe86e62.camel@redhat.com>
 <f16f1c7717b388fcd1445b74340c1372719c2d7e.camel@hoster-ok.com>
 <ae3bf117-9f10-8e2c-edf5-2b54f13f9264@redhat.com>
Message-ID: <1741ae70998.27ed.79e33407a6dbe711362be892215840ad@hoster-ok.com>

Good to know it is now not needed. You are correct about logic, yes, I just 
forgot details. I just recall that sbd added too much load during cluster 
start and recoveries.

Thank you!

On August 23, 2020 1:23:37 PM Klaus Wenninger <kwenning at redhat.com> wrote:
> On 8/21/20 8:55 PM, Vladislav Bogdanov wrote:
>> Hi,
>>
>> btw, is sbd is now able to handle cib diffs internally?
>> Last time I tried to use it with frequently changing CIB, it became a CPU 
>> hog - it requested full CIB copy on every change.
> Actually sbd should have been able to handle cib-diffs since ever.
> Are you sure it requested a full copy of the CIB with every change?
> Atm it should request a full update roughly twice every watchdog-timeout
> and in between just noop-pings to the cib-api - as long as imbedding the
> diffs goes OK of course.
> In general we need full cib-updates as otherwise loss of a cib-diff
> would mean possibly missing node-state updates.
> What it on top does is convert the cib to a cluster-state roughly
> every second or with every 10th cib-diff. The latter might impose
> some cpu-usage when cib is updating at a high rate of course and
> might not be really needed.
> With the new pacemakerd-API we don't need the cib-diffs anymore
> for graceful-shutdown-detection. Thus easiest might be to disable
> diff-handling completely when pacemakerd-API is used.
>>
>>
>> Fri, 21/08/2020 ? 13:16 -0500, Ken Gaillot wrote:
>>> Hi all,
>>>
>>> Looking ahead to the Pacemaker 2.0.5 release expected toward the end of
>>> this year, we will have improvements of interest to anyone running
>>> clusters with sbd.
>>>
>>> Previously at start-up, if sbd was blocked from contacting Pacemaker's
>>> CIB in a way that looked like pacemaker wasn't running (SELinux being a
>>> good example), pacemaker would run resources without protection from
>>> sbd. Now, if sbd is running, pacemaker will wait until sbd contacts it
>>> before it will start any resources, so the cluster is protected in this
>>> situation.
>>>
>>> Additionally, sbd will now periodically contact the main pacemaker
>>> daemon for a status report. Currently, this is just an immediate
>>> response, but it ensures that the main pacemaker daemon is responsive
>>> to IPC requests. This is a bit more assurance that pacemaker is not
>>> only running, but functioning properly. In future versions, we will
>>> have even more in-depth health checks as part of this feature.
>>>
>>> Previously at shutdown, sbd determined a clean pacemaker shutdown by
>>> checking whether any resources were running at shutdown. This would
>>> lead to sbd fencing if pacemaker shut down in maintenance mode with
>>> resources active. Now, sbd will determine clean shutdowns as part of
>>> the status report described above, avoiding that situation.
>>>
>>> These behaviors will be controlled by a new option in
>>> /etc/sysconfig/sbd or /etc/default/sbd, SBD_SYNC_RESOURCE_STARTUP. This
>>> defaults to "no" for backward compatibility when a newer sbd is used
>>> with an older pacemaker or vice versa. Distributions may change the
>>> value to "yes" since they can ensure both sbd and pacemaker versions
>>> support it; users who build their own installations can set it
>>> themselves if both versions support it.
>>
>>
>> _______________________________________________
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: 
>> https://www.clusterlabs.org/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200823/2f0d5881/attachment-0001.htm>

From Ulrich.Windl at rz.uni-regensburg.de  Mon Aug 24 01:56:58 2020
From: Ulrich.Windl at rz.uni-regensburg.de (Ulrich Windl)
Date: Mon, 24 Aug 2020 07:56:58 +0200
Subject: [ClusterLabs] Antw: [EXT]  Active-Active cluster CentOS 8
In-Reply-To: <MWHPR1801MB1999D37DAED79625545BCA92EC5B0@MWHPR1801MB1999.namprd18.prod.outlook.com>
References: <MWHPR1801MB1999D37DAED79625545BCA92EC5B0@MWHPR1801MB1999.namprd18.prod.outlook.com>
Message-ID: <5F4356AA020000A10003AC39@gwsmtp.uni-regensburg.de>

>>> Mark Battistella <mark.battistella at outlook.com> schrieb am 21.08.2020 um
09:04
in Nachricht
<MWHPR1801MB1999D37DAED79625545BCA92EC5B0 at MWHPR1801MB1999.namprd18.prod.outlook.
om>:
> Hi,
> 
> I was wondering if I could get some help when following along with the 
> Clusters from Scratch part 9: 
>
https://clusterlabs.org/pacemaker/doc/en?US/Pacemaker/2.0/html?single/Cluster

> s_from_Scratch/index.html#_install_cluster_filesystem_software
> 
> The first step is to install dlm which cant be found ? but dlm?lib can.
> 
> Is there any update to the installation or alternative? I'd love to be able

> to have an active?active filesystem.

In SLES DLM seems to be a part of the kernel, with only dlm_controld.pcmk
being known in user-land, which in turn is part of package libdlm.

> 
> I've looked through the repo to see if there were any answers but I cant 
> find anything
> 
> Thanks,
> Mark




From Ulrich.Windl at rz.uni-regensburg.de  Mon Aug 24 02:04:26 2020
From: Ulrich.Windl at rz.uni-regensburg.de (Ulrich Windl)
Date: Mon, 24 Aug 2020 08:04:26 +0200
Subject: [ClusterLabs] Antw: [EXT] Re: Coming in Pacemaker 2.0.5: better
 start-up/shutdown coordination with sbd
In-Reply-To: <f16f1c7717b388fcd1445b74340c1372719c2d7e.camel@hoster-ok.com>
References: <b7e8542b0e9fbdc4ef008cea524acb76afe86e62.camel@redhat.com>
 <f16f1c7717b388fcd1445b74340c1372719c2d7e.camel@hoster-ok.com>
Message-ID: <5F43586A020000A10003AC3D@gwsmtp.uni-regensburg.de>

>>> Vladislav Bogdanov <bubble at hoster-ok.com> schrieb am 21.08.2020 um 20:55
in
Nachricht <f16f1c7717b388fcd1445b74340c1372719c2d7e.camel at hoster-ok.com>:
> Hi,
> 
> btw, is sbd is now able to handle cib diffs internally?
> Last time I tried to use it with frequently changing CIB, it became a
> CPU hog - it requested full CIB copy on every change.

Hi!

I also wonder whether sbd is a tool to fence hosts, or a node-quorum maker
that controls the cluster.
I think the cluster should control sbd, not the other way 'round.

Regards,
Ulrich

> 
> 
> Fri, 21/08/2020 ? 13:16 -0500, Ken Gaillot wrote:
>> Hi all,
>> 
>> Looking ahead to the Pacemaker 2.0.5 release expected toward the end of
>> this year, we will have improvements of interest to anyone running
>> clusters with sbd.
>> 
>> Previously at start-up, if sbd was blocked from contacting Pacemaker's
>> CIB in a way that looked like pacemaker wasn't running (SELinux being a
>> good example), pacemaker would run resources without protection from
>> sbd. Now, if sbd is running, pacemaker will wait until sbd contacts it
>> before it will start any resources, so the cluster is protected in this
>> situation.
>> 
>> Additionally, sbd will now periodically contact the main pacemaker
>> daemon for a status report. Currently, this is just an immediate
>> response, but it ensures that the main pacemaker daemon is responsive
>> to IPC requests. This is a bit more assurance that pacemaker is not
>> only running, but functioning properly. In future versions, we will
>> have even more in-depth health checks as part of this feature.
>> 
>> Previously at shutdown, sbd determined a clean pacemaker shutdown by
>> checking whether any resources were running at shutdown. This would
>> lead to sbd fencing if pacemaker shut down in maintenance mode with
>> resources active. Now, sbd will determine clean shutdowns as part of
>> the status report described above, avoiding that situation.
>> 
>> These behaviors will be controlled by a new option in
>> /etc/sysconfig/sbd or /etc/default/sbd, SBD_SYNC_RESOURCE_STARTUP. This
>> defaults to "no" for backward compatibility when a newer sbd is used
>> with an older pacemaker or vice versa. Distributions may change the
>> value to "yes" since they can ensure both sbd and pacemaker versions
>> support it; users who build their own installations can set it
>> themselves if both versions support it.




From kwenning at redhat.com  Mon Aug 24 03:52:19 2020
From: kwenning at redhat.com (Klaus Wenninger)
Date: Mon, 24 Aug 2020 09:52:19 +0200
Subject: [ClusterLabs] Antw: [EXT] Re: Coming in Pacemaker 2.0.5: better
 start-up/shutdown coordination with sbd
In-Reply-To: <5F43586A020000A10003AC3D@gwsmtp.uni-regensburg.de>
References: <b7e8542b0e9fbdc4ef008cea524acb76afe86e62.camel@redhat.com>
 <f16f1c7717b388fcd1445b74340c1372719c2d7e.camel@hoster-ok.com>
 <5F43586A020000A10003AC3D@gwsmtp.uni-regensburg.de>
Message-ID: <20c772b6-8573-f1db-4a1d-50f0a0fb8c2a@redhat.com>

On 8/24/20 8:04 AM, Ulrich Windl wrote:
>>>> Vladislav Bogdanov <bubble at hoster-ok.com> schrieb am 21.08.2020 um 20:55
> in
> Nachricht <f16f1c7717b388fcd1445b74340c1372719c2d7e.camel at hoster-ok.com>:
>> Hi,
>>
>> btw, is sbd is now able to handle cib diffs internally?
>> Last time I tried to use it with frequently changing CIB, it became a
>> CPU hog - it requested full CIB copy on every change.
> Hi!
>
> I also wonder whether sbd is a tool to fence hosts, or a node-quorum maker
> that controls the cluster.
> I think the cluster should control sbd, not the other way 'round.
None of those I suppose - at least not solely ;-)

sbd definitely has no influence on quorum - so no quorum-maker.
In its purest way of operation with 3 shared disks and no
cluster-awarenesssbd probably is close to what you envision.
Unfortunately 3 disks are quite some extra effort and you may want
to go with a single disk that doesn't become a spof. For certain
scenariospure watchdog-fencing may become interesting as well.

For all of these cases sbd needs to observe the local node health
and ifit is part of a quorate cluster-partition (sees the peer
in the 2-node case).
And as we are not living in a perfect world we can of course not
relyon this supervision never to be stuck or something.
Fortunately sbd is simpleenough that the main supervision is
done in a simple loop that can beeasily supervised by a
hardware watchdog.

Of course we could have done the hardware watchdog supervision
in pacemaker. But sbd & corosync need such a thing as well so
that wouldhave meant to pull that all into pacemaker - loosing
that simplicitymentioned. (For completeness: There meanwhile
is a heartbeat between corosync and sbd as well to compensate
for the hardware-watchdog-interface corosync would offer
alternatively but which would hog the hardware watchdog.)

We could have used one of the mechanisms that
provide multiple watchdogs supervised by a hardware watchdog
(e.g. systemd - that per design makes it hard to name a time
when you actually can assume a node to be rebooted when
misbehaving) but there is actually nothing you find on every
Linux platform.

So you see that the architecture is kind of natural and makes
sense. Introduction of the pacemakerd-API and it being used
in sbd definitely goes in the direction of moving intelligence
out of sbd. Not saying everything is perfect and no improvement
possible - of course ;-)

Klaus
>
> Regards,
> Ulrich
>
>>
>> Fri, 21/08/2020 ? 13:16 -0500, Ken Gaillot wrote:
>>> Hi all,
>>>
>>> Looking ahead to the Pacemaker 2.0.5 release expected toward the end of
>>> this year, we will have improvements of interest to anyone running
>>> clusters with sbd.
>>>
>>> Previously at start-up, if sbd was blocked from contacting Pacemaker's
>>> CIB in a way that looked like pacemaker wasn't running (SELinux being a
>>> good example), pacemaker would run resources without protection from
>>> sbd. Now, if sbd is running, pacemaker will wait until sbd contacts it
>>> before it will start any resources, so the cluster is protected in this
>>> situation.
>>>
>>> Additionally, sbd will now periodically contact the main pacemaker
>>> daemon for a status report. Currently, this is just an immediate
>>> response, but it ensures that the main pacemaker daemon is responsive
>>> to IPC requests. This is a bit more assurance that pacemaker is not
>>> only running, but functioning properly. In future versions, we will
>>> have even more in-depth health checks as part of this feature.
>>>
>>> Previously at shutdown, sbd determined a clean pacemaker shutdown by
>>> checking whether any resources were running at shutdown. This would
>>> lead to sbd fencing if pacemaker shut down in maintenance mode with
>>> resources active. Now, sbd will determine clean shutdowns as part of
>>> the status report described above, avoiding that situation.
>>>
>>> These behaviors will be controlled by a new option in
>>> /etc/sysconfig/sbd or /etc/default/sbd, SBD_SYNC_RESOURCE_STARTUP. This
>>> defaults to "no" for backward compatibility when a newer sbd is used
>>> with an older pacemaker or vice versa. Distributions may change the
>>> value to "yes" since they can ensure both sbd and pacemaker versions
>>> support it; users who build their own installations can set it
>>> themselves if both versions support it.
>
>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/


From rohitsaini111.forum at gmail.com  Tue Aug 25 02:58:53 2020
From: rohitsaini111.forum at gmail.com (Rohit Saini)
Date: Tue, 25 Aug 2020 12:28:53 +0530
Subject: [ClusterLabs] Behavior of corosync kill
Message-ID: <CA+ZsWTWt7t7rNTsRXpzbfUL4bsy6vR6wy0ZQ642fSfubtSjiDQ@mail.gmail.com>

Hi All,
I am seeing the following behavior. Can someone clarify if this is
intended behavior. If yes, then why so? Please let me know if logs are
needed for better clarity.

1. Without Stonith:
Continuous corosync kill on master causes switchover and makes another node
as master. But as soon as this corosync recovers, it becomes master again.
Shouldn't it become slave now?

2. With Stonith:
Sometimes, on corosync kill, that node gets shooted by stonith but
sometimes not. Not able to understand this fluctuating behavior. Does it
have to do anything with faster recovery of corosync, which stonith fails
to detect?

I am using
corosync-2.4.5-4.el7.x86_64
pacemaker-1.1.19-8.el7.x86_64
centos 7.6.1810

Thanks,
Rohit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200825/1852c2d6/attachment.htm>

From arvidjaar at gmail.com  Tue Aug 25 04:28:49 2020
From: arvidjaar at gmail.com (Andrei Borzenkov)
Date: Tue, 25 Aug 2020 11:28:49 +0300
Subject: [ClusterLabs] Behavior of corosync kill
In-Reply-To: <CA+ZsWTWt7t7rNTsRXpzbfUL4bsy6vR6wy0ZQ642fSfubtSjiDQ@mail.gmail.com>
References: <CA+ZsWTWt7t7rNTsRXpzbfUL4bsy6vR6wy0ZQ642fSfubtSjiDQ@mail.gmail.com>
Message-ID: <CAA91j0W+X3gfKXNt-3YBobKke2B+3tEcx-3AVwL1Odi_GEiJ9Q@mail.gmail.com>

On Tue, Aug 25, 2020 at 10:00 AM Rohit Saini
<rohitsaini111.forum at gmail.com> wrote:
>
> Hi All,
> I am seeing the following behavior. Can someone clarify if this is intended behavior. If yes, then why so? Please let me know if logs are needed for better clarity.
>
> 1. Without Stonith:
> Continuous corosync kill on master causes switchover and makes another node as master. But as soon as this corosync recovers, it becomes master again. Shouldn't it become slave now?


It is rather unclear what you are asking. Nodes cannot be master or
slave. Do you mean specific master/slave resource in pacemaker
configuration?

>
> 2. With Stonith:
> Sometimes, on corosync kill, that node gets shooted by stonith but sometimes not. Not able to understand this fluctuating behavior. Does it have to do anything with faster recovery of corosync, which stonith fails to detect?

This could be, but logs in both cases may give more hints.

>
> I am using
> corosync-2.4.5-4.el7.x86_64
> pacemaker-1.1.19-8.el7.x86_64
> centos 7.6.1810
>
> Thanks,
> Rohit
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/

From tojeline at redhat.com  Tue Aug 25 04:50:14 2020
From: tojeline at redhat.com (Tomas Jelinek)
Date: Tue, 25 Aug 2020 10:50:14 +0200
Subject: [ClusterLabs] Antw: [EXT] Re: Format of '--lifetime' in 'pcs
 resource move'
In-Reply-To: <5F3F6225020000A10003ABA8@gwsmtp.uni-regensburg.de>
References: <743e6d36-c0b8-5352-5ca9-738628e38239@alteeve.ca>
 <329B5D02-2BCB-4A2C-BC2B-CA3030E6A3B7@yahoo.com>
 <5F3F6225020000A10003ABA8@gwsmtp.uni-regensburg.de>
Message-ID: <8bd5fada-f11f-50e4-94dc-4d6c8e429674@redhat.com>

Hi all,

The lifetime value is indeed expected to be ISO 8601 duration. I updated 
pcs documentation to clarify that:
https://github.com/ClusterLabs/pcs/commit/1e9650a8fd5b8a0a22911ddca1010de582684971

Please note constraints are not removed from CIB when their lifetime 
expires. They are rendered ineffective but still preserved in CIB. See 
the following bugzilla for more details:
https://bugzilla.redhat.com/show_bug.cgi?id=1442116

Regards,
Tomas


Dne 21. 08. 20 v 7:56 Ulrich Windl napsal(a):
>>>> Strahil Nikolov <hunter86_bg at yahoo.com> schrieb am 20.08.2020 um 18:25 in
> Nachricht <329B5D02-2BCB-4A2C-BC2B-CA3030E6A3B7 at yahoo.com>:
>> Have you tried ISO 8601 format.
>> For example: 'PT20M'
> 
> And watch out not to mix Minutes wth Months ;-)
> 
>>
>> The  ISo format  is described at:
>> https://manpages.debian.org/testing/crmsh/crm.8.en.html
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> ?? 20 ?????? 2020 ?. 13:40:16 GMT+03:00, Digimer <lists at alteeve.ca> ??????:
>>> Hi all,
>>>
>>>   Reading the pcs man page for the 'move' action, it talks about
>>> '--lifetime' switch that appears to control when the location
>>> constraint
>>> is removed;
>>>
>>> ====
>>>        move   <resource   id>  [destination  node]  [--master]  [life?
>>>        time=<lifetime>] [--wait[=n]]
>>>               Move the resource off the node it is  currently  running
>>>               on  by  creating  a -INFINITY location constraint to ban
>>>               the node. If destination node is specified the  resource
>>>               will be moved to that node by creating an INFINITY loca?
>>>               tion constraint  to  prefer  the  destination  node.  If
>>>               --master  is used the scope of the command is limited to
>>>               the master role and you must use the promotable clone id
>>>               (instead  of  the resource id). If lifetime is specified
>>>               then the constraint will expire after that time,  other?
>>>               wise  it  defaults to infinity and the constraint can be
>>>               cleared manually with 'pcs resource clear' or 'pcs  con?
>>>               straint  delete'.  If --wait is specified, pcs will wait
>>>               up to 'n' seconds for the  resource  to  move  and  then
>>>               return  0 on success or 1 on error. If 'n' is not speci?
>>>               fied it defaults to 60 minutes. If you want the resource
>>>               to preferably avoid running on some nodes but be able to
>>>               failover to them use 'pcs constraint location avoids'.
>>> ====
>>>
>>> I think I want to use this, as we move resources manually for various
>>> reasons where the old host is still able to host the resource should a
>>> node failure occur. So we'd love to immediately remove the location
>>> constraint as soon as the move completes.
>>>
>>> I tries using '--lifetime=60' as a test, assuming the format was
>>> 'seconds', but that was invalid. How is this switch meant to be used?
>>>
>>> Cheers
>>>
>>> -- 
>>> Digimer
>>> Papers and Projects: https://alteeve.com/w/
>>> "I am, somehow, less interested in the weight and convolutions of
>>> Einstein?s brain than in the near certainty that people of equal talent
>>> have lived and died in cotton fields and sweatshops." - Stephen Jay
>>> Gould
>>> _______________________________________________
>>> Manage your subscription:
>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>> ClusterLabs home: https://www.clusterlabs.org/
>> _______________________________________________
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>
>> ClusterLabs home: https://www.clusterlabs.org/
> 
> 
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> ClusterLabs home: https://www.clusterlabs.org/
> 


From kgaillot at redhat.com  Tue Aug 25 10:36:50 2020
From: kgaillot at redhat.com (Ken Gaillot)
Date: Tue, 25 Aug 2020 09:36:50 -0500
Subject: [ClusterLabs] Behavior of corosync kill
In-Reply-To: <CA+ZsWTWt7t7rNTsRXpzbfUL4bsy6vR6wy0ZQ642fSfubtSjiDQ@mail.gmail.com>
References: <CA+ZsWTWt7t7rNTsRXpzbfUL4bsy6vR6wy0ZQ642fSfubtSjiDQ@mail.gmail.com>
Message-ID: <ac13deedca38e2fccc5e14add9ca19bac4511a2f.camel@redhat.com>

On Tue, 2020-08-25 at 12:28 +0530, Rohit Saini wrote:
> Hi All,
> I am seeing the following behavior. Can someone clarify if this is
> intended behavior. If yes, then why so? Please let me know if logs
> are needed for better clarity.
> 
> 1. Without Stonith:
> Continuous corosync kill on master causes switchover and makes
> another node as master. But as soon as this corosync recovers, it
> becomes master again. Shouldn't it become slave now?

Where resources are active or take on the master role depends on the
cluster configuration, not past node issues.

You may be interested in the resource-stickiness property:

https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/2.0/html-single/Pacemaker_Explained/index.html#_resource_meta_attributes


> 2. With Stonith:
> Sometimes, on corosync kill, that node gets shooted by stonith but
> sometimes not. Not able to understand this fluctuating behavior. Does
> it have to do anything with faster recovery of corosync, which
> stonith fails to detect?

It's not failing to detect it, but recovering satisfactorily without
fencing.

At any given time, one of the cluster nodes is elected the designated
controller (DC). When new events occur, such as a node leaving the
corosync ring unexpectedly, the DC runs pacemaker's scheduler to see
what needs to be done about it. In the case of a lost node, it will
also erase the node's resource history, to indicate that the state of
resources on the node is no longer accurately known.

If no further events happened during that time, the scheduler would
schedule fencing, and the cluster would carry it out.

However, systemd monitors corosync and will restart it if it dies. If
systemd respawns corosync fast enough (it often is sub-second), the
node will rejoin the cluster before the scheduler completes its
calculations and fencing is initiated. Rejoining the cluster includes
re-sync'ing its resource history with the other nodes.

The node join is considered new information, so the former scheduler
run is cancelled (the "transition" is "aborted") and a new one is
started. Since the node is now happily part of the cluster, and the
resource history tells us the state of all resources on the node, no
fencing is needed.


> I am using
> corosync-2.4.5-4.el7.x86_64
> pacemaker-1.1.19-8.el7.x86_64
> centos 7.6.1810
> 
> Thanks,
> Rohit
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> ClusterLabs home: https://www.clusterlabs.org/
-- 
Ken Gaillot <kgaillot at redhat.com>


From rohitsaini111.forum at gmail.com  Tue Aug 25 10:45:47 2020
From: rohitsaini111.forum at gmail.com (Rohit Saini)
Date: Tue, 25 Aug 2020 20:15:47 +0530
Subject: [ClusterLabs] Behavior of corosync kill
In-Reply-To: <ac13deedca38e2fccc5e14add9ca19bac4511a2f.camel@redhat.com>
References: <CA+ZsWTWt7t7rNTsRXpzbfUL4bsy6vR6wy0ZQ642fSfubtSjiDQ@mail.gmail.com>
 <ac13deedca38e2fccc5e14add9ca19bac4511a2f.camel@redhat.com>
Message-ID: <CA+ZsWTWRp9tMd4-2uZ2BTYkOGe=i_Y9G9SJeZ-C0MkYptqh+8A@mail.gmail.com>

Thanks Ken. Let me check resource-stickiness property at my end.

Regards,
Rohit

On Tue, Aug 25, 2020 at 8:07 PM Ken Gaillot <kgaillot at redhat.com> wrote:

> On Tue, 2020-08-25 at 12:28 +0530, Rohit Saini wrote:
> > Hi All,
> > I am seeing the following behavior. Can someone clarify if this is
> > intended behavior. If yes, then why so? Please let me know if logs
> > are needed for better clarity.
> >
> > 1. Without Stonith:
> > Continuous corosync kill on master causes switchover and makes
> > another node as master. But as soon as this corosync recovers, it
> > becomes master again. Shouldn't it become slave now?
>
> Where resources are active or take on the master role depends on the
> cluster configuration, not past node issues.
>
> You may be interested in the resource-stickiness property:
>
>
> https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/2.0/html-single/Pacemaker_Explained/index.html#_resource_meta_attributes
>
>
> > 2. With Stonith:
> > Sometimes, on corosync kill, that node gets shooted by stonith but
> > sometimes not. Not able to understand this fluctuating behavior. Does
> > it have to do anything with faster recovery of corosync, which
> > stonith fails to detect?
>
> It's not failing to detect it, but recovering satisfactorily without
> fencing.
>
> At any given time, one of the cluster nodes is elected the designated
> controller (DC). When new events occur, such as a node leaving the
> corosync ring unexpectedly, the DC runs pacemaker's scheduler to see
> what needs to be done about it. In the case of a lost node, it will
> also erase the node's resource history, to indicate that the state of
> resources on the node is no longer accurately known.
>
> If no further events happened during that time, the scheduler would
> schedule fencing, and the cluster would carry it out.
>
> However, systemd monitors corosync and will restart it if it dies. If
> systemd respawns corosync fast enough (it often is sub-second), the
> node will rejoin the cluster before the scheduler completes its
> calculations and fencing is initiated. Rejoining the cluster includes
> re-sync'ing its resource history with the other nodes.
>
> The node join is considered new information, so the former scheduler
> run is cancelled (the "transition" is "aborted") and a new one is
> started. Since the node is now happily part of the cluster, and the
> resource history tells us the state of all resources on the node, no
> fencing is needed.
>
>
> > I am using
> > corosync-2.4.5-4.el7.x86_64
> > pacemaker-1.1.19-8.el7.x86_64
> > centos 7.6.1810
> >
> > Thanks,
> > Rohit
> > _______________________________________________
> > Manage your subscription:
> > https://lists.clusterlabs.org/mailman/listinfo/users
> >
> > ClusterLabs home: https://www.clusterlabs.org/
> --
> Ken Gaillot <kgaillot at redhat.com>
>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200825/f088de8a/attachment-0001.htm>

From citron_vert at hotmail.com  Wed Aug 26 09:24:04 2020
From: citron_vert at hotmail.com (Citron Vert)
Date: Wed, 26 Aug 2020 15:24:04 +0200
Subject: [ClusterLabs] Resources restart when a node joins in
Message-ID: <DB7PR02MB53710C146E5E8F238EA8F84A88540@DB7PR02MB5371.eurprd02.prod.outlook.com>

Hello,
I am contacting you because I have a problem with my cluster and I 
cannot find (nor understand) any information that can help me.

I have a 2 nodes cluster (pacemaker, corosync, pcs) installed on CentOS 
7 with a set of configuration.
Everything seems to works fine, but here is what happens:

  * Node1 and Node2 are running well with Node1 as primary
  * I reboot Node2 wich is passive (no changes on Node1)
  * Node2 comes back in the cluster as passive
  * corosync logs shows resources getting started then stopped on Node2
  * "crm_mon" command shows some ressources on Node1 getting restarted

I don't understand how it should work.
If a node comes back, and becomes passive (since Node1 is running 
primary), there is no reason for the resources to be started then 
stopped on the new passive node ?

One of my resources becomes unstable because it gets started and then 
stoped too quickly on Node2, wich seems to make it restart on Node1 
without a failover.

I tried several things and solution proposed by different sites and 
forums but without success.


Is there a way so that the node, which joins the cluster as passive, 
does not start its own resources ?


thanks in advance


Here are some information just in case :

$?rpm?-qa?|?grep?-E "corosync|pacemaker|pcs"
corosync-2.4.5-4.el7.x86_64
pacemaker-cli-1.1.21-4.el7.x86_64
pacemaker-1.1.21-4.el7.x86_64
pcs-0.9.168-4.el7.centos.x86_64
corosynclib-2.4.5-4.el7.x86_64
pacemaker-libs-1.1.21-4.el7.x86_64
pacemaker-cluster-libs-1.1.21-4.el7.x86_64


 ????????<nvpair?id="cib-bootstrap-options-stonith-enabled"?name="stonith-enabled"?value="false"/>
 ????????<nvpair?id="cib-bootstrap-options-no-quorum-policy"?name="no-quorum-policy"?value="ignore"/>
 ????????<nvpair?id="cib-bootstrap-options-dc-deadtime"?name="dc-deadtime"?value="120s"/>
 ????????<nvpair?id="cib-bootstrap-options-have-watchdog"?name="have-watchdog"?value="false"/>
 ????????<nvpair?id="cib-bootstrap-options-dc-version"?name="dc-version"?value="1.1.21-4.el7-f14e36fd43"/>
 ????????<nvpair?id="cib-bootstrap-options-cluster-infrastructure"?name="cluster-infrastructure"?value="corosync"/>
 ????????<nvpair?id="cib-bootstrap-options-cluster-name"?name="cluster-name"?value="CLUSTER"/>
 ????????<nvpair?id="cib-bootstrap-options-last-lrm-refresh"?name="last-lrm-refresh"?value="1598446314"/>
 ????????<nvpair?id="cib-bootstrap-options-default-resource-stickiness"?name="default-resource-stickiness"?value="0"/>




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200826/6d66ab1c/attachment.htm>

From nwahl at redhat.com  Wed Aug 26 14:16:53 2020
From: nwahl at redhat.com (Reid Wahl)
Date: Wed, 26 Aug 2020 11:16:53 -0700
Subject: [ClusterLabs] Resources restart when a node joins in
In-Reply-To: <DB7PR02MB53710C146E5E8F238EA8F84A88540@DB7PR02MB5371.eurprd02.prod.outlook.com>
References: <DB7PR02MB53710C146E5E8F238EA8F84A88540@DB7PR02MB5371.eurprd02.prod.outlook.com>
Message-ID: <CAPiuu9-RW5AzyKhzNBOcMZhVuAdzCY1nvK9mJmpvzEFCbpW47w@mail.gmail.com>

Hi, Citron.

Based on your description, it sounds like some resources **might** be
moving from node 1 to node 2, failing on node 2, and then moving back to
node 1. If that's what's happening (and even if it's not), then it's
probably smart to set some resource stickiness as a resource default. The
below command sets a resource stickiness score of 1.

    # pcs resource defaults resource-stickiness=1

Also note that the "default-resource-stickiness" cluster property is
deprecated and should not be used.

Finally, an explicit default resource stickiness score of 0 can interfere
with the placement of cloned resource instances. If you don't want any
stickiness, then it's better to leave stickiness unset. That way,
primitives will have a stickiness of 0, but clone instances will have a
stickiness of 1.

If adding stickiness does not resolve the issue, can you share your cluster
configuration and some logs that show the issue happening? Off the top of
my head I'm not sure why resources would start and stop on node 2 without
moving away from node1, unless they're clone instances that are starting
and then failing a monitor operation on node 2.

On Wed, Aug 26, 2020 at 8:42 AM Citron Vert <citron_vert at hotmail.com> wrote:

> Hello,
> I am contacting you because I have a problem with my cluster and I cannot
> find (nor understand) any information that can help me.
>
> I have a 2 nodes cluster (pacemaker, corosync, pcs) installed on CentOS 7
> with a set of configuration.
> Everything seems to works fine, but here is what happens:
>
>    - Node1 and Node2 are running well with Node1 as primary
>    - I reboot Node2 wich is passive (no changes on Node1)
>    - Node2 comes back in the cluster as passive
>    - corosync logs shows resources getting started then stopped on Node2
>    - "crm_mon" command shows some ressources on Node1 getting restarted
>
> I don't understand how it should work.
> If a node comes back, and becomes passive (since Node1 is running
> primary), there is no reason for the resources to be started then stopped
> on the new passive node ?
>
> One of my resources becomes unstable because it gets started and then
> stoped too quickly on Node2, wich seems to make it restart on Node1 without
> a failover.
>
> I tried several things and solution proposed by different sites and forums
> but without success.
>
>
> Is there a way so that the node, which joins the cluster as passive, does
> not start its own resources ?
>
>
> thanks in advance
>
>
> Here are some information just in case :
> $ rpm -qa | grep -E "corosync|pacemaker|pcs"
> corosync-2.4.5-4.el7.x86_64
> pacemaker-cli-1.1.21-4.el7.x86_64
> pacemaker-1.1.21-4.el7.x86_64
> pcs-0.9.168-4.el7.centos.x86_64
> corosynclib-2.4.5-4.el7.x86_64
> pacemaker-libs-1.1.21-4.el7.x86_64
> pacemaker-cluster-libs-1.1.21-4.el7.x86_64
>
>
>         <nvpair id="cib-bootstrap-options-stonith-enabled" name=
> "stonith-enabled" value="false"/>
>         <nvpair id="cib-bootstrap-options-no-quorum-policy" name=
> "no-quorum-policy" value="ignore"/>
>         <nvpair id="cib-bootstrap-options-dc-deadtime" name="dc-deadtime"
>  value="120s"/>
>         <nvpair id="cib-bootstrap-options-have-watchdog" name=
> "have-watchdog" value="false"/>
>         <nvpair id="cib-bootstrap-options-dc-version" name="dc-version"
>  value="1.1.21-4.el7-f14e36fd43"/>
>         <nvpair id="cib-bootstrap-options-cluster-infrastructure" name=
> "cluster-infrastructure" value="corosync"/>
>         <nvpair id="cib-bootstrap-options-cluster-name" name=
> "cluster-name" value="CLUSTER"/>
>         <nvpair id="cib-bootstrap-options-last-lrm-refresh" name=
> "last-lrm-refresh" value="1598446314"/>
>         <nvpair id="cib-bootstrap-options-default-resource-stickiness"
>  name="default-resource-stickiness" value="0"/>
>
>
>
>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>


-- 
Regards,

Reid Wahl, RHCA
Software Maintenance Engineer, Red Hat
CEE - Platform Support Delivery - ClusterHA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200826/735792f3/attachment.htm>

From arvidjaar at gmail.com  Thu Aug 27 01:34:58 2020
From: arvidjaar at gmail.com (Andrei Borzenkov)
Date: Thu, 27 Aug 2020 08:34:58 +0300
Subject: [ClusterLabs] [ClusterLabs Developers] Fencing with a Quorum
 Device
In-Reply-To: <CA+tJmF=rS7Cc9WHBwQkKxsp9hAb0kX0ysRmxjY5X8WY8TSMEEA@mail.gmail.com>
References: <CA+tJmF=rS7Cc9WHBwQkKxsp9hAb0kX0ysRmxjY5X8WY8TSMEEA@mail.gmail.com>
Message-ID: <2dedd93d-3ebd-8aba-4049-0f06aef6bcd6@gmail.com>

I changed list to users because it is general usage question, not
development topic.

26.08.2020 23:33, Hayden Pfeiffer ?????:
> Hello,
> 
> 
> I am in the process of configuring fencing in an AWS cluster of two
> hosts. I have done so and nodes are correctly fenced when
> communication is broken with ifconfig down tests.
> 

It is highly recommended to test split brain by blocking communication
between nodes (either on external LAN equipment or e.g. using iptables
on host). Setting interface down does not always do what one expects
with corosync.

> 
> However, I would also like to use a quorum device in my setup. I have

Why? What exactly do you need it for?

> configured a third host to act as a tiebreaker, but when I try to
> break communication on one of the two pacemaker hosts, the affected
> host is marked as "unclean" and is not fenced.
> 

It is unclear what you did. Do you corosync-qdevice? Post actual
configuration (corosync.conf and pacemaker config) and your
corosync/pacemaker versions.

>  Is it not recommended to use fencing devices and a quorum device in
> the same cluster? I cannot find any documentation relating to this
> issue. Thanks,
> 

corosync-qdevice just affects quorate state of individual nodes. What
happens when node goes out of quorum is determined entirely by pacemaker
configuration. For a long time I have been using two node cluster with
no-quorum-policy=ignore where corosync-device is entirely useless.


From nwahl at redhat.com  Thu Aug 27 03:56:29 2020
From: nwahl at redhat.com (Reid Wahl)
Date: Thu, 27 Aug 2020 00:56:29 -0700
Subject: [ClusterLabs] Resources restart when a node joins in
In-Reply-To: <DB7PR02MB53716B42197E31D8D15CA0A288550@DB7PR02MB5371.eurprd02.prod.outlook.com>
References: <DB7PR02MB53710C146E5E8F238EA8F84A88540@DB7PR02MB5371.eurprd02.prod.outlook.com>
 <CAPiuu9-RW5AzyKhzNBOcMZhVuAdzCY1nvK9mJmpvzEFCbpW47w@mail.gmail.com>
 <DB7PR02MB53716B42197E31D8D15CA0A288550@DB7PR02MB5371.eurprd02.prod.outlook.com>
Message-ID: <CAPiuu9-smjtSws3NnqT7k6teQ0mZWFxDadbQP5OYcQ9fvwz04g@mail.gmail.com>

Hi, Quentin. Thanks for the logs!

I see you highlighted the fact that SERVICE1 was in "Stopping" state on
both node 1 and node 2 when node 1 was rejoining the cluster. I also noted
the following later in the logs, as well as some similar messages earlier:

Aug 27 08:47:02 [1330] NODE2    pengine:     info:
determine_op_status:       Operation monitor found resource SERVICE1
active on NODE1
Aug 27 08:47:02 [1330] NODE2    pengine:     info:
determine_op_status:       Operation monitor found resource SERVICE1
active on NODE1
Aug 27 08:47:02 [1330] NODE2    pengine:     info:
determine_op_status:       Operation monitor found resource SERVICE4
active on NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info:
determine_op_status:       Operation monitor found resource SERVICE1
active on NODE2
...
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:
       1 : NODE1
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:
       2 : NODE2
...
Aug 27 08:47:02 [1330] NODE2    pengine:    error:
native_create_actions:     Resource SERVICE1 is active on 2 nodes
(attempting recovery)
Aug 27 08:47:02 [1330] NODE2    pengine:   notice:
native_create_actions:     See
https://wiki.clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more
information

Can you make sure that all the cluster-managed systemd services are
disabled from starting at boot (i.e., `systemctl is-enabled service1`,
and the same for all the others) on both nodes? If they are enabled,
disable them.


On Thu, Aug 27, 2020 at 12:46 AM Citron Vert <citron_vert at hotmail.com>
wrote:

> Hi,
>
> Sorry for using this email adress, my name is Quentin. Thank you for your
> reply.
>
> I have already tried the stickiness solution (with the deprecated  value).
> I tried the one you gave me, and it does not change anything.
>
> Resources don't seem to move from node to node (i don't see the changes
> with crm_mon command).
>
>
> In the logs i found this line *"error: native_create_actions:
> Resource SERVICE1 is active on 2 nodes*"
>
> Which led me to contact you to understand and learn a little more about
> this cluster. And why there are running resources on the passive node.
>
>
> You will find attached the logs during the reboot of the passive node and
> my cluster configuration.
>
> I think I'm missing out on something in the configuration / logs that I
> don't understand..
>
>
> Thank you in advance for your help,
>
> Quentin
>
>
> Le 26/08/2020 ? 20:16, Reid Wahl a ?crit :
>
> Hi, Citron.
>
> Based on your description, it sounds like some resources **might** be
> moving from node 1 to node 2, failing on node 2, and then moving back to
> node 1. If that's what's happening (and even if it's not), then it's
> probably smart to set some resource stickiness as a resource default. The
> below command sets a resource stickiness score of 1.
>
>     # pcs resource defaults resource-stickiness=1
>
> Also note that the "default-resource-stickiness" cluster property is
> deprecated and should not be used.
>
> Finally, an explicit default resource stickiness score of 0 can interfere
> with the placement of cloned resource instances. If you don't want any
> stickiness, then it's better to leave stickiness unset. That way,
> primitives will have a stickiness of 0, but clone instances will have a
> stickiness of 1.
>
> If adding stickiness does not resolve the issue, can you share your
> cluster configuration and some logs that show the issue happening? Off the
> top of my head I'm not sure why resources would start and stop on node 2
> without moving away from node1, unless they're clone instances that are
> starting and then failing a monitor operation on node 2.
>
> On Wed, Aug 26, 2020 at 8:42 AM Citron Vert <citron_vert at hotmail.com>
> wrote:
>
>> Hello,
>> I am contacting you because I have a problem with my cluster and I cannot
>> find (nor understand) any information that can help me.
>>
>> I have a 2 nodes cluster (pacemaker, corosync, pcs) installed on CentOS 7
>> with a set of configuration.
>> Everything seems to works fine, but here is what happens:
>>
>>    - Node1 and Node2 are running well with Node1 as primary
>>    - I reboot Node2 wich is passive (no changes on Node1)
>>    - Node2 comes back in the cluster as passive
>>    - corosync logs shows resources getting started then stopped on Node2
>>    - "crm_mon" command shows some ressources on Node1 getting restarted
>>
>> I don't understand how it should work.
>> If a node comes back, and becomes passive (since Node1 is running
>> primary), there is no reason for the resources to be started then stopped
>> on the new passive node ?
>>
>> One of my resources becomes unstable because it gets started and then
>> stoped too quickly on Node2, wich seems to make it restart on Node1 without
>> a failover.
>>
>> I tried several things and solution proposed by different sites and
>> forums but without success.
>>
>>
>> Is there a way so that the node, which joins the cluster as passive, does
>> not start its own resources ?
>>
>>
>> thanks in advance
>>
>>
>> Here are some information just in case :
>> $ rpm -qa | grep -E "corosync|pacemaker|pcs"
>> corosync-2.4.5-4.el7.x86_64
>> pacemaker-cli-1.1.21-4.el7.x86_64
>> pacemaker-1.1.21-4.el7.x86_64
>> pcs-0.9.168-4.el7.centos.x86_64
>> corosynclib-2.4.5-4.el7.x86_64
>> pacemaker-libs-1.1.21-4.el7.x86_64
>> pacemaker-cluster-libs-1.1.21-4.el7.x86_64
>>
>>
>>         <nvpair id="cib-bootstrap-options-stonith-enabled" name=
>> "stonith-enabled" value="false"/>
>>         <nvpair id="cib-bootstrap-options-no-quorum-policy" name=
>> "no-quorum-policy" value="ignore"/>
>>         <nvpair id="cib-bootstrap-options-dc-deadtime" name="dc-deadtime"
>>  value="120s"/>
>>         <nvpair id="cib-bootstrap-options-have-watchdog" name=
>> "have-watchdog" value="false"/>
>>         <nvpair id="cib-bootstrap-options-dc-version" name="dc-version"
>>  value="1.1.21-4.el7-f14e36fd43"/>
>>         <nvpair id="cib-bootstrap-options-cluster-infrastructure" name=
>> "cluster-infrastructure" value="corosync"/>
>>         <nvpair id="cib-bootstrap-options-cluster-name" name=
>> "cluster-name" value="CLUSTER"/>
>>         <nvpair id="cib-bootstrap-options-last-lrm-refresh" name=
>> "last-lrm-refresh" value="1598446314"/>
>>         <nvpair id="cib-bootstrap-options-default-resource-stickiness"
>>  name="default-resource-stickiness" value="0"/>
>>
>>
>>
>>
>> _______________________________________________
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>
>> ClusterLabs home: https://www.clusterlabs.org/
>>
>
>
> --
> Regards,
>
> Reid Wahl, RHCA
> Software Maintenance Engineer, Red Hat
> CEE - Platform Support Delivery - ClusterHA
>
>

-- 
Regards,

Reid Wahl, RHCA
Software Maintenance Engineer, Red Hat
CEE - Platform Support Delivery - ClusterHA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200827/12f133d5/attachment-0001.htm>

From citron_vert at hotmail.com  Thu Aug 27 03:46:14 2020
From: citron_vert at hotmail.com (Citron Vert)
Date: Thu, 27 Aug 2020 09:46:14 +0200
Subject: [ClusterLabs] Resources restart when a node joins in
In-Reply-To: <CAPiuu9-RW5AzyKhzNBOcMZhVuAdzCY1nvK9mJmpvzEFCbpW47w@mail.gmail.com>
References: <DB7PR02MB53710C146E5E8F238EA8F84A88540@DB7PR02MB5371.eurprd02.prod.outlook.com>
 <CAPiuu9-RW5AzyKhzNBOcMZhVuAdzCY1nvK9mJmpvzEFCbpW47w@mail.gmail.com>
Message-ID: <DB7PR02MB53716B42197E31D8D15CA0A288550@DB7PR02MB5371.eurprd02.prod.outlook.com>

Hi,

Sorry for using this email adress, my name is Quentin. Thank you for 
your reply.

I have already tried the stickiness solution (with the deprecated? 
value). I tried the one you gave me, and it does not change anything.

Resources don't seem to move from node to node (i don't see the changes 
with crm_mon command).


In the logs i found this line /"error: native_create_actions:???? 
Resource SERVICE1 is active on 2 nodes/"

Which led me to contact you to understand and learn a little more about 
this cluster. And why there are running resources on the passive node.


You will find attached the logs during the reboot of the passive node 
and my cluster configuration.

I think I'm missing out on something in the configuration / logs that I 
don't understand..


Thank you in advance for your help,

Quentin


Le 26/08/2020 ? 20:16, Reid Wahl a ?crit?:
> Hi, Citron.
>
> Based on your description, it sounds like some resources **might** be 
> moving from node 1 to node 2, failing on node 2, and then moving back 
> to node 1. If that's what's happening (and even if it's not), then 
> it's probably smart to set some resource stickiness as a resource 
> default. The below command sets a resource stickiness score of 1.
>
> ??? # pcs resource defaults resource-stickiness=1
>
> Also note that the "default-resource-stickiness" cluster property is 
> deprecated and should not be used.
>
> Finally, an explicit default resource stickiness score of 0 can 
> interfere with the placement of cloned resource instances. If you 
> don't want any stickiness, then it's better to leave stickiness unset. 
> That way, primitives will have a stickiness of 0, but clone instances 
> will have a stickiness of 1.
>
> If adding stickiness does not resolve the issue, can you share your 
> cluster configuration and some logs that show the issue happening? Off 
> the top of my head I'm not sure why resources would start and stop on 
> node 2 without moving away from node1, unless they're clone instances 
> that are starting and then failing a monitor operation on node 2.
>
> On Wed, Aug 26, 2020 at 8:42 AM Citron Vert <citron_vert at hotmail.com 
> <mailto:citron_vert at hotmail.com>> wrote:
>
>     Hello,
>     I am contacting you because I have a problem with my cluster and I
>     cannot find (nor understand) any information that can help me.
>
>     I have a 2 nodes cluster (pacemaker, corosync, pcs) installed on
>     CentOS 7 with a set of configuration.
>     Everything seems to works fine, but here is what happens:
>
>       * Node1 and Node2 are running well with Node1 as primary
>       * I reboot Node2 wich is passive (no changes on Node1)
>       * Node2 comes back in the cluster as passive
>       * corosync logs shows resources getting started then stopped on
>         Node2
>       * "crm_mon" command shows some ressources on Node1 getting
>         restarted
>
>     I don't understand how it should work.
>     If a node comes back, and becomes passive (since Node1 is running
>     primary), there is no reason for the resources to be started then
>     stopped on the new passive node ?
>
>     One of my resources becomes unstable because it gets started and
>     then stoped too quickly on Node2, wich seems to make it restart on
>     Node1 without a failover.
>
>     I tried several things and solution proposed by different sites
>     and forums but without success.
>
>
>     Is there a way so that the node, which joins the cluster as
>     passive, does not start its own resources ?
>
>
>     thanks in advance
>
>
>     Here are some information just in case :
>
>     $?rpm?-qa?|?grep?-E "corosync|pacemaker|pcs"
>     corosync-2.4.5-4.el7.x86_64
>     pacemaker-cli-1.1.21-4.el7.x86_64
>     pacemaker-1.1.21-4.el7.x86_64
>     pcs-0.9.168-4.el7.centos.x86_64
>     corosynclib-2.4.5-4.el7.x86_64
>     pacemaker-libs-1.1.21-4.el7.x86_64
>     pacemaker-cluster-libs-1.1.21-4.el7.x86_64
>
>
>     ????????<nvpair?id="cib-bootstrap-options-stonith-enabled"?name="stonith-enabled"?value="false"/>
>     ????????<nvpair?id="cib-bootstrap-options-no-quorum-policy"?name="no-quorum-policy"?value="ignore"/>
>     ????????<nvpair?id="cib-bootstrap-options-dc-deadtime"?name="dc-deadtime"?value="120s"/>
>     ????????<nvpair?id="cib-bootstrap-options-have-watchdog"?name="have-watchdog"?value="false"/>
>     ????????<nvpair?id="cib-bootstrap-options-dc-version"?name="dc-version"?value="1.1.21-4.el7-f14e36fd43"/>
>     ????????<nvpair?id="cib-bootstrap-options-cluster-infrastructure"?name="cluster-infrastructure"?value="corosync"/>
>     ????????<nvpair?id="cib-bootstrap-options-cluster-name"?name="cluster-name"?value="CLUSTER"/>
>     ????????<nvpair?id="cib-bootstrap-options-last-lrm-refresh"?name="last-lrm-refresh"?value="1598446314"/>
>     ????????<nvpair?id="cib-bootstrap-options-default-resource-stickiness"?name="default-resource-stickiness"?value="0"/>
>
>
>
>
>     _______________________________________________
>     Manage your subscription:
>     https://lists.clusterlabs.org/mailman/listinfo/users
>
>     ClusterLabs home: https://www.clusterlabs.org/
>
>
>
> -- 
> Regards,
>
> Reid Wahl, RHCA
> Software Maintenance Engineer, Red Hat
> CEE - Platform Support Delivery - ClusterHA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200827/de555ff8/attachment-0001.htm>
-------------- next part --------------
$ pcs cluster cib
<cib crm_feature_set="3.0.14" validate-with="pacemaker-2.10" epoch="451" num_updates="2" admin_epoch="0" cib-last-written="Thu Aug 27 08:48:57 2020" update-origin="NODE1" update-client="crmd" update-user="hacluster" have-quorum="1" dc-uuid="2">
  <configuration>
    <crm_config>
      <cluster_property_set id="cib-bootstrap-options">
        <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="false"/>
        <nvpair id="cib-bootstrap-options-no-quorum-policy" name="no-quorum-policy" value="ignore"/>
        <nvpair id="cib-bootstrap-options-dc-deadtime" name="dc-deadtime" value="120s"/>
        <nvpair id="cib-bootstrap-options-have-watchdog" name="have-watchdog" value="false"/>
        <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.21-4.el7-f14e36fd43"/>
        <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/>
        <nvpair id="cib-bootstrap-options-cluster-name" name="cluster-name" value="CLUSTER"/>
        <nvpair id="cib-bootstrap-options-last-lrm-refresh" name="last-lrm-refresh" value="1598510960"/>
      </cluster_property_set>
    </crm_config>
    <nodes>
      <node id="1" uname="NODE1">
        <instance_attributes id="nodes-1"/>
      </node>
      <node id="2" uname="NODE2">
        <instance_attributes id="nodes-2"/>
      </node>
    </nodes>
    <resources>
      <group id="IPV">
        <primitive class="ocf" id="VIRTUALIP" provider="heartbeat" type="IPaddr2">
          <instance_attributes id="VIRTUALIP-instance_attributes">
            <nvpair id="VIRTUALIP-instance_attributes-cidr_netmask" name="cidr_netmask" value="16"/>
            <nvpair id="VIRTUALIP-instance_attributes-ip" name="ip" value="10.130.0.3"/>
          </instance_attributes>
          <operations>
            <op id="VIRTUALIP-monitor-interval-5s" interval="5s" name="monitor"/>
            <op id="VIRTUALIP-start-interval-0s" interval="0s" name="start" timeout="20s"/>
            <op id="VIRTUALIP-stop-interval-0s" interval="0s" name="stop" timeout="20s"/>
          </operations>
        </primitive>
        <primitive class="ocf" id="SOURCEIP" provider="heartbeat" type="IPsrcaddr">
          <instance_attributes id="SOURCEIP-instance_attributes">
            <nvpair id="SOURCEIP-instance_attributes-cidr_netmask" name="cidr_netmask" value="16"/>
            <nvpair id="SOURCEIP-instance_attributes-ipaddress" name="ipaddress" value="10.130.0.3"/>
          </instance_attributes>
          <operations>
            <op id="SOURCEIP-monitor-interval-5s" interval="5s" name="monitor"/>
            <op id="SOURCEIP-start-interval-0s" interval="0s" name="start" timeout="20s"/>
            <op id="SOURCEIP-stop-interval-0s" interval="0s" name="stop" timeout="20s"/>
          </operations>
        </primitive>
      </group>
      <primitive class="systemd" id="SERVICE1" type="service1">
        <operations>
          <op id="SERVICE1-monitor-interval-5s" interval="5s" name="monitor"/>
          <op id="SERVICE1-start-interval-0s" interval="0s" name="start" timeout="20s"/>
          <op id="SERVICE1-stop-interval-0s" interval="0s" name="stop" on-fail="ignore" timeout="20s"/>
        </operations>
      </primitive>
      <primitive class="systemd" id="SERVICE2" type="service2">
        <operations>
          <op id="SERVICE2-monitor-interval-5s" interval="5s" name="monitor"/>
          <op id="SERVICE2-start-interval-0s" interval="0s" name="start" timeout="20s"/>
          <op id="SERVICE2-stop-interval-0s" interval="0s" name="stop" on-fail="ignore" timeout="20s"/>
        </operations>
        <meta_attributes id="SERVICE2-meta_attributes"/>
      </primitive>
      <primitive class="systemd" id="SERVICE3" type="service3">
        <operations>
          <op id="SERVICE3-monitor-interval-5s" interval="5s" name="monitor"/>
          <op id="SERVICE3-start-interval-0s" interval="0s" name="start" timeout="20s"/>
          <op id="SERVICE3-stop-interval-0s" interval="0s" name="stop" on-fail="ignore" timeout="20s"/>
        </operations>
      </primitive>
      <primitive class="systemd" id="SERVICE4" type="service4">
        <operations>
          <op id="SERVICE4-monitor-interval-5s" interval="5s" name="monitor"/>
          <op id="SERVICE4-start-interval-0s" interval="0s" name="start" timeout="20s"/>
          <op id="SERVICE4-stop-interval-0s" interval="0s" name="stop" on-fail="ignore" timeout="20s"/>
        </operations>
        <meta_attributes id="SERVICE4-meta_attributes"/>
      </primitive>
      <primitive class="systemd" id="SERVICE5" type="service5">
        <operations>
          <op id="SERVICE5-monitor-interval-5s" interval="5s" name="monitor"/>
          <op id="SERVICE5-start-interval-0s" interval="0s" name="start" timeout="20s"/>
          <op id="SERVICE5-stop-interval-0s" interval="0s" name="stop" on-fail="ignore" timeout="20s"/>
        </operations>
      </primitive>
      <primitive class="systemd" id="SERVICE6" type="service6">
        <operations>
          <op id="SERVICE6-monitor-interval-5s" interval="5s" name="monitor"/>
          <op id="SERVICE6-start-interval-0s" interval="0s" name="start" timeout="20s"/>
          <op id="SERVICE6-stop-interval-0s" interval="0s" name="stop" on-fail="ignore" timeout="20s"/>
        </operations>
        <meta_attributes id="SERVICE6-meta_attributes"/>
      </primitive>
      <primitive class="systemd" id="SERVICE7" type="service7">
        <operations>
          <op id="SERVICE7-monitor-interval-5s" interval="5s" name="monitor"/>
          <op id="SERVICE7-start-interval-0s" interval="0s" name="start" timeout="20s"/>
          <op id="SERVICE7-stop-interval-0s" interval="0s" name="stop" on-fail="ignore" timeout="20s"/>
        </operations>
      </primitive>
      <primitive class="systemd" id="SERVICE8" type="service8">
        <operations>
          <op id="SERVICE8-monitor-interval-5s" interval="5s" name="monitor"/>
          <op id="SERVICE8-start-interval-0s" interval="0s" name="start" timeout="20s"/>
          <op id="SERVICE8-stop-interval-0s" interval="0s" name="stop" on-fail="ignore" timeout="20s"/>
        </operations>
      </primitive>
      <primitive class="systemd" id="SERVICE9" type="service9">
        <operations>
          <op id="SERVICE9-monitor-interval-5s" interval="5s" name="monitor"/>
          <op id="SERVICE9-start-interval-0s" interval="0s" name="start" timeout="20s"/>
          <op id="SERVICE9-stop-interval-0s" interval="0s" name="stop" on-fail="ignore" timeout="20s"/>
        </operations>
      </primitive>
      <primitive class="systemd" id="SERVICE10" type="service10">
        <operations>
          <op id="SERVICE10-monitor-interval-5s" interval="5s" name="monitor"/>
          <op id="SERVICE10-start-interval-0s" interval="0s" name="start" timeout="20s"/>
          <op id="SERVICE10-stop-interval-0s" interval="0s" name="stop" on-fail="ignore" timeout="20s"/>
        </operations>
      </primitive>
      <primitive class="systemd" id="SERVICE11" type="service11">
        <operations>
          <op id="SERVICE11-monitor-interval-5s" interval="5s" name="monitor"/>
          <op id="SERVICE11-start-interval-0s" interval="0s" name="start" timeout="20s"/>
          <op id="SERVICE11-stop-interval-0s" interval="0s" name="stop" on-fail="ignore" timeout="20s"/>
        </operations>
        <meta_attributes id="SERVICE11-meta_attributes"/>
      </primitive>
      <primitive class="systemd" id="SERVICE12" type="service12">
        <operations>
          <op id="SERVICE12-monitor-interval-5s" interval="5s" name="monitor"/>
          <op id="SERVICE12-start-interval-0s" interval="0s" name="start" timeout="20s"/>
          <op id="SERVICE12-stop-interval-0s" interval="0s" name="stop" on-fail="ignore" timeout="20s"/>
        </operations>
      </primitive>
      <primitive class="systemd" id="SERVICE13" type="service13">
        <operations>
          <op id="SERVICE13-monitor-interval-5s" interval="5s" name="monitor"/>
          <op id="SERVICE13-start-interval-0s" interval="0s" name="start" timeout="20s"/>
          <op id="SERVICE13-stop-interval-0s" interval="0s" name="stop" on-fail="ignore" timeout="20s"/>
        </operations>
      </primitive>
      <clone id="SERVICE14-clone">
        <primitive class="systemd" id="SERVICE14" type="service14">
          <operations>
            <op id="SERVICE14-monitor-interval-10s" interval="10s" name="monitor" timeout="60s"/>
            <op id="SERVICE14-start-interval-0s" interval="0s" name="start" timeout="60s"/>
            <op id="SERVICE14-stop-interval-0s" interval="0s" name="stop" timeout="60s"/>
          </operations>
        </primitive>
      </clone>
      <clone id="SERVICE15-clone">
        <primitive class="systemd" id="SERVICE15" type="service15">
          <operations>
            <op id="SERVICE15-monitor-interval-10s" interval="10s" name="monitor" timeout="60s"/>
            <op id="SERVICE15-start-interval-0s" interval="0s" name="start" timeout="60s"/>
            <op id="SERVICE15-stop-interval-0s" interval="0s" name="stop" timeout="60s"/>
          </operations>
        </primitive>
        <meta_attributes id="SERVICE15-clone-meta_attributes"/>
      </clone>
    </resources>
    <constraints>
      <rsc_colocation id="colocation-SERVICE1-VIRTUALIP-INFINITY" rsc="SERVICE1" score="INFINITY" with-rsc="VIRTUALIP"/>
      <rsc_colocation id="colocation-SERVICE2-SERVICE1-INFINITY" rsc="SERVICE2" score="INFINITY" with-rsc="SERVICE1"/>
      <rsc_colocation id="colocation-SERVICE3-SERVICE1-INFINITY" rsc="SERVICE3" score="INFINITY" with-rsc="SERVICE1"/>
      <rsc_colocation id="colocation-SERVICE4-SERVICE1-INFINITY" rsc="SERVICE4" score="INFINITY" with-rsc="SERVICE1"/>
      <rsc_colocation id="colocation-SERVICE5-VIRTUALIP-INFINITY" rsc="SERVICE5" score="INFINITY" with-rsc="VIRTUALIP"/>
      <rsc_colocation id="colocation-SERVICE6-VIRTUALIP-INFINITY" rsc="SERVICE6" score="INFINITY" with-rsc="VIRTUALIP"/>
      <rsc_colocation id="colocation-SERVICE7-VIRTUALIP-INFINITY" rsc="SERVICE7" score="INFINITY" with-rsc="VIRTUALIP"/>
      <rsc_colocation id="colocation-SERVICE8-VIRTUALIP-INFINITY" rsc="SERVICE8" score="INFINITY" with-rsc="VIRTUALIP"/>
      <rsc_colocation id="colocation-SERVICE9-VIRTUALIP-INFINITY" rsc="SERVICE9" score="INFINITY" with-rsc="VIRTUALIP"/>
      <rsc_colocation id="colocation-SERVICE10-VIRTUALIP-INFINITY" rsc="SERVICE10" score="INFINITY" with-rsc="VIRTUALIP"/>
      <rsc_colocation id="colocation-SERVICE11-VIRTUALIP-INFINITY" rsc="SERVICE11" score="INFINITY" with-rsc="VIRTUALIP"/>
      <rsc_colocation id="colocation-SERVICE12-VIRTUALIP-INFINITY" rsc="SERVICE12" score="INFINITY" with-rsc="VIRTUALIP"/>
      <rsc_colocation id="colocation-SERVICE13-VIRTUALIP-INFINITY" rsc="SERVICE13" score="INFINITY" with-rsc="VIRTUALIP"/>
    </constraints>
    <rsc_defaults>
      <meta_attributes id="rsc_defaults-options">
        <nvpair id="rsc_defaults-options-migration-threshold" name="migration-threshold" value="1"/>
        <nvpair id="rsc_defaults-options-failure-timeout" name="failure-timeout" value="30s"/>
        <nvpair id="rsc_defaults-options-resource-stickiness" name="resource-stickiness" value="1"/>
      </meta_attributes>
    </rsc_defaults>
  </configuration>
  <status>
    <node_state id="1" uname="NODE1" in_ccm="true" crmd="online" crm-debug-origin="do_update_resource" join="member" expected="member">
      <lrm id="1">
        <lrm_resources>
          <lrm_resource id="VIRTUALIP" type="IPaddr2" class="ocf" provider="heartbeat">
            <lrm_rsc_op id="VIRTUALIP_last_0" operation_key="VIRTUALIP_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="18:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:7;18:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="5" rc-code="7" op-status="0" interval="0" last-run="1598510843" last-rc-change="1598510843" exec-time="441" queue-time="0" op-digest="81452beaabf19c71189916d97c5ea2a8"/>
          </lrm_resource>
          <lrm_resource id="SOURCEIP" type="IPsrcaddr" class="ocf" provider="heartbeat">
            <lrm_rsc_op id="SOURCEIP_last_0" operation_key="SOURCEIP_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="19:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:7;19:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="9" rc-code="7" op-status="0" interval="0" last-run="1598510844" last-rc-change="1598510844" exec-time="40" queue-time="0" op-digest="4238bbb83fc1ef2f001ab06eb7d14fe0"/>
          </lrm_resource>
          <lrm_resource id="SERVICE2" type="service2" class="systemd">
            <lrm_rsc_op id="SERVICE2_last_0" operation_key="SERVICE2_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="21:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:7;21:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="17" rc-code="7" op-status="0" interval="0" last-run="1598510844" last-rc-change="1598510844" exec-time="239" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
          </lrm_resource>
          <lrm_resource id="SERVICE3" type="service3" class="systemd">
            <lrm_rsc_op id="SERVICE3_last_0" operation_key="SERVICE3_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="22:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:7;22:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="21" rc-code="7" op-status="0" interval="0" last-run="1598510844" last-rc-change="1598510844" exec-time="240" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
          </lrm_resource>
          <lrm_resource id="SERVICE4" type="service4" class="systemd">
            <lrm_rsc_op id="SERVICE4_last_0" operation_key="SERVICE4_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="23:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:7;23:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="25" rc-code="7" op-status="0" interval="0" last-run="1598510844" last-rc-change="1598510844" exec-time="241" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
          </lrm_resource>
          <lrm_resource id="SERVICE5" type="service5" class="systemd">
            <lrm_rsc_op id="SERVICE5_last_0" operation_key="SERVICE5_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="24:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:7;24:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="29" rc-code="7" op-status="0" interval="0" last-run="1598510844" last-rc-change="1598510844" exec-time="241" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
          </lrm_resource>
          <lrm_resource id="SERVICE6" type="service6" class="systemd">
            <lrm_rsc_op id="SERVICE6_last_0" operation_key="SERVICE6_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="25:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:7;25:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="33" rc-code="7" op-status="0" interval="0" last-run="1598510844" last-rc-change="1598510844" exec-time="242" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
          </lrm_resource>
          <lrm_resource id="SERVICE7" type="service7" class="systemd">
            <lrm_rsc_op id="SERVICE7_last_0" operation_key="SERVICE7_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="18:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:7;18:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="38" rc-code="7" op-status="0" interval="0" last-run="1598510844" last-rc-change="1598510844" exec-time="125" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
          </lrm_resource>
          <lrm_resource id="SERVICE8" type="service8" class="systemd">
            <lrm_rsc_op id="SERVICE8_last_0" operation_key="SERVICE8_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="19:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:7;19:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="42" rc-code="7" op-status="0" interval="0" last-run="1598510844" last-rc-change="1598510844" exec-time="123" queue-time="1" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
          </lrm_resource>
          <lrm_resource id="SERVICE9" type="service9" class="systemd">
            <lrm_rsc_op id="SERVICE9_last_0" operation_key="SERVICE9_stop_0" operation="stop" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="42:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;42:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="68" rc-code="0" op-status="0" interval="0" last-run="1598510864" last-rc-change="1598510864" exec-time="2006" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE9_last_failure_0" operation_key="SERVICE9_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="20:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;20:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="46" rc-code="0" op-status="0" interval="0" last-run="1598510844" last-rc-change="1598510844" exec-time="122" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
          </lrm_resource>
          <lrm_resource id="SERVICE10" type="service10" class="systemd">
            <lrm_rsc_op id="SERVICE10_last_0" operation_key="SERVICE10_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="21:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:7;21:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="50" rc-code="7" op-status="0" interval="0" last-run="1598510844" last-rc-change="1598510844" exec-time="121" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
          </lrm_resource>
          <lrm_resource id="SERVICE11" type="service11" class="systemd">
            <lrm_rsc_op id="SERVICE11_last_0" operation_key="SERVICE11_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="22:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:7;22:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="54" rc-code="7" op-status="0" interval="0" last-run="1598510844" last-rc-change="1598510844" exec-time="118" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
          </lrm_resource>
          <lrm_resource id="SERVICE12" type="service12" class="systemd">
            <lrm_rsc_op id="SERVICE12_last_0" operation_key="SERVICE12_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="23:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:7;23:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="58" rc-code="7" op-status="0" interval="0" last-run="1598510844" last-rc-change="1598510844" exec-time="117" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
          </lrm_resource>
          <lrm_resource id="SERVICE13" type="service13" class="systemd">
            <lrm_rsc_op id="SERVICE13_last_0" operation_key="SERVICE13_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="24:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:7;24:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="62" rc-code="7" op-status="0" interval="0" last-run="1598510844" last-rc-change="1598510844" exec-time="118" queue-time="1" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
          </lrm_resource>
          <lrm_resource id="SERVICE14" type="service14" class="systemd">
            <lrm_rsc_op id="SERVICE14_last_0" operation_key="SERVICE14_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="55:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;55:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="74" rc-code="0" op-status="0" interval="0" last-run="1598510864" last-rc-change="1598510864" exec-time="2209" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE14_monitor_10000" operation_key="SERVICE14_monitor_10000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="56:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;56:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="76" rc-code="0" op-status="0" interval="10000" last-rc-change="1598510866" exec-time="2" queue-time="0" op-digest="873ed4f07792aa8ff18f3254244675ea"/>
          </lrm_resource>
          <lrm_resource id="SERVICE15" type="service15" class="systemd">
            <lrm_rsc_op id="SERVICE15_last_0" operation_key="SERVICE15_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="63:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;63:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="75" rc-code="0" op-status="0" interval="0" last-run="1598510864" last-rc-change="1598510864" exec-time="2316" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE15_monitor_10000" operation_key="SERVICE15_monitor_10000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="64:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;64:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="77" rc-code="0" op-status="0" interval="10000" last-rc-change="1598510866" exec-time="1" queue-time="0" op-digest="873ed4f07792aa8ff18f3254244675ea"/>
          </lrm_resource>
          <lrm_resource id="SERVICE1" type="service1" class="systemd">
            <lrm_rsc_op id="SERVICE1_last_0" operation_key="SERVICE1_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="20:370:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:7;20:370:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="193" rc-code="7" op-status="0" interval="0" last-run="1598510961" last-rc-change="1598510961" exec-time="1" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
          </lrm_resource>
        </lrm_resources>
      </lrm>
      <transient_attributes id="1">
        <instance_attributes id="status-1"/>
      </transient_attributes>
    </node_state>
    <node_state id="2" uname="NODE2" in_ccm="true" crmd="online" crm-debug-origin="do_update_resource" join="member" expected="member">
      <transient_attributes id="2">
        <instance_attributes id="status-2"/>
      </transient_attributes>
      <lrm id="2">
        <lrm_resources>
          <lrm_resource id="SERVICE4" type="service4" class="systemd">
            <lrm_rsc_op id="SERVICE4_last_failure_0" operation_key="SERVICE4_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="7:1079:7:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;7:1079:7:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="97" rc-code="0" op-status="0" interval="0" last-run="1598436620" last-rc-change="1598436620" exec-time="1" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE4_last_0" operation_key="SERVICE4_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="33:226:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;33:226:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="385" rc-code="0" op-status="0" interval="0" last-run="1598455186" last-rc-change="1598455186" exec-time="2417" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE4_monitor_5000" operation_key="SERVICE4_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="34:226:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;34:226:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="386" rc-code="0" op-status="0" interval="5000" last-rc-change="1598455188" exec-time="2" queue-time="0" op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
          </lrm_resource>
          <lrm_resource id="SERVICE9" type="service9" class="systemd">
            <lrm_rsc_op id="SERVICE9_last_0" operation_key="SERVICE9_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="44:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;44:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="433" rc-code="0" op-status="0" interval="0" last-run="1598510844" last-rc-change="1598510844" exec-time="2079" queue-time="1" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE9_monitor_5000" operation_key="SERVICE9_monitor_5000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="2:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;2:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="434" rc-code="0" op-status="0" interval="5000" last-rc-change="1598510846" exec-time="5" queue-time="0" op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
          </lrm_resource>
          <lrm_resource id="SERVICE13" type="service13" class="systemd">
            <lrm_rsc_op id="SERVICE13_last_0" operation_key="SERVICE13_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="48:0:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;48:0:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="108" rc-code="0" op-status="0" interval="0" last-run="1598436624" last-rc-change="1598436624" exec-time="2129" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE13_monitor_5000" operation_key="SERVICE13_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="51:1:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;51:1:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="110" rc-code="0" op-status="0" interval="5000" last-rc-change="1598436626" exec-time="2" queue-time="0" op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
          </lrm_resource>
          <lrm_resource id="SERVICE11" type="service11" class="systemd">
            <lrm_rsc_op id="SERVICE11_last_0" operation_key="SERVICE11_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="43:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;43:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="104" rc-code="0" op-status="0" interval="0" last-run="1598436620" last-rc-change="1598436620" exec-time="2060" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE11_monitor_5000" operation_key="SERVICE11_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="45:0:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;45:0:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="106" rc-code="0" op-status="0" interval="5000" last-rc-change="1598436624" exec-time="10" queue-time="0" op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
          </lrm_resource>
          <lrm_resource id="SERVICE6" type="service6" class="systemd">
            <lrm_rsc_op id="SERVICE6_last_0" operation_key="SERVICE6_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="46:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;46:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="85" rc-code="0" op-status="0" interval="0" last-run="1598436614" last-rc-change="1598436614" exec-time="2055" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE6_monitor_5000" operation_key="SERVICE6_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="30:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;30:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="99" rc-code="0" op-status="0" interval="5000" last-rc-change="1598436620" exec-time="1" queue-time="0" op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
          </lrm_resource>
          <lrm_resource id="SERVICE1" type="service1" class="systemd">
            <lrm_rsc_op id="SERVICE1_last_failure_0" operation_key="SERVICE1_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="19:296:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;19:296:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="397" rc-code="0" op-status="0" interval="0" last-run="1598510455" last-rc-change="1598510455" exec-time="4" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE1_last_0" operation_key="SERVICE1_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="30:369:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;30:369:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="465" rc-code="0" op-status="0" interval="0" last-run="1598510934" last-rc-change="1598510934" exec-time="2208" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE1_monitor_5000" operation_key="SERVICE1_monitor_5000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="9:369:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;9:369:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="466" rc-code="0" op-status="0" interval="5000" last-rc-change="1598510936" exec-time="21" queue-time="1" op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
          </lrm_resource>
          <lrm_resource id="SERVICE2" type="service2" class="systemd">
            <lrm_rsc_op id="SERVICE2_last_0" operation_key="SERVICE2_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="29:154:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;29:154:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="249" rc-code="0" op-status="0" interval="0" last-run="1598440560" last-rc-change="1598440560" exec-time="2072" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE2_monitor_5000" operation_key="SERVICE2_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="30:154:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;30:154:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="250" rc-code="0" op-status="0" interval="5000" last-rc-change="1598440562" exec-time="1" queue-time="0" op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
          </lrm_resource>
          <lrm_resource id="VIRTUALIP" type="IPaddr2" class="ocf" provider="heartbeat">
            <lrm_rsc_op id="VIRTUALIP_last_0" operation_key="VIRTUALIP_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="21:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;21:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="76" rc-code="0" op-status="0" interval="0" last-run="1598436612" last-rc-change="1598436612" exec-time="508" queue-time="0" op-digest="81452beaabf19c71189916d97c5ea2a8"/>
            <lrm_rsc_op id="VIRTUALIP_monitor_5000" operation_key="VIRTUALIP_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="22:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;22:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="77" rc-code="0" op-status="0" interval="5000" last-rc-change="1598436612" exec-time="44" queue-time="0" op-digest="c9f93be4b942aa24f2bef92561fe636f"/>
          </lrm_resource>
          <lrm_resource id="SERVICE3" type="service3" class="systemd">
            <lrm_rsc_op id="SERVICE3_last_0" operation_key="SERVICE3_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="37:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;37:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="82" rc-code="0" op-status="0" interval="0" last-run="1598436614" last-rc-change="1598436614" exec-time="2130" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE3_monitor_5000" operation_key="SERVICE3_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="38:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;38:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="90" rc-code="0" op-status="0" interval="5000" last-rc-change="1598436616" exec-time="1" queue-time="0" op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
          </lrm_resource>
          <lrm_resource id="SERVICE10" type="service10" class="systemd">
            <lrm_rsc_op id="SERVICE10_last_0" operation_key="SERVICE10_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="58:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;58:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="91" rc-code="0" op-status="0" interval="0" last-run="1598436616" last-rc-change="1598436616" exec-time="2058" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE10_monitor_5000" operation_key="SERVICE10_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="42:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;42:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="103" rc-code="0" op-status="0" interval="5000" last-rc-change="1598436620" exec-time="4" queue-time="0" op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
          </lrm_resource>
          <lrm_resource id="SERVICE8" type="service8" class="systemd">
            <lrm_rsc_op id="SERVICE8_last_0" operation_key="SERVICE8_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="52:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;52:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="87" rc-code="0" op-status="0" interval="0" last-run="1598436615" last-rc-change="1598436615" exec-time="2055" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE8_monitor_5000" operation_key="SERVICE8_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="36:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;36:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="101" rc-code="0" op-status="0" interval="5000" last-rc-change="1598436620" exec-time="1" queue-time="0" op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
          </lrm_resource>
          <lrm_resource id="SERVICE7" type="service7" class="systemd">
            <lrm_rsc_op id="SERVICE7_last_0" operation_key="SERVICE7_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="49:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;49:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="86" rc-code="0" op-status="0" interval="0" last-run="1598436614" last-rc-change="1598436614" exec-time="2085" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE7_monitor_5000" operation_key="SERVICE7_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="33:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;33:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="100" rc-code="0" op-status="0" interval="5000" last-rc-change="1598436620" exec-time="0" queue-time="1" op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
          </lrm_resource>
          <lrm_resource id="SERVICE15" type="service15" class="systemd">
            <lrm_rsc_op id="SERVICE15_last_0" operation_key="SERVICE15_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="79:1077:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;79:1077:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="73" rc-code="0" op-status="0" interval="0" last-run="1598436522" last-rc-change="1598436522" exec-time="2237" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE15_monitor_10000" operation_key="SERVICE15_monitor_10000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="80:1077:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;80:1077:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="75" rc-code="0" op-status="0" interval="10000" last-rc-change="1598436525" exec-time="3" queue-time="0" op-digest="873ed4f07792aa8ff18f3254244675ea"/>
          </lrm_resource>
          <lrm_resource id="SERVICE14" type="service14" class="systemd">
            <lrm_rsc_op id="SERVICE14_last_0" operation_key="SERVICE14_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="71:1077:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;71:1077:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="72" rc-code="0" op-status="0" interval="0" last-run="1598436522" last-rc-change="1598436522" exec-time="2053" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE14_monitor_10000" operation_key="SERVICE14_monitor_10000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="72:1077:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;72:1077:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="74" rc-code="0" op-status="0" interval="10000" last-rc-change="1598436524" exec-time="1" queue-time="0" op-digest="873ed4f07792aa8ff18f3254244675ea"/>
          </lrm_resource>
          <lrm_resource id="SERVICE5" type="service5" class="systemd">
            <lrm_rsc_op id="SERVICE5_last_0" operation_key="SERVICE5_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="43:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;43:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="84" rc-code="0" op-status="0" interval="0" last-run="1598436614" last-rc-change="1598436614" exec-time="2098" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE5_monitor_5000" operation_key="SERVICE5_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="27:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;27:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="98" rc-code="0" op-status="0" interval="5000" last-rc-change="1598436620" exec-time="2" queue-time="0" op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
          </lrm_resource>
          <lrm_resource id="SOURCEIP" type="IPsrcaddr" class="ocf" provider="heartbeat">
            <lrm_rsc_op id="SOURCEIP_last_0" operation_key="SOURCEIP_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="24:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;24:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="78" rc-code="0" op-status="0" interval="0" last-run="1598436612" last-rc-change="1598436612" exec-time="216" queue-time="0" op-digest="4238bbb83fc1ef2f001ab06eb7d14fe0"/>
            <lrm_rsc_op id="SOURCEIP_monitor_5000" operation_key="SOURCEIP_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="25:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;25:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="79" rc-code="0" op-status="0" interval="5000" last-rc-change="1598436613" exec-time="31" queue-time="0" op-digest="44fe0c6ab563ac3595f20eb3473b89fa"/>
          </lrm_resource>
          <lrm_resource id="SERVICE12" type="service12" class="systemd">
            <lrm_rsc_op id="SERVICE12_last_0" operation_key="SERVICE12_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="46:0:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;46:0:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="107" rc-code="0" op-status="0" interval="0" last-run="1598436624" last-rc-change="1598436624" exec-time="2060" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
            <lrm_rsc_op id="SERVICE12_monitor_5000" operation_key="SERVICE12_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="48:1:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;48:1:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="109" rc-code="0" op-status="0" interval="5000" last-rc-change="1598436626" exec-time="2" queue-time="0" op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
          </lrm_resource>
        </lrm_resources>
      </lrm>
    </node_state>
  </status>
</cib>

-------------- next part --------------
## CLUSTER IS RUNNING 

*****
Stack: corosync
Current DC: NODE2 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum
Last updated: Thu Aug 27 08:42:21 2020
Last change: Thu Aug 27 08:42:18 2020 by hacluster via crmd on NODE1

2 nodes configured
19 resources configured

Online: [ NODE1 NODE2 ]

Active resources:

 Resource Group: IPV
     VIRTUALIP  (ocf::heartbeat:IPaddr2):       Started NODE2
     SOURCEIP   (ocf::heartbeat:IPsrcaddr):      Started NODE2
SERVICE1        (systemd:service1):     Started NODE2
SERVICE2    (systemd:service2): Started NODE2
SERVICE3       (systemd:service3):  Started NODE2
SERVICE4       (systemd:service4):  Started NODE2
SERVICE5      (systemd:service5): Started NODE2
SERVICE6 (systemd:service6):    Started NODE2
SERVICE7  (systemd:service7):      Started NODE2
SERVICE8   (systemd:service8):  Started NODE2
SERVICE9        (systemd:service9):   Started NODE2
SERVICE10        (systemd:service10):        Started NODE2
SERVICE11       (systemd:service11):       Started NODE2
SERVICE12   (systemd:service12):       Started NODE2
SERVICE13  (systemd:service13):       Started NODE2
 Clone Set: SERVICE14-clone [SERVICE14]
     Started: [ NODE1 NODE2 ]
 Clone Set: SERVICE15-clone [SERVICE15]
     Started: [ NODE1 NODE2 ]

*****


## REBOOTING NODE1
# /var/log/cluster/corosync.log

Aug 27 08:46:16 [1331] NODE2       crmd:     info: crm_update_peer_expected:  handle_request: Node NODE1[1] - expected state is now down (was member)
Aug 27 08:46:16 [1331] NODE2       crmd:     info: handle_shutdown_request:   Creating shutdown request for NODE1 (state=S_IDLE)
Aug 27 08:46:16 [1329] NODE2      attrd:     info: attrd_peer_update: Setting shutdown[NODE1]: (null) -> 1598510776 from NODE2
Aug 27 08:46:16 [1329] NODE2      attrd:     info: write_attribute:   Sent CIB request 167 with 1 change for shutdown (id n/a, set n/a)
Aug 27 08:46:16 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/attrd/167)
Aug 27 08:46:16 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.2 2
Aug 27 08:46:16 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.3 (null)
Aug 27 08:46:16 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=3
Aug 27 08:46:16 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']:  <nvpair id="status-1-shutdown" name="shutdown" value="1598510776"/>
Aug 27 08:46:16 [1331] NODE2       crmd:     info: abort_transition_graph:    Transition aborted by status-1-shutdown doing create shutdown=1598510776: Transient attribute change | cib=0.434.3 source=abort_unless_down:356 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true
Aug 27 08:46:16 [1331] NODE2       crmd:   notice: do_state_transition:       State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Aug 27 08:46:16 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/attrd/167, version=0.434.3)
Aug 27 08:46:16 [1329] NODE2      attrd:     info: attrd_cib_callback:        CIB update 167 result for shutdown: OK | rc=0
Aug 27 08:46:16 [1329] NODE2      attrd:     info: attrd_cib_callback:        * shutdown[NODE1]=1598510776
Aug 27 08:46:16 [1330] NODE2    pengine:   notice: unpack_config:     On loss of CCM Quorum: Ignore
Aug 27 08:46:16 [1330] NODE2    pengine:     info: determine_online_status:   Node NODE1 is shutting down
Aug 27 08:46:16 [1330] NODE2    pengine:     info: determine_online_status:   Node NODE2 is online
Aug 27 08:46:16 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE9 active on NODE1
Aug 27 08:46:16 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE4 active on NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE1 active on NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 1 is already processed
Aug 27 08:46:16 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 2 is already processed
Aug 27 08:46:16 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 1 is already processed
Aug 27 08:46:16 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 2 is already processed
Aug 27 08:46:16 [1330] NODE2    pengine:     info: group_print:        Resource Group: IPV
Aug 27 08:46:16 [1330] NODE2    pengine:     info: common_print:           VIRTUALIP  (ocf::heartbeat:IPaddr2):       Started NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: common_print:           SOURCEIP   (ocf::heartbeat:IPsrcaddr):      Started NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: common_print:      SERVICE1        (systemd:service1):     Started NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: common_print:      SERVICE2    (systemd:service2): Started NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: common_print:      SERVICE3       (systemd:service3):  Started NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: common_print:      SERVICE4       (systemd:service4):  Started NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: common_print:      SERVICE5      (systemd:service5): Started NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: common_print:      SERVICE6 (systemd:service6):    Started NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: common_print:      SERVICE7  (systemd:service7):      Started NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: common_print:      SERVICE8   (systemd:service8):  Started NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: common_print:      SERVICE9        (systemd:service9):   Started NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: common_print:      SERVICE10        (systemd:service10):        Started NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: common_print:      SERVICE11       (systemd:service11):       Started NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: common_print:      SERVICE12   (systemd:service12):       Started NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: common_print:      SERVICE13  (systemd:service13):       Started NODE2
Aug 27 08:46:16 [1330] NODE2    pengine:     info: clone_print:        Clone Set: SERVICE14-clone [SERVICE14]
Aug 27 08:46:16 [1330] NODE2    pengine:     info: short_print:            Started: [ NODE1 NODE2 ]
Aug 27 08:46:16 [1330] NODE2    pengine:     info: clone_print:        Clone Set: SERVICE15-clone [SERVICE15]
Aug 27 08:46:16 [1330] NODE2    pengine:     info: short_print:            Started: [ NODE1 NODE2 ]
Aug 27 08:46:16 [1330] NODE2    pengine:     info: native_color:      Resource SERVICE14:0 cannot run anywhere
Aug 27 08:46:16 [1330] NODE2    pengine:     info: native_color:      Resource SERVICE15:0 cannot run anywhere
Aug 27 08:46:16 [1330] NODE2    pengine:   notice: sched_shutdown_op: Scheduling shutdown of node NODE1
Aug 27 08:46:16 [1330] NODE2    pengine:   notice: LogNodeActions:     * Shutdown NODE1
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   VIRTUALIP       (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   SOURCEIP        (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE1        (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE2    (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE3       (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE4       (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE5      (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE6 (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE7  (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE8   (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE9        (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE10        (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE11       (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE12   (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE13  (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:   notice: LogAction:  * Stop       SERVICE14:0         (            NODE1 )   due to node availability
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE14:1        (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:   notice: LogAction:  * Stop       SERVICE15:0                  (            NODE1 )   due to node availability
Aug 27 08:46:16 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE15:1 (Started NODE2)
Aug 27 08:46:16 [1330] NODE2    pengine:   notice: process_pe_message:        Calculated transition 331, saving inputs in /var/lib/pacemaker/pengine/pe-input-1508.bz2
Aug 27 08:46:16 [1331] NODE2       crmd:     info: do_state_transition:       State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Aug 27 08:46:16 [1331] NODE2       crmd:     info: do_te_invoke:      Processing graph 331 (ref=pe_calc-dc-1598510776-962) derived from /var/lib/pacemaker/pengine/pe-input-1508.bz2
Aug 27 08:46:16 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating stop operation SERVICE14_stop_0 on NODE1 | action 54
Aug 27 08:46:16 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating stop operation SERVICE15_stop_0 on NODE1 | action 61
Aug 27 08:46:16 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.3 2
Aug 27 08:46:16 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.4 (null)
Aug 27 08:46:16 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=4
Aug 27 08:46:16 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE14']/lrm_rsc_op[@id='SERVICE14_last_0']:  @operation_key=SERVICE14_stop_0, @operation=stop, @transition-key=54:331:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @transition-magic=-1:193;54:331:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1598510798, @last-rc-change=1598510798, @exec-time=0
Aug 27 08:46:16 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/198, version=0.434.4)
Aug 27 08:46:16 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.4 2
Aug 27 08:46:16 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.5 (null)
Aug 27 08:46:16 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=5
Aug 27 08:46:16 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE15']/lrm_rsc_op[@id='SERVICE15_last_0']:  @operation_key=SERVICE15_stop_0, @operation=stop, @transition-key=61:331:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @transition-magic=-1:193;61:331:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1598510798, @last-rc-change=1598510798, @exec-time=0
Aug 27 08:46:16 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/199, version=0.434.5)
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.5 2
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.6 (null)
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=6
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE14']/lrm_rsc_op[@id='SERVICE14_last_0']:  @transition-magic=0:0;54:331:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=186, @rc-code=0, @op-status=0, @exec-time=2052
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/200, version=0.434.6)
Aug 27 08:46:18 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE14_stop_0 (54) confirmed on NODE1 (rc=0)
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.6 2
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.7 (null)
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=7
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE15']/lrm_rsc_op[@id='SERVICE15_last_0']:  @transition-magic=0:0;61:331:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=188, @rc-code=0, @op-status=0, @exec-time=2104
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/201, version=0.434.7)
Aug 27 08:46:18 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE15_stop_0 (61) confirmed on NODE1 (rc=0)
Aug 27 08:46:18 [1331] NODE2       crmd:     info: te_crm_command:    Executing crm-event (68): do_shutdown on NODE1
Aug 27 08:46:18 [1331] NODE2       crmd:   notice: run_graph: Transition 331 (Complete=7, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1508.bz2): Complete
Aug 27 08:46:18 [1331] NODE2       crmd:     info: do_log:    Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
Aug 27 08:46:18 [1331] NODE2       crmd:   notice: do_state_transition:       State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Aug 27 08:46:18 [1331] NODE2       crmd:     info: pcmk_cpg_membership:       Group crmd event 13: NODE1 (node 1 pid 1261) left via cpg_leave
Aug 27 08:46:18 [1331] NODE2       crmd:     info: crm_update_peer_proc:      pcmk_cpg_membership: Node NODE1[1] - corosync-cpg is now offline
Aug 27 08:46:18 [1331] NODE2       crmd:     info: peer_update_callback:      Client NODE1/peer now has status [offline] (DC=true, changed=4000000)
Aug 27 08:46:18 [1331] NODE2       crmd:     info: controld_delete_node_state:        Deleting transient attributes for node NODE1 (via CIB call 1357) | xpath=//node_state[@uname='NODE1']/transient_attributes
Aug 27 08:46:18 [1331] NODE2       crmd:   notice: peer_update_callback:      do_shutdown of peer NODE1 is complete | op=68
Aug 27 08:46:18 [1331] NODE2       crmd:     info: pcmk_cpg_membership:       Group crmd event 13: NODE2 (node 2 pid 1331) is member
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_delete operation for section //node_state[@uname='NODE1']/transient_attributes to all (origin=local/crmd/1357)
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1358)
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.7 2
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.8 b48848cd3dacdd25352599c5c5add0cc
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_perform_op:    -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=8
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_delete operation for section //node_state[@uname='NODE1']/transient_attributes: OK (rc=0, origin=NODE2/crmd/1357, version=0.434.8)
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.8 2
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.9 (null)
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=9
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']:  @crmd=offline, @crm-debug-origin=peer_update_callback, @join=down, @expected=down
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1358, version=0.434.9)
Aug 27 08:46:18 [1329] NODE2      attrd:     info: pcmk_cpg_membership:       Group attrd event 13: NODE1 (node 1 pid 1259) left for unknown reason
Aug 27 08:46:18 [1329] NODE2      attrd:     info: crm_update_peer_proc:      pcmk_cpg_membership: Node NODE1[1] - corosync-cpg is now offline
Aug 27 08:46:18 [1329] NODE2      attrd:   notice: crm_update_peer_state_iter:        Node NODE1 state is now lost | nodeid=1 previous=member source=crm_update_peer_proc
Aug 27 08:46:18 [1329] NODE2      attrd:   notice: attrd_peer_remove: Removing all NODE1 attributes for peer loss
Aug 27 08:46:18 [1329] NODE2      attrd:     info: crm_reap_dead_member:      Removing node with name NODE1 and id 1 from membership cache
Aug 27 08:46:18 [1329] NODE2      attrd:   notice: reap_crm_member:   Purged 1 peer with id=1 and/or uname=NODE1 from the membership cache
Aug 27 08:46:18 [1329] NODE2      attrd:     info: pcmk_cpg_membership:       Group attrd event 13: NODE2 (node 2 pid 1329) is member
Aug 27 08:46:18 [1327] NODE2 stonith-ng:     info: pcmk_cpg_membership:       Group stonith-ng event 13: NODE1 (node 1 pid 1257) left for unknown reason
Aug 27 08:46:18 [1327] NODE2 stonith-ng:     info: crm_update_peer_proc:      pcmk_cpg_membership: Node NODE1[1] - corosync-cpg is now offline
Aug 27 08:46:18 [1327] NODE2 stonith-ng:   notice: crm_update_peer_state_iter:        Node NODE1 state is now lost | nodeid=1 previous=member source=crm_update_peer_proc
Aug 27 08:46:18 [1327] NODE2 stonith-ng:     info: crm_reap_dead_member:      Removing node with name NODE1 and id 1 from membership cache
Aug 27 08:46:18 [1327] NODE2 stonith-ng:   notice: reap_crm_member:   Purged 1 peer with id=1 and/or uname=NODE1 from the membership cache
Aug 27 08:46:18 [1327] NODE2 stonith-ng:     info: pcmk_cpg_membership:       Group stonith-ng event 13: NODE2 (node 2 pid 1327) is member
Aug 27 08:46:18 [1326] NODE2        cib:     info: cib_process_shutdown_req:  Shutdown REQ from NODE1
Aug 27 08:46:18 [1326] NODE2        cib:     info: pcmk_cpg_membership:       Group cib event 13: NODE1 (node 1 pid 1256) left via cpg_leave
Aug 27 08:46:18 [1326] NODE2        cib:     info: crm_update_peer_proc:      pcmk_cpg_membership: Node NODE1[1] - corosync-cpg is now offline
Aug 27 08:46:18 [1326] NODE2        cib:   notice: crm_update_peer_state_iter:        Node NODE1 state is now lost | nodeid=1 previous=member source=crm_update_peer_proc
Aug 27 08:46:18 [1326] NODE2        cib:     info: crm_reap_dead_member:      Removing node with name NODE1 and id 1 from membership cache
Aug 27 08:46:18 [1326] NODE2        cib:   notice: reap_crm_member:   Purged 1 peer with id=1 and/or uname=NODE1 from the membership cache
Aug 27 08:46:18 [1326] NODE2        cib:     info: pcmk_cpg_membership:       Group cib event 13: NODE2 (node 2 pid 1326) is member
Aug 27 08:46:18 [1240] NODE2 pacemakerd:     info: pcmk_cpg_membership:       Group pacemakerd event 13: NODE1 (node 1 pid 1247) left via cpg_leave
Aug 27 08:46:18 [1240] NODE2 pacemakerd:     info: crm_update_peer_proc:      pcmk_cpg_membership: Node NODE1[1] - corosync-cpg is now offline
Aug 27 08:46:18 [1240] NODE2 pacemakerd:     info: pcmk_cpg_membership:       Group pacemakerd event 13: NODE2 (node 2 pid 1240) is member
Aug 27 08:46:18 [1240] NODE2 pacemakerd:     info: mcp_cpg_deliver:   Ignoring process list sent by peer for local node
[1160] NODE2 corosyncnotice  [TOTEM ] A new membership (10.130.10.2:208639) was formed. Members left: 1
[1160] NODE2 corosyncwarning [CPG   ] downlist left_list: 1 received
[1160] NODE2 corosyncnotice  [QUORUM] Members[1]: 2
[1160] NODE2 corosyncnotice  [MAIN  ] Completed service synchronization, ready to provide service.
Aug 27 08:46:19 [1240] NODE2 pacemakerd:     info: pcmk_quorum_notification:  Quorum retained | membership=208639 members=1
Aug 27 08:46:19 [1240] NODE2 pacemakerd:   notice: crm_update_peer_state_iter:        Node NODE1 state is now lost | nodeid=1 previous=member source=crm_reap_unseen_nodes
Aug 27 08:46:19 [1331] NODE2       crmd:     info: pcmk_quorum_notification:  Quorum retained | membership=208639 members=1
Aug 27 08:46:19 [1331] NODE2       crmd:   notice: crm_update_peer_state_iter:        Node NODE1 state is now lost | nodeid=1 previous=member source=crm_reap_unseen_nodes
Aug 27 08:46:19 [1331] NODE2       crmd:     info: peer_update_callback:      Cluster node NODE1 is now lost (was member)
Aug 27 08:46:19 [1331] NODE2       crmd:   notice: peer_update_callback:      do_shutdown of peer NODE1 is complete | op=68
Aug 27 08:46:19 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1359)
Aug 27 08:46:19 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1362)
Aug 27 08:46:19 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1363)
Aug 27 08:46:19 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1359, version=0.434.9)
Aug 27 08:46:19 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section nodes: OK (rc=0, origin=NODE2/crmd/1362, version=0.434.9)
Aug 27 08:46:19 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.9 2
Aug 27 08:46:19 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.10 (null)
Aug 27 08:46:19 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=10
Aug 27 08:46:19 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']:  @in_ccm=false, @crm-debug-origin=post_cache_update
Aug 27 08:46:19 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']:  @crm-debug-origin=post_cache_update
Aug 27 08:46:19 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1363, version=0.434.10)
Aug 27 08:46:24 [1326] NODE2        cib:     info: cib_process_ping:  Reporting our current digest to NODE2: 6dfb70e00efc6d1de9f85e9eac251ba6 for 0.434.10 (0x563b74317400 0)


*****
Stack: corosync
Current DC: NODE2 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum
Last updated: Thu Aug 27 08:46:21 2020
Last change: Thu Aug 27 08:42:18 2020 by hacluster via crmd on NODE1

2 nodes configured
19 resources configured

Online: [ NODE2 ]
OFFLINE: [ NODE1 ]

Active resources:

 Resource Group: IPV
     VIRTUALIP  (ocf::heartbeat:IPaddr2):       Started NODE2
     SOURCEIP   (ocf::heartbeat:IPsrcaddr):      Started NODE2
SERVICE1        (systemd:service1):     Started NODE2
SERVICE2    (systemd:service2): Started NODE2
SERVICE3       (systemd:service3):  Started NODE2
SERVICE4       (systemd:service4):  Started NODE2
SERVICE5      (systemd:service5): Started NODE2
SERVICE6 (systemd:service6):    Started NODE2
SERVICE7  (systemd:service7):      Started NODE2
SERVICE8   (systemd:service8):  Started NODE2
SERVICE9        (systemd:service9):   Started NODE2
SERVICE10        (systemd:service10):        Started NODE2
SERVICE11       (systemd:service11):       Started NODE2
SERVICE12   (systemd:service12):       Started NODE2
SERVICE13  (systemd:service13):       Started NODE2
 Clone Set: SERVICE14-clone [SERVICE14]
     Started: [ NODE2 ]
 Clone Set: SERVICE15-clone [SERVICE15]
     Started: [ NODE2 ]
*****





## WAITING FOR NODE1 TO COME BACK




*****
Stack: corosync
Current DC: NODE2 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum
Last updated: Thu Aug 27 08:47:02 2020
Last change: Thu Aug 27 08:42:18 2020 by hacluster via crmd on NODE1

2 nodes configured
19 resources configured

Online: [ NODE1 NODE2 ]

Active resources:

 Resource Group: IPV
     VIRTUALIP  (ocf::heartbeat:IPaddr2):       Started NODE2
     SOURCEIP   (ocf::heartbeat:IPsrcaddr):      Started NODE2
SERVICE1        (systemd:service1):     Stopping[ NODE1 NODE2 ]                      # <-- !!!!!!!!!!
SERVICE2    (systemd:service2): Started NODE2
SERVICE3       (systemd:service3):  Started NODE2
SERVICE4       (systemd:service4):  Started NODE2
SERVICE5      (systemd:service5): Started NODE2
SERVICE6 (systemd:service6):    Started NODE2
SERVICE7  (systemd:service7):      Started NODE2
SERVICE8   (systemd:service8):  Started NODE2
SERVICE9        (systemd:service9):   Started NODE2
SERVICE10        (systemd:service10):        Started NODE2
SERVICE11       (systemd:service11):       Started NODE2
SERVICE12   (systemd:service12):       Started NODE2
SERVICE13  (systemd:service13):       Started NODE2
 Clone Set: SERVICE14-clone [SERVICE14]
     Started: [ NODE2 ]
 Clone Set: SERVICE15-clone [SERVICE15]
     Started: [ NODE2 ]
*****



# /var/log/cluster/corosync.log
[1160] NODE2 corosyncnotice  [TOTEM ] A new membership (10.130.10.1:208644) was formed. Members joined: 1
[1160] NODE2 corosyncwarning [CPG   ] downlist left_list: 0 received
[1160] NODE2 corosyncwarning [CPG   ] downlist left_list: 0 received
[1160] NODE2 corosyncnotice  [QUORUM] Members[2]: 1 2
[1160] NODE2 corosyncnotice  [MAIN  ] Completed service synchronization, ready to provide service.
Aug 27 08:46:54 [1240] NODE2 pacemakerd:     info: pcmk_quorum_notification:  Quorum retained | membership=208644 members=2
Aug 27 08:46:54 [1240] NODE2 pacemakerd:   notice: crm_update_peer_state_iter:        Node NODE1 state is now member | nodeid=1 previous=lost source=pcmk_quorum_notification
Aug 27 08:46:54 [1331] NODE2       crmd:     info: pcmk_quorum_notification:  Quorum retained | membership=208644 members=2
Aug 27 08:46:54 [1331] NODE2       crmd:   notice: crm_update_peer_state_iter:        Node NODE1 state is now member | nodeid=1 previous=lost source=pcmk_quorum_notification
Aug 27 08:46:54 [1331] NODE2       crmd:     info: peer_update_callback:      Cluster node NODE1 is now member (was lost)
Aug 27 08:46:54 [1331] NODE2       crmd:   notice: peer_update_callback:      do_shutdown of peer NODE1 is complete | op=68
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1364)
Aug 27 08:46:54 [1331] NODE2       crmd:   notice: do_state_transition:       State transition S_IDLE -> S_INTEGRATION | input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=check_join_state
Aug 27 08:46:54 [1331] NODE2       crmd:     info: do_dc_join_offer_one:      Making join-1 offers to any unconfirmed nodes because an unknown node joined
Aug 27 08:46:54 [1331] NODE2       crmd:     info: join_make_offer:   Making join-1 offers based on membership event 208644
Aug 27 08:46:54 [1331] NODE2       crmd:     info: join_make_offer:   Not making join-1 offer to already known node NODE2 (confirmed)
Aug 27 08:46:54 [1331] NODE2       crmd:     info: join_make_offer:   Not making join-1 offer to inactive node NODE1
Aug 27 08:46:54 [1331] NODE2       crmd:     info: abort_transition_graph:    Transition aborted: Peer Halt | source=do_te_invoke:150 complete=true
Aug 27 08:46:54 [1331] NODE2       crmd:     info: do_state_transition:       State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state
Aug 27 08:46:54 [1331] NODE2       crmd:     info: do_state_transition:       State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1367)
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1368)
Aug 27 08:46:54 [1331] NODE2       crmd:     info: abort_transition_graph:    Transition aborted: Peer Cancelled | source=do_te_invoke:143 complete=true
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1371)
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1372)
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section cib to all (origin=local/crmd/1373)
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.10 2
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.11 (null)
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=11
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']:  @crm-debug-origin=peer_update_callback
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1364, version=0.434.11)
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section nodes: OK (rc=0, origin=NODE2/crmd/1367, version=0.434.11)
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.11 2
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.12 (null)
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=12
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']:  @in_ccm=true, @crm-debug-origin=post_cache_update
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1368, version=0.434.12)
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section nodes: OK (rc=0, origin=NODE2/crmd/1371, version=0.434.12)
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.12 2
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.13 (null)
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=13
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']:  @crm-debug-origin=do_state_transition
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']:  @crm-debug-origin=do_state_transition
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1372, version=0.434.13)
Aug 27 08:46:54 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section cib: OK (rc=0, origin=NODE2/crmd/1373, version=0.434.13)
Aug 27 08:46:55 [1330] NODE2    pengine:   notice: unpack_config:     On loss of CCM Quorum: Ignore
Aug 27 08:46:55 [1330] NODE2    pengine:     info: determine_online_status:   Node NODE2 is online
Aug 27 08:46:55 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE4 active on NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE1 active on NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 2 is already processed
Aug 27 08:46:55 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 2 is already processed
Aug 27 08:46:55 [1330] NODE2    pengine:     info: group_print:        Resource Group: IPV
Aug 27 08:46:55 [1330] NODE2    pengine:     info: common_print:           VIRTUALIP  (ocf::heartbeat:IPaddr2):       Started NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: common_print:           SOURCEIP   (ocf::heartbeat:IPsrcaddr):      Started NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: common_print:      SERVICE1        (systemd:service1):     Started NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: common_print:      SERVICE2    (systemd:service2): Started NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: common_print:      SERVICE3       (systemd:service3):  Started NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: common_print:      SERVICE4       (systemd:service4):  Started NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: common_print:      SERVICE5      (systemd:service5): Started NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: common_print:      SERVICE6 (systemd:service6):    Started NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: common_print:      SERVICE7  (systemd:service7):      Started NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: common_print:      SERVICE8   (systemd:service8):  Started NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: common_print:      SERVICE9        (systemd:service9):   Started NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: common_print:      SERVICE10        (systemd:service10):        Started NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: common_print:      SERVICE11       (systemd:service11):       Started NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: common_print:      SERVICE12   (systemd:service12):       Started NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: common_print:      SERVICE13  (systemd:service13):       Started NODE2
Aug 27 08:46:55 [1330] NODE2    pengine:     info: clone_print:        Clone Set: SERVICE14-clone [SERVICE14]
Aug 27 08:46:55 [1330] NODE2    pengine:     info: short_print:            Started: [ NODE2 ]
Aug 27 08:46:55 [1330] NODE2    pengine:     info: short_print:            Stopped: [ NODE1 ]
Aug 27 08:46:55 [1330] NODE2    pengine:     info: clone_print:        Clone Set: SERVICE15-clone [SERVICE15]
Aug 27 08:46:55 [1330] NODE2    pengine:     info: short_print:            Started: [ NODE2 ]
Aug 27 08:46:55 [1330] NODE2    pengine:     info: short_print:            Stopped: [ NODE1 ]
Aug 27 08:46:55 [1330] NODE2    pengine:     info: native_color:      Resource SERVICE14:1 cannot run anywhere
Aug 27 08:46:55 [1330] NODE2    pengine:     info: native_color:      Resource SERVICE15:1 cannot run anywhere
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   VIRTUALIP       (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SOURCEIP        (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE1        (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE2    (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE3       (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE4       (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE5      (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE6 (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE7  (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE8   (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE9        (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE10        (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE11       (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE12   (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE13  (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE14:0        (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE14:1        (Stopped)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE15:0 (Started NODE2)
Aug 27 08:46:55 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE15:1 (Stopped)
Aug 27 08:46:55 [1330] NODE2    pengine:   notice: process_pe_message:        Calculated transition 332, saving inputs in /var/lib/pacemaker/pengine/pe-input-1509.bz2
Aug 27 08:46:55 [1331] NODE2       crmd:     info: do_state_transition:       State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Aug 27 08:46:55 [1331] NODE2       crmd:     info: do_te_invoke:      Processing graph 332 (ref=pe_calc-dc-1598510815-968) derived from /var/lib/pacemaker/pengine/pe-input-1509.bz2
Aug 27 08:46:55 [1331] NODE2       crmd:   notice: run_graph: Transition 332 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1509.bz2): Complete
Aug 27 08:46:55 [1331] NODE2       crmd:     info: do_log:    Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
Aug 27 08:46:55 [1331] NODE2       crmd:   notice: do_state_transition:       State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Aug 27 08:46:56 [1240] NODE2 pacemakerd:     info: pcmk_cpg_membership:       Group pacemakerd event 14: node 1 pid 1254 joined via cpg_join
Aug 27 08:46:56 [1240] NODE2 pacemakerd:     info: pcmk_cpg_membership:       Group pacemakerd event 14: NODE1 (node 1 pid 1254) is member
Aug 27 08:46:56 [1240] NODE2 pacemakerd:     info: crm_update_peer_proc:      pcmk_cpg_membership: Node NODE1[1] - corosync-cpg is now online
Aug 27 08:46:56 [1240] NODE2 pacemakerd:     info: pcmk_cpg_membership:       Group pacemakerd event 14: NODE2 (node 2 pid 1240) is member
Aug 27 08:46:56 [1240] NODE2 pacemakerd:     info: mcp_cpg_deliver:   Ignoring process list sent by peer for local node
Aug 27 08:46:56 [1329] NODE2      attrd:     info: pcmk_cpg_membership:       Group attrd event 14: node 1 pid 1262 joined via cpg_join
Aug 27 08:46:56 [1329] NODE2      attrd:     info: crm_get_peer:      Created entry 90dd23fb-b65a-41cb-99d6-45b858fa9f67/0x562bfd3a2510 for node NODE1/1 (2 total)
Aug 27 08:46:56 [1329] NODE2      attrd:     info: crm_get_peer:      Node 1 is now known as NODE1
Aug 27 08:46:56 [1329] NODE2      attrd:  warning: crm_update_peer_uname:     Node names with capitals are discouraged, consider changing 'NODE1'
Aug 27 08:46:56 [1329] NODE2      attrd:     info: crm_get_peer:      Node 1 has uuid 1
Aug 27 08:46:56 [1329] NODE2      attrd:     info: pcmk_cpg_membership:       Group attrd event 14: NODE1 (node 1 pid 1262) is member
Aug 27 08:46:56 [1329] NODE2      attrd:     info: crm_update_peer_proc:      pcmk_cpg_membership: Node NODE1[1] - corosync-cpg is now online
Aug 27 08:46:56 [1329] NODE2      attrd:   notice: crm_update_peer_state_iter:        Node NODE1 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc
Aug 27 08:46:56 [1329] NODE2      attrd:     info: pcmk_cpg_membership:       Group attrd event 14: NODE2 (node 2 pid 1329) is member
Aug 27 08:46:56 [1327] NODE2 stonith-ng:     info: pcmk_cpg_membership:       Group stonith-ng event 14: node 1 pid 1260 joined via cpg_join
Aug 27 08:46:56 [1327] NODE2 stonith-ng:     info: crm_get_peer:      Created entry 7da09b35-a6ad-4694-ab1f-f7572dec96be/0x5592dc152db0 for node NODE1/1 (2 total)
Aug 27 08:46:56 [1327] NODE2 stonith-ng:     info: crm_get_peer:      Node 1 is now known as NODE1
Aug 27 08:46:56 [1327] NODE2 stonith-ng:  warning: crm_update_peer_uname:     Node names with capitals are discouraged, consider changing 'NODE1'
Aug 27 08:46:56 [1327] NODE2 stonith-ng:     info: crm_get_peer:      Node 1 has uuid 1
Aug 27 08:46:56 [1327] NODE2 stonith-ng:     info: pcmk_cpg_membership:       Group stonith-ng event 14: NODE1 (node 1 pid 1260) is member
Aug 27 08:46:56 [1327] NODE2 stonith-ng:     info: crm_update_peer_proc:      pcmk_cpg_membership: Node NODE1[1] - corosync-cpg is now online
Aug 27 08:46:56 [1327] NODE2 stonith-ng:   notice: crm_update_peer_state_iter:        Node NODE1 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc
Aug 27 08:46:56 [1327] NODE2 stonith-ng:     info: pcmk_cpg_membership:       Group stonith-ng event 14: NODE2 (node 2 pid 1327) is member
Aug 27 08:46:57 [1326] NODE2        cib:     info: pcmk_cpg_membership:       Group cib event 14: node 1 pid 1259 joined via cpg_join
Aug 27 08:46:57 [1326] NODE2        cib:     info: crm_get_peer:      Created entry 9b5a2951-192e-49bd-b5eb-eab2bd4873db/0x563b74302580 for node NODE1/1 (2 total)
Aug 27 08:46:57 [1326] NODE2        cib:     info: crm_get_peer:      Node 1 is now known as NODE1
Aug 27 08:46:57 [1326] NODE2        cib:  warning: crm_update_peer_uname:     Node names with capitals are discouraged, consider changing 'NODE1'
Aug 27 08:46:57 [1326] NODE2        cib:     info: crm_get_peer:      Node 1 has uuid 1
Aug 27 08:46:57 [1326] NODE2        cib:     info: pcmk_cpg_membership:       Group cib event 14: NODE1 (node 1 pid 1259) is member
Aug 27 08:46:57 [1326] NODE2        cib:     info: crm_update_peer_proc:      pcmk_cpg_membership: Node NODE1[1] - corosync-cpg is now online
Aug 27 08:46:57 [1326] NODE2        cib:   notice: crm_update_peer_state_iter:        Node NODE1 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc
Aug 27 08:46:57 [1326] NODE2        cib:     info: pcmk_cpg_membership:       Group cib event 14: NODE2 (node 2 pid 1326) is member
Aug 27 08:46:57 [1331] NODE2       crmd:     info: pcmk_cpg_membership:       Group crmd event 14: node 1 pid 1264 joined via cpg_join
Aug 27 08:46:57 [1331] NODE2       crmd:     info: pcmk_cpg_membership:       Group crmd event 14: NODE1 (node 1 pid 1264) is member
Aug 27 08:46:57 [1331] NODE2       crmd:     info: crm_update_peer_proc:      pcmk_cpg_membership: Node NODE1[1] - corosync-cpg is now online
Aug 27 08:46:57 [1331] NODE2       crmd:     info: peer_update_callback:      Client NODE1/peer now has status [online] (DC=true, changed=4000000)
Aug 27 08:46:57 [1331] NODE2       crmd:     info: te_trigger_stonith_history_sync:   Fence history will be synchronized cluster-wide within 5 seconds
Aug 27 08:46:57 [1331] NODE2       crmd:     info: pcmk_cpg_membership:       Group crmd event 14: NODE2 (node 2 pid 1331) is member
Aug 27 08:46:57 [1331] NODE2       crmd:   notice: do_state_transition:       State transition S_IDLE -> S_INTEGRATION | input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=peer_update_callback
Aug 27 08:46:57 [1331] NODE2       crmd:     info: do_dc_join_offer_one:      Making join-1 offers to any unconfirmed nodes because an unknown node joined
Aug 27 08:46:57 [1331] NODE2       crmd:     info: join_make_offer:   Not making join-1 offer to already known node NODE2 (confirmed)
Aug 27 08:46:57 [1331] NODE2       crmd:     info: join_make_offer:   Sending join-1 offer to NODE1
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1376)
Aug 27 08:46:57 [1331] NODE2       crmd:     info: abort_transition_graph:    Transition aborted: Peer Halt | source=do_te_invoke:150 complete=true
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.13 2
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.14 (null)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=14
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']:  @crmd=online, @crm-debug-origin=peer_update_callback
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1376, version=0.434.14)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_delete operation for section //node_state[@uname='NODE1']/transient_attributes: OK (rc=0, origin=NODE1/attrd/2, version=0.434.14)
Aug 27 08:46:57 [1329] NODE2      attrd:     info: election_count_vote:       election-attrd round 1 (owner node ID 1) pass: vote from NODE1 (Uptime)
Aug 27 08:46:57 [1329] NODE2      attrd:     info: attrd_peer_update: Setting #attrd-protocol[NODE1]: (null) -> 2 from NODE1
Aug 27 08:46:57 [1329] NODE2      attrd:     info: election_check:    election-attrd won by local node
Aug 27 08:46:57 [1329] NODE2      attrd:   notice: attrd_declare_winner:      Recorded local node as attribute writer (was unset)
Aug 27 08:46:57 [1329] NODE2      attrd:     info: write_attribute:   Sent CIB request 168 with 1 change for fail-count-SERVICE1#start_0 (id n/a, set n/a)
Aug 27 08:46:57 [1329] NODE2      attrd:     info: write_attribute:   Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a
Aug 27 08:46:57 [1329] NODE2      attrd:     info: write_attribute:   Sent CIB request 169 with 1 change for fail-count-SERVICE4#start_0 (id n/a, set n/a)
Aug 27 08:46:57 [1329] NODE2      attrd:     info: write_attribute:   Sent CIB request 170 with 1 change for last-failure-SERVICE1#start_0 (id n/a, set n/a)
Aug 27 08:46:57 [1329] NODE2      attrd:     info: write_attribute:   Sent CIB request 171 with 1 change for last-failure-SERVICE4#start_0 (id n/a, set n/a)
Aug 27 08:46:57 [1329] NODE2      attrd:     info: write_attribute:   Sent CIB request 172 with 1 change for last-failure-SERVICE1#stop_0 (id n/a, set n/a)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/attrd/168)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/attrd/169)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/attrd/170)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/attrd/171)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/attrd/172)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.14 2
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.15 (null)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=15
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/attrd/168, version=0.434.15)
Aug 27 08:46:57 [1329] NODE2      attrd:     info: attrd_cib_callback:        CIB update 168 result for fail-count-SERVICE1#start_0: OK | rc=0
Aug 27 08:46:57 [1329] NODE2      attrd:     info: attrd_cib_callback:        * fail-count-SERVICE1#start_0[NODE2]=(null)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.15 2
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.16 (null)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=16
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/attrd/169, version=0.434.16)
Aug 27 08:46:57 [1329] NODE2      attrd:     info: attrd_cib_callback:        CIB update 169 result for fail-count-SERVICE4#start_0: OK | rc=0
Aug 27 08:46:57 [1329] NODE2      attrd:     info: attrd_cib_callback:        * fail-count-SERVICE4#start_0[NODE2]=(null)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.16 2
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.17 (null)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=17
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/attrd/170, version=0.434.17)
Aug 27 08:46:57 [1329] NODE2      attrd:     info: attrd_cib_callback:        CIB update 170 result for last-failure-SERVICE1#start_0: OK | rc=0
Aug 27 08:46:57 [1329] NODE2      attrd:     info: attrd_cib_callback:        * last-failure-SERVICE1#start_0[NODE2]=(null)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.17 2
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.18 (null)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=18
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/attrd/171, version=0.434.18)
Aug 27 08:46:57 [1329] NODE2      attrd:     info: attrd_cib_callback:        CIB update 171 result for last-failure-SERVICE4#start_0: OK | rc=0
Aug 27 08:46:57 [1329] NODE2      attrd:     info: attrd_cib_callback:        * last-failure-SERVICE4#start_0[NODE2]=(null)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.18 2
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.19 (null)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=19
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/attrd/172, version=0.434.19)
Aug 27 08:46:57 [1329] NODE2      attrd:     info: attrd_cib_callback:        CIB update 172 result for last-failure-SERVICE1#stop_0: OK | rc=0
Aug 27 08:46:57 [1329] NODE2      attrd:     info: attrd_cib_callback:        * last-failure-SERVICE1#stop_0[NODE2]=(null)
Aug 27 08:46:57 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section nodes: OK (rc=0, origin=NODE1/crmd/5, version=0.434.19)
Aug 27 08:46:58 [1331] NODE2       crmd:     info: join_make_offer:   Sending join-1 offer to NODE1
Aug 27 08:46:58 [1331] NODE2       crmd:     info: join_make_offer:   Sending join-1 offer to NODE2
Aug 27 08:46:58 [1331] NODE2       crmd:     info: abort_transition_graph:    Transition aborted: Node join | source=do_dc_join_offer_one:269 complete=true
Aug 27 08:46:58 [1331] NODE2       crmd:     info: do_dc_join_offer_one:      Waiting on join-1 requests from 2 outstanding nodes
Aug 27 08:46:59 [1331] NODE2       crmd:     info: crm_update_peer_expected:  do_dc_join_filter_offer: Node NODE1[1] - expected state is now member (was down)
Aug 27 08:46:59 [1331] NODE2       crmd:     info: do_state_transition:       State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1379)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1380)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_replace:       Digest matched on replace from NODE2: a73e34b494d0ff4a4eadead9620ea5b4
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_replace:       Replaced 0.434.19 with 0.434.19 from NODE2
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_replace operation for section 'all': OK (rc=0, origin=NODE2/crmd/1378, version=0.434.19)
Aug 27 08:46:59 [1331] NODE2       crmd:     info: do_state_transition:       State transition S_FINALIZE_JOIN -> S_INTEGRATION | input=I_JOIN_REQUEST cause=C_HA_MESSAGE origin=route_message
Aug 27 08:46:59 [1331] NODE2       crmd:     info: join_make_offer:   Sending join-1 offer to NODE1
Aug 27 08:46:59 [1331] NODE2       crmd:     info: join_make_offer:   Sending join-1 offer to NODE2
Aug 27 08:46:59 [1331] NODE2       crmd:     info: abort_transition_graph:    Transition aborted: Node join | source=do_dc_join_offer_one:269 complete=true
Aug 27 08:46:59 [1331] NODE2       crmd:     info: do_dc_join_offer_one:      Waiting on join-1 requests from 2 outstanding nodes
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section nodes: OK (rc=0, origin=NODE2/crmd/1379, version=0.434.19)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section nodes: OK (rc=0, origin=NODE2/crmd/1380, version=0.434.19)
Aug 27 08:46:59 [1331] NODE2       crmd:     info: do_log:    Input I_JOIN_RESULT received in state S_INTEGRATION from route_message
Aug 27 08:46:59 [1331] NODE2       crmd:     info: do_dc_join_ack:    Ignoring out-of-sequence join-1 confirmation from NODE2 (currently welcomed not finalized)
Aug 27 08:46:59 [1331] NODE2       crmd:     info: do_log:    Input I_JOIN_RESULT received in state S_INTEGRATION from route_message
Aug 27 08:46:59 [1331] NODE2       crmd:     info: do_dc_join_ack:    Ignoring out-of-sequence join-1 confirmation from NODE1 (currently welcomed not finalized)
Aug 27 08:46:59 [1331] NODE2       crmd:     info: do_state_transition:       State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1383)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1384)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_replace:       Digest matched on replace from NODE2: a73e34b494d0ff4a4eadead9620ea5b4
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_replace:       Replaced 0.434.19 with 0.434.19 from NODE2
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_replace operation for section 'all': OK (rc=0, origin=NODE2/crmd/1382, version=0.434.19)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section nodes: OK (rc=0, origin=NODE2/crmd/1383, version=0.434.19)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section nodes: OK (rc=0, origin=NODE2/crmd/1384, version=0.434.19)
Aug 27 08:46:59 [1331] NODE2       crmd:     info: controld_delete_node_state:        Deleting resource history for node NODE1 (via CIB call 1385) | xpath=//node_state[@uname='NODE1']/lrm
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_delete operation for section //node_state[@uname='NODE1']/lrm to all (origin=local/crmd/1385)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1386)
Aug 27 08:46:59 [1331] NODE2       crmd:     info: controld_delete_node_state:        Deleting resource history for node NODE2 (via CIB call 1387) | xpath=//node_state[@uname='NODE2']/lrm
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_delete operation for section //node_state[@uname='NODE2']/lrm to all (origin=local/crmd/1387)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1388)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.19 2
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.20 (null)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    -- /cib/status/node_state[@id='1']/lrm[@id='1']
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=20
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_delete operation for section //node_state[@uname='NODE1']/lrm: OK (rc=0, origin=NODE2/crmd/1385, version=0.434.20)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.20 2
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.21 (null)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=21
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']:  @crm-debug-origin=do_lrm_query_internal, @join=member, @expected=member
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']:  <lrm id="1"/>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                     <lrm_resources/>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                   </lrm>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1386, version=0.434.21)
Aug 27 08:46:59 [1331] NODE2       crmd:     info: do_state_transition:       State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state
Aug 27 08:46:59 [1331] NODE2       crmd:     info: abort_transition_graph:    Transition aborted: Peer Cancelled | source=do_te_invoke:143 complete=true
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.21 2
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.22 (null)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    -- /cib/status/node_state[@id='2']/lrm[@id='2']
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=22
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_delete operation for section //node_state[@uname='NODE2']/lrm: OK (rc=0, origin=NODE2/crmd/1387, version=0.434.22)
Aug 27 08:46:59 [1331] NODE2       crmd:     info: abort_transition_graph:    Transition aborted by deletion of lrm[@id='2']: Resource state removal | cib=0.434.22 source=abort_unless_down:370 path=/cib/status/node_state[@id='2']/lrm[@id='2'] complete=true
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.22 2
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.23 (null)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=23
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']:  @crm-debug-origin=do_lrm_query_internal
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='2']:  <lrm id="2"/>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                     <lrm_resources>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="SERVICE4" type="service4" class="systemd">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE4_last_failure_0" operation_key="SERVICE4_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="7:1079:7:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;7:1079:7:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="97" rc-code="0" op-status="0" interval="0" last-ru
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE4_last_0" operation_key="SERVICE4_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="33:226:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;33:226:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="385" rc-code="0" op-status="0" interval="0" last-run="15984551
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE4_monitor_5000" operation_key="SERVICE4_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="34:226:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;34:226:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="386" rc-code="0" op-status="0" interval="5000" la
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="SERVICE9" type="service9" class="systemd">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE9_last_0" operation_key="SERVICE9_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="44:296:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;44:296:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="398" rc-code="0" op-status="0" interval="0" last-run="1598510455" last-rc-change
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE9_monitor_5000" operation_key="SERVICE9_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="46:297:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;46:297:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="400" rc-code="0" op-status="0" interval="5000" last-rc-change="1598
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="SERVICE13" type="service13" class="systemd">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE13_last_0" operation_key="SERVICE13_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="48:0:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;48:0:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="108" rc-code="0" op-status="0" interval="0" last-run="1598436624" last-rc-change="159843
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE13_monitor_5000" operation_key="SERVICE13_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="51:1:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;51:1:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="110" rc-code="0" op-status="0" interval="5000" last-rc-change="1598436626"
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="SERVICE11" type="service11" class="systemd">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE11_last_0" operation_key="SERVICE11_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="43:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;43:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="104" rc-code="0" op-status="0" interval="0" last-run="1598436620" last-rc-ch
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE11_monitor_5000" operation_key="SERVICE11_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="45:0:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;45:0:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="106" rc-code="0" op-status="0" interval="5000" last-rc-change="159843
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="SERVICE6" type="service6" class="systemd">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE6_last_0" operation_key="SERVICE6_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="46:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;46:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="85" rc-code="0" op-status="0" interval="0" last-run="1598436614"
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE6_monitor_5000" operation_key="SERVICE6_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="30:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;30:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="99" rc-code="0" op-status="0" interval="5000" last-
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="SERVICE1" type="service1" class="systemd">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE1_last_failure_0" operation_key="SERVICE1_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="19:296:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;19:296:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="397" rc-code="0" op-status="0" interval="0" last-run="1598510455" la
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE1_last_0" operation_key="SERVICE1_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="30:328:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;30:328:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="425" rc-code="0" op-status="0" interval="0" last-run="1598510531" last-rc-change
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE1_monitor_5000" operation_key="SERVICE1_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="20:328:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;20:328:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="426" rc-code="0" op-status="0" interval="5000" last-rc-change="1598
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="SERVICE2" type="service2" class="systemd">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE2_last_0" operation_key="SERVICE2_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="29:154:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;29:154:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="249" rc-code="0" op-status="0" interval="0" last-run="15
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE2_monitor_5000" operation_key="SERVICE2_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="30:154:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;30:154:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="250" rc-code="0" op-status="0" interval="50
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="VIRTUALIP" type="IPaddr2" class="ocf" provider="heartbeat">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="VIRTUALIP_last_0" operation_key="VIRTUALIP_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="21:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;21:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="76" rc-code="0" op-status="0" interval="0" last-run="1598436612" last-rc-cha
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="VIRTUALIP_monitor_5000" operation_key="VIRTUALIP_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="22:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;22:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="77" rc-code="0" op-status="0" interval="5000" last-rc-change="1
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="SERVICE3" type="service3" class="systemd">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE3_last_0" operation_key="SERVICE3_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="37:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;37:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="82" rc-code="0" op-status="0" interval="0" l
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE3_monitor_5000" operation_key="SERVICE3_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="38:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;38:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="90" rc-code="0" op-status="0" i
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="SERVICE10" type="service10" class="systemd">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE10_last_0" operation_key="SERVICE10_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="58:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;58:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="91" rc-code="0" op-status="0" interval="0" last-run="1598436616" last-rc-chang
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE10_monitor_5000" operation_key="SERVICE10_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="42:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;42:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="103" rc-code="0" op-status="0" interval="5000" last-rc-change="15
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="SERVICE8" type="service8" class="systemd">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE8_last_0" operation_key="SERVICE8_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="52:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;52:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="87" rc-code="0" op-status="0" interval="0" last-run="1598436615" las
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE8_monitor_5000" operation_key="SERVICE8_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="36:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;36:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="101" rc-code="0" op-status="0" interval="5000" last-rc-
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="SERVICE7" type="service7" class="systemd">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE7_last_0" operation_key="SERVICE7_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="49:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;49:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="86" rc-code="0" op-status="0" interval="0" last-run="1598436614" l
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE7_monitor_5000" operation_key="SERVICE7_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="33:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;33:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="100" rc-code="0" op-status="0" interval="5000" last-r
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="SERVICE15" type="service15" class="systemd">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE15_last_0" operation_key="SERVICE15_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="79:1077:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;79:1077:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="73" rc-code="0" op-status="0" interval="0" last-run="1598436522" last-rc-change="159
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE15_monitor_10000" operation_key="SERVICE15_monitor_10000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="80:1077:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;80:1077:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="75" rc-code="0" op-status="0" interval="10000" last-rc-change="159843
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="SERVICE14" type="service14" class="systemd">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE14_last_0" operation_key="SERVICE14_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="71:1077:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;71:1077:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="72" rc-code="0" op-status="0" interval="0" last-run="1598436522" l
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE14_monitor_10000" operation_key="SERVICE14_monitor_10000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="72:1077:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;72:1077:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="74" rc-code="0" op-status="0" interval="10000" last
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="SERVICE5" type="service5" class="systemd">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE5_last_0" operation_key="SERVICE5_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="43:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;43:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="84" rc-code="0" op-status="0" interval="0" last-run="1598436614" last-rc-c
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE5_monitor_5000" operation_key="SERVICE5_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="27:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;27:1079:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="98" rc-code="0" op-status="0" interval="5000" last-rc-change=
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="SOURCEIP" type="IPsrcaddr" class="ocf" provider="heartbeat">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SOURCEIP_last_0" operation_key="SOURCEIP_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="24:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;24:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="78" rc-code="0" op-status="0" interval="0" last-run="1598436612" last-rc-chang
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SOURCEIP_monitor_5000" operation_key="SOURCEIP_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="25:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" transition-magic="0:0;25:1078:0:96431d62-66ac-46b4-9fbc-1cd89548a77d" exit-reason="" on_node="NODE2" call-id="79" rc-code="0" op-status="0" interval="5000" last-rc-change="159
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <lrm_resource id="SERVICE12" type="service12" class="systemd">
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE12_last_0" operation_key="SERVICE12_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="46:0:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;46:0:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="107" rc-code="0" op-status="0" interval="0" last-run="1598436624" last-rc-
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                         <lrm_rsc_op id="SERVICE12_monitor_5000" operation_key="SERVICE12_monitor_5000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="48:1:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;48:1:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE2" call-id="109" rc-code="0" op-status="0" interval="5000" last-rc-change
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       </lrm_resource>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                     </lrm_resources>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_perform_op:    ++                                   </lrm>
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1388, version=0.434.23)
Aug 27 08:46:59 [1331] NODE2       crmd:     info: abort_transition_graph:    Transition aborted: LRM Refresh | source=process_resource_updates:294 complete=true
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1391)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1392)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section cib to all (origin=local/crmd/1393)
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_file_backup:   Archived previous version as /var/lib/pacemaker/cib/cib-54.raw
Aug 27 08:47:00 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section nodes: OK (rc=0, origin=NODE2/crmd/1391, version=0.434.23)
Aug 27 08:47:00 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.23 2
Aug 27 08:47:00 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.24 (null)
Aug 27 08:47:00 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=24
Aug 27 08:47:00 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']:  @crm-debug-origin=do_state_transition
Aug 27 08:47:00 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']:  @crm-debug-origin=do_state_transition
Aug 27 08:47:00 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1392, version=0.434.24)
Aug 27 08:47:00 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section cib: OK (rc=0, origin=NODE2/crmd/1393, version=0.434.24)
Aug 27 08:47:00 [1326] NODE2        cib:     info: cib_file_write_with_digest:        Wrote version 0.434.0 of the CIB to disk (digest: 5698b0502258efe142e4750fe055e120)
Aug 27 08:47:00 [1326] NODE2        cib:     info: cib_file_write_with_digest:        Reading cluster configuration file /var/lib/pacemaker/cib/cib.xnvdSF (digest: /var/lib/pacemaker/cib/cib.m8Q2QM)
Aug 27 08:47:00 [1326] NODE2        cib:     info: cib_file_backup:   Archived previous version as /var/lib/pacemaker/cib/cib-55.raw
Aug 27 08:47:00 [1326] NODE2        cib:     info: cib_file_write_with_digest:        Wrote version 0.434.0 of the CIB to disk (digest: 6230f319a9bccbd7c4b84b80c2299632)
Aug 27 08:47:00 [1326] NODE2        cib:     info: cib_file_write_with_digest:        Reading cluster configuration file /var/lib/pacemaker/cib/cib.AaHAsG (digest: /var/lib/pacemaker/cib/cib.Zq949N)
Aug 27 08:47:01 [1330] NODE2    pengine:   notice: unpack_config:     On loss of CCM Quorum: Ignore
Aug 27 08:47:01 [1330] NODE2    pengine:     info: determine_online_status:   Node NODE1 is online
Aug 27 08:47:01 [1330] NODE2    pengine:     info: determine_online_status:   Node NODE2 is online
Aug 27 08:47:01 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE4 active on NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE1 active on NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 1 is already processed
Aug 27 08:47:01 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 2 is already processed
Aug 27 08:47:01 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 1 is already processed
Aug 27 08:47:01 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 2 is already processed
Aug 27 08:47:01 [1330] NODE2    pengine:     info: group_print:        Resource Group: IPV
Aug 27 08:47:01 [1330] NODE2    pengine:     info: common_print:           VIRTUALIP  (ocf::heartbeat:IPaddr2):       Started NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: common_print:           SOURCEIP   (ocf::heartbeat:IPsrcaddr):      Started NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: common_print:      SERVICE1        (systemd:service1):     Started NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: common_print:      SERVICE2    (systemd:service2): Started NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: common_print:      SERVICE3       (systemd:service3):  Started NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: common_print:      SERVICE4       (systemd:service4):  Started NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: common_print:      SERVICE5      (systemd:service5): Started NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: common_print:      SERVICE6 (systemd:service6):    Started NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: common_print:      SERVICE7  (systemd:service7):      Started NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: common_print:      SERVICE8   (systemd:service8):  Started NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: common_print:      SERVICE9        (systemd:service9):   Started NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: common_print:      SERVICE10        (systemd:service10):        Started NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: common_print:      SERVICE11       (systemd:service11):       Started NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: common_print:      SERVICE12   (systemd:service12):       Started NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: common_print:      SERVICE13  (systemd:service13):       Started NODE2
Aug 27 08:47:01 [1330] NODE2    pengine:     info: clone_print:        Clone Set: SERVICE14-clone [SERVICE14]
Aug 27 08:47:01 [1330] NODE2    pengine:     info: short_print:            Started: [ NODE2 ]
Aug 27 08:47:01 [1330] NODE2    pengine:     info: short_print:            Stopped: [ NODE1 ]
Aug 27 08:47:01 [1330] NODE2    pengine:     info: clone_print:        Clone Set: SERVICE15-clone [SERVICE15]
Aug 27 08:47:01 [1330] NODE2    pengine:     info: short_print:            Started: [ NODE2 ]
Aug 27 08:47:01 [1330] NODE2    pengine:     info: short_print:            Stopped: [ NODE1 ]
Aug 27 08:47:01 [1330] NODE2    pengine:     info: RecurringOp:        Start recurring monitor (10s) for SERVICE14:1 on NODE1
Aug 27 08:47:01 [1330] NODE2    pengine:     info: RecurringOp:        Start recurring monitor (10s) for SERVICE15:1 on NODE1
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   VIRTUALIP       (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   SOURCEIP        (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE1        (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE2    (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE3       (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE4       (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE5      (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE6 (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE7  (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE8   (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE9        (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE10        (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE11       (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE12   (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE13  (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE14:0        (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:   notice: LogAction:  * Start      SERVICE14:1         (            NODE1 )
Aug 27 08:47:01 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE15:0 (Started NODE2)
Aug 27 08:47:01 [1330] NODE2    pengine:   notice: LogAction:  * Start      SERVICE15:1                  (            NODE1 )
Aug 27 08:47:01 [1330] NODE2    pengine:   notice: process_pe_message:        Calculated transition 333, saving inputs in /var/lib/pacemaker/pengine/pe-input-1510.bz2
Aug 27 08:47:01 [1331] NODE2       crmd:     info: do_state_transition:       State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Aug 27 08:47:01 [1331] NODE2       crmd:     info: do_te_invoke:      Processing graph 333 (ref=pe_calc-dc-1598510821-982) derived from /var/lib/pacemaker/pengine/pe-input-1510.bz2
Aug 27 08:47:01 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation VIRTUALIP_monitor_0 on NODE1 | action 18
Aug 27 08:47:01 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SOURCEIP_monitor_0 on NODE1 | action 19
Aug 27 08:47:01 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE1_monitor_0 on NODE1 | action 20
Aug 27 08:47:01 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE2_monitor_0 on NODE1 | action 21
Aug 27 08:47:01 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE3_monitor_0 on NODE1 | action 22
Aug 27 08:47:01 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE4_monitor_0 on NODE1 | action 23
Aug 27 08:47:01 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE5_monitor_0 on NODE1 | action 24
Aug 27 08:47:01 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE6_monitor_0 on NODE1 | action 25
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.24 2
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.25 (null)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=25
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']:  @crm-debug-origin=do_update_resource
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="VIRTUALIP" type="IPaddr2" class="ocf" provider="heartbeat"/>
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="VIRTUALIP_last_0" operation_key="VIRTUALIP_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="18:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;18:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" op-status="-1" interval="0"
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/12, version=0.434.25)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.25 2
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.26 (null)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=26
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="SOURCEIP" type="IPsrcaddr" class="ocf" provider="heartbeat"/>
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="SOURCEIP_last_0" operation_key="SOURCEIP_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="19:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;19:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" op-status="-1" interval="0" la
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/13, version=0.434.26)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.26 2
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.27 (null)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=27
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="SERVICE1" type="service1" class="systemd"/>
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="SERVICE1_last_0" operation_key="SERVICE1_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="20:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;20:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" op-status="-1" interval="0" la
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/14, version=0.434.27)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.27 2
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.28 (null)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=28
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="SERVICE2" type="service2" class="systemd"/>
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="SERVICE2_last_0" operation_key="SERVICE2_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="21:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;21:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" op-sta
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/15, version=0.434.28)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.28 2
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.29 (null)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=29
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="SERVICE3" type="service3" class="systemd"/>
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="SERVICE3_last_0" operation_key="SERVICE3_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="22:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;22:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="1
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/16, version=0.434.29)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.29 2
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.30 (null)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=30
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="SERVICE4" type="service4" class="systemd"/>
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="SERVICE4_last_0" operation_key="SERVICE4_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="23:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;23:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" op-status="-
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/17, version=0.434.30)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.30 2
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.31 (null)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=31
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="SERVICE5" type="service5" class="systemd"/>
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="SERVICE5_last_0" operation_key="SERVICE5_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="24:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;24:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" op-status="-1" interval="0
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/18, version=0.434.31)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.31 2
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.32 (null)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=32
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE1']/lrm_rsc_op[@id='SERVICE1_last_0']:  @transition-magic=0:0;20:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=13, @rc-code=0, @op-status=0, @exec-time=4
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE1']:  <lrm_rsc_op id="SERVICE1_last_failure_0" operation_key="SERVICE1_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="20:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;20:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="13" rc-code="0"
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/19, version=0.434.32)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.32 2
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.33 (null)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=33
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="SERVICE6" type="service6" class="systemd"/>
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="SERVICE6_last_0" operation_key="SERVICE6_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="25:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;25:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" op-status="-1" i
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/20, version=0.434.33)
Aug 27 08:47:01 [1331] NODE2       crmd:  warning: status_from_rc:    Action 20 (SERVICE1_monitor_0) on NODE1 failed (target: 7 vs. rc: 0): Error
Aug 27 08:47:01 [1331] NODE2       crmd:   notice: abort_transition_graph:    Transition aborted by operation SERVICE1_monitor_0 'modify' on NODE1: Event failed | magic=0:0;20:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e cib=0.434.32 source=match_graph_event:299 complete=false
Aug 27 08:47:01 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE1_monitor_0 (20) confirmed on NODE1 (rc=0)
Aug 27 08:47:01 [1331] NODE2       crmd:     info: process_graph_event:       Detected action (333.20) SERVICE1_monitor_0.13=ok: failed
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.33 2
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.34 (null)
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=34
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SOURCEIP']/lrm_rsc_op[@id='SOURCEIP_last_0']:  @transition-magic=0:7;19:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=9, @rc-code=7, @op-status=0, @last-run=1598510844, @last-rc-change=1598510844, @exec-time=40
Aug 27 08:47:01 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/21, version=0.434.34)
Aug 27 08:47:01 [1331] NODE2       crmd:     info: match_graph_event: Action SOURCEIP_monitor_0 (19) confirmed on NODE1 (rc=7)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.34 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.35 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=35
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='VIRTUALIP']/lrm_rsc_op[@id='VIRTUALIP_last_0']:  @transition-magic=0:7;18:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=5, @rc-code=7, @op-status=0, @exec-time=441
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/22, version=0.434.35)
Aug 27 08:47:02 [1331] NODE2       crmd:     info: match_graph_event: Action VIRTUALIP_monitor_0 (18) confirmed on NODE1 (rc=7)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.35 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.36 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=36
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE2']/lrm_rsc_op[@id='SERVICE2_last_0']:  @transition-magic=0:7;21:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=17, @rc-code=7, @op-status=0, @exec-time=239
Aug 27 08:47:02 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE2_monitor_0 (21) confirmed on NODE1 (rc=7)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/23, version=0.434.36)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.36 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.37 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=37
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE3']/lrm_rsc_op[@id='SERVICE3_last_0']:  @transition-magic=0:7;22:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=21, @rc-code=7, @op-status=0, @exec-time=240
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/24, version=0.434.37)
Aug 27 08:47:02 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE3_monitor_0 (22) confirmed on NODE1 (rc=7)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.37 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.38 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=38
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE4']/lrm_rsc_op[@id='SERVICE4_last_0']:  @transition-magic=0:7;23:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=25, @rc-code=7, @op-status=0, @exec-time=241
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/25, version=0.434.38)
Aug 27 08:47:02 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE4_monitor_0 (23) confirmed on NODE1 (rc=7)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.38 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.39 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=39
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE5']/lrm_rsc_op[@id='SERVICE5_last_0']:  @transition-magic=0:7;24:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=29, @rc-code=7, @op-status=0, @exec-time=241
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/26, version=0.434.39)
Aug 27 08:47:02 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE5_monitor_0 (24) confirmed on NODE1 (rc=7)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.39 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.40 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=40
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE6']/lrm_rsc_op[@id='SERVICE6_last_0']:  @transition-magic=0:7;25:333:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=33, @rc-code=7, @op-status=0, @exec-time=242
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/27, version=0.434.40)
Aug 27 08:47:02 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE6_monitor_0 (25) confirmed on NODE1 (rc=7)
Aug 27 08:47:02 [1331] NODE2       crmd:   notice: run_graph: Transition 333 (Complete=8, Pending=0, Fired=0, Skipped=9, Incomplete=17, Source=/var/lib/pacemaker/pengine/pe-input-1510.bz2): Stopped
Aug 27 08:47:02 [1331] NODE2       crmd:     info: do_state_transition:       State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
Aug 27 08:47:02 [1330] NODE2    pengine:   notice: unpack_config:     On loss of CCM Quorum: Ignore
Aug 27 08:47:02 [1330] NODE2    pengine:     info: determine_online_status:   Node NODE1 is online
Aug 27 08:47:02 [1330] NODE2    pengine:     info: determine_online_status:   Node NODE2 is online
Aug 27 08:47:02 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE1 active on NODE1
Aug 27 08:47:02 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE1 active on NODE1
Aug 27 08:47:02 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE4 active on NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE1 active on NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 1 is already processed
Aug 27 08:47:02 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 2 is already processed
Aug 27 08:47:02 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 1 is already processed
Aug 27 08:47:02 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 2 is already processed
Aug 27 08:47:02 [1330] NODE2    pengine:     info: group_print:        Resource Group: IPV
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:           VIRTUALIP  (ocf::heartbeat:IPaddr2):       Started NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:           SOURCEIP   (ocf::heartbeat:IPsrcaddr):      Started NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:      SERVICE1        (systemd:service1):     Started
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:              1 : NODE1
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:              2 : NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:      SERVICE2    (systemd:service2): Started NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:      SERVICE3       (systemd:service3):  Started NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:      SERVICE4       (systemd:service4):  Started NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:      SERVICE5      (systemd:service5): Started NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:      SERVICE6 (systemd:service6):    Started NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:      SERVICE7  (systemd:service7):      Started NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:      SERVICE8   (systemd:service8):  Started NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:      SERVICE9        (systemd:service9):   Started NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:      SERVICE10        (systemd:service10):        Started NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:      SERVICE11       (systemd:service11):       Started NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:      SERVICE12   (systemd:service12):       Started NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:      SERVICE13  (systemd:service13):       Started NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: clone_print:        Clone Set: SERVICE14-clone [SERVICE14]
Aug 27 08:47:02 [1330] NODE2    pengine:     info: short_print:            Started: [ NODE2 ]
Aug 27 08:47:02 [1330] NODE2    pengine:     info: short_print:            Stopped: [ NODE1 ]
Aug 27 08:47:02 [1330] NODE2    pengine:     info: clone_print:        Clone Set: SERVICE15-clone [SERVICE15]
Aug 27 08:47:02 [1330] NODE2    pengine:     info: short_print:            Started: [ NODE2 ]
Aug 27 08:47:02 [1330] NODE2    pengine:     info: short_print:            Stopped: [ NODE1 ]
Aug 27 08:47:02 [1330] NODE2    pengine:    error: native_create_actions:     Resource SERVICE1 is active on 2 nodes (attempting recovery)
Aug 27 08:47:02 [1330] NODE2    pengine:   notice: native_create_actions:     See https://wiki.clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information
Aug 27 08:47:02 [1330] NODE2    pengine:     info: RecurringOp:        Start recurring monitor (5s) for SERVICE1 on NODE2
Aug 27 08:47:02 [1330] NODE2    pengine:     info: RecurringOp:        Start recurring monitor (10s) for SERVICE14:1 on NODE1
Aug 27 08:47:02 [1330] NODE2    pengine:     info: RecurringOp:        Start recurring monitor (10s) for SERVICE15:1 on NODE1
Aug 27 08:47:02 [1330] NODE2    pengine:     info: LogActions:        Leave   VIRTUALIP       (Started NODE2)
Aug 27 08:47:02 [1330] NODE2    pengine:     info: LogActions:        Leave   SOURCEIP        (Started NODE2)
Aug 27 08:47:02 [1330] NODE2    pengine:   notice: LogAction:  * Move       SERVICE1                 ( NODE1 -> NODE2 )
Aug 27 08:47:02 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE2    (Started NODE2)
Aug 27 08:47:02 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE3       (Started NODE2)
Aug 27 08:47:02 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE4       (Started NODE2)
Aug 27 08:47:02 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE5      (Started NODE2)
Aug 27 08:47:02 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE6 (Started NODE2)
Aug 27 08:47:02 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE7  (Started NODE2)
Aug 27 08:47:02 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE8   (Started NODE2)
Aug 27 08:47:02 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE9        (Started NODE2)
Aug 27 08:47:02 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE10        (Started NODE2)
Aug 27 08:47:02 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE11       (Started NODE2)
Aug 27 08:47:02 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE12   (Started NODE2)
Aug 27 08:47:02 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE13  (Started NODE2)
Aug 27 08:47:02 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE14:0        (Started NODE2)
Aug 27 08:47:02 [1330] NODE2    pengine:   notice: LogAction:  * Start      SERVICE14:1         (            NODE1 )
Aug 27 08:47:02 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE15:0 (Started NODE2)
Aug 27 08:47:02 [1330] NODE2    pengine:   notice: LogAction:  * Start      SERVICE15:1                  (            NODE1 )
Aug 27 08:47:02 [1330] NODE2    pengine:    error: process_pe_message:        Calculated transition 334 (with errors), saving inputs in /var/lib/pacemaker/pengine/pe-error-171.bz2
Aug 27 08:47:02 [1331] NODE2       crmd:     info: do_state_transition:       State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Aug 27 08:47:02 [1331] NODE2       crmd:     info: do_te_invoke:      Processing graph 334 (ref=pe_calc-dc-1598510822-991) derived from /var/lib/pacemaker/pengine/pe-error-171.bz2
Aug 27 08:47:02 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating stop operation SERVICE1_stop_0 locally on NODE2 | action 36
Aug 27 08:47:02 [1328] NODE2       lrmd:     info: cancel_recurring_action:   Cancelling systemd operation SERVICE1_status_5000
Aug 27 08:47:02 [1331] NODE2       crmd:     info: do_lrm_rsc_op:     Performing key=36:334:0:d10dd5e7-af4d-4bba-a226-516824f8f60e op=SERVICE1_stop_0
Aug 27 08:47:02 [1328] NODE2       lrmd:     info: log_execute:       executing - rsc:SERVICE1 action:stop call_id:428
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1398)
Aug 27 08:47:02 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating stop operation SERVICE1_stop_0 on NODE1 | action 35
Aug 27 08:47:02 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE7_monitor_0 on NODE1 | action 18
Aug 27 08:47:02 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE8_monitor_0 on NODE1 | action 19
Aug 27 08:47:02 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE9_monitor_0 on NODE1 | action 20
Aug 27 08:47:02 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE10_monitor_0 on NODE1 | action 21
Aug 27 08:47:02 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE11_monitor_0 on NODE1 | action 22
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.40 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.41 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=41
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']:  @crm-debug-origin=do_update_resource
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='SERVICE1']/lrm_rsc_op[@id='SERVICE1_last_0']:  @operation_key=SERVICE1_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=36:334:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @transition-magic=-1:193;36:334:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1598510822, @last-rc-change=1598510822, @
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1398, version=0.434.41)
Aug 27 08:47:02 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE12_monitor_0 on NODE1 | action 23
Aug 27 08:47:02 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE13_monitor_0 on NODE1 | action 24
Aug 27 08:47:02 [1331] NODE2       crmd:     info: process_lrm_event: Result of monitor operation for SERVICE1 on NODE2: Cancelled | call=426 key=SERVICE1_monitor_5000 confirmed=true
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.41 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.42 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=42
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE1']/lrm_rsc_op[@id='SERVICE1_last_0']:  @operation_key=SERVICE1_stop_0, @operation=stop, @transition-key=35:334:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @transition-magic=-1:193;35:334:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/28, version=0.434.42)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.42 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.43 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=43
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="SERVICE7" type="service7" class="systemd"/>
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="SERVICE7_last_0" operation_key="SERVICE7_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="18:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;18:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" op-status="-1" int
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/29, version=0.434.43)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.43 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.44 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=44
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="SERVICE8" type="service8" class="systemd"/>
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="SERVICE8_last_0" operation_key="SERVICE8_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="19:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;19:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" op-status="-1" inter
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/30, version=0.434.44)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.44 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.45 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=45
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="SERVICE9" type="service9" class="systemd"/>
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="SERVICE9_last_0" operation_key="SERVICE9_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="20:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;20:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" op-status="-1" interval="0" la
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/31, version=0.434.45)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.45 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.46 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=46
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="SERVICE10" type="service10" class="systemd"/>
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="SERVICE10_last_0" operation_key="SERVICE10_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="21:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;21:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" op-status="-1" interval="0" la
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/32, version=0.434.46)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.46 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.47 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=47
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="SERVICE11" type="service11" class="systemd"/>
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="SERVICE11_last_0" operation_key="SERVICE11_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="22:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;22:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" op-status="-1" interval="0"
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/33, version=0.434.47)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.47 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.48 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=48
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="SERVICE12" type="service12" class="systemd"/>
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="SERVICE12_last_0" operation_key="SERVICE12_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="23:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;23:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" op-status="-1" inter
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/34, version=0.434.48)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.48 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.49 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=49
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="SERVICE13" type="service13" class="systemd"/>
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="SERVICE13_last_0" operation_key="SERVICE13_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="24:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;24:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" op-status="-1" interval="0" last-r
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/35, version=0.434.49)
Aug 27 08:47:02 [1328] NODE2       lrmd:     info: systemd_exec_result:       Call to stop passed: /org/freedesktop/systemd1/job/42670
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.49 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.50 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=50
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE7']/lrm_rsc_op[@id='SERVICE7_last_0']:  @transition-magic=0:7;18:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=38, @rc-code=7, @op-status=0, @exec-time=125
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/36, version=0.434.50)
Aug 27 08:47:02 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE7_monitor_0 (18) confirmed on NODE1 (rc=7)
Aug 27 08:47:02 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE14:1_monitor_0 on NODE1 | action 25
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.50 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.51 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=51
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE8']/lrm_rsc_op[@id='SERVICE8_last_0']:  @transition-magic=0:7;19:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=42, @rc-code=7, @op-status=0, @exec-time=123, @queue-time=1
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/37, version=0.434.51)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.51 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.52 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=52
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE9']/lrm_rsc_op[@id='SERVICE9_last_0']:  @transition-magic=0:0;20:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=46, @rc-code=0, @op-status=0, @exec-time=122
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE9']:  <lrm_rsc_op id="SERVICE9_last_failure_0" operation_key="SERVICE9_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="20:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="0:0;20:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="46" rc-code="0"
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/38, version=0.434.52)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.52 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.53 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=53
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE10']/lrm_rsc_op[@id='SERVICE10_last_0']:  @transition-magic=0:7;21:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=50, @rc-code=7, @op-status=0, @exec-time=121
Aug 27 08:47:02 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE8_monitor_0 (19) confirmed on NODE1 (rc=7)
Aug 27 08:47:02 [1331] NODE2       crmd:  warning: status_from_rc:    Action 20 (SERVICE9_monitor_0) on NODE1 failed (target: 7 vs. rc: 0): Error
Aug 27 08:47:02 [1331] NODE2       crmd:   notice: abort_transition_graph:    Transition aborted by operation SERVICE9_monitor_0 'modify' on NODE1: Event failed | magic=0:0;20:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e cib=0.434.52 source=match_graph_event:299 complete=false
Aug 27 08:47:02 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE9_monitor_0 (20) confirmed on NODE1 (rc=0)
Aug 27 08:47:02 [1331] NODE2       crmd:     info: process_graph_event:       Detected action (334.20) SERVICE9_monitor_0.46=ok: failed
Aug 27 08:47:02 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE10_monitor_0 (21) confirmed on NODE1 (rc=7)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/39, version=0.434.53)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.53 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.54 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=54
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE11']/lrm_rsc_op[@id='SERVICE11_last_0']:  @transition-magic=0:7;22:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=54, @rc-code=7, @op-status=0, @exec-time=118
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/40, version=0.434.54)
Aug 27 08:47:02 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE11_monitor_0 (22) confirmed on NODE1 (rc=7)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.54 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.55 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=55
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE12']/lrm_rsc_op[@id='SERVICE12_last_0']:  @transition-magic=0:7;23:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=58, @rc-code=7, @op-status=0, @exec-time=117
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/41, version=0.434.55)
Aug 27 08:47:02 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE12_monitor_0 (23) confirmed on NODE1 (rc=7)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.55 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.56 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=56
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE13']/lrm_rsc_op[@id='SERVICE13_last_0']:  @transition-magic=0:7;24:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=62, @rc-code=7, @op-status=0, @exec-time=118, @queue-time=1
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/42, version=0.434.56)
Aug 27 08:47:02 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE13_monitor_0 (24) confirmed on NODE1 (rc=7)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.56 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.57 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=57
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="SERVICE14" type="service14" class="systemd"/>
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="SERVICE14_last_0" operation_key="SERVICE14_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="25:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;25:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" op-status="-1" int
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/43, version=0.434.57)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.57 2
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.58 (null)
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=58
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE14']/lrm_rsc_op[@id='SERVICE14_last_0']:  @transition-magic=0:7;25:334:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=67, @rc-code=7, @op-status=0, @exec-time=46
Aug 27 08:47:02 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/44, version=0.434.58)
Aug 27 08:47:02 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE14_monitor_0 (25) confirmed on NODE1 (rc=7)
Aug 27 08:47:04 [1331] NODE2       crmd:   notice: process_lrm_event: Result of stop operation for SERVICE1 on NODE2: 0 (ok) | call=428 key=SERVICE1_stop_0 confirmed=true cib-update=1399
Aug 27 08:47:04 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1399)
Aug 27 08:47:04 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.58 2
Aug 27 08:47:04 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.59 (null)
Aug 27 08:47:04 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=59
Aug 27 08:47:04 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='SERVICE1']/lrm_rsc_op[@id='SERVICE1_last_0']:  @transition-magic=0:0;36:334:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=428, @rc-code=0, @op-status=0, @exec-time=2072
Aug 27 08:47:04 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1399, version=0.434.59)
Aug 27 08:47:04 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE1_stop_0 (36) confirmed on NODE2 (rc=0)
Aug 27 08:47:09 [1326] NODE2        cib:     info: cib_process_ping:  Reporting our current digest to NODE2: dff1c3d3f4cfcb4d59dd341bb0268506 for 0.434.59 (0x563b7430a810 0)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.59 2
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.60 (null)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=60
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE1']/lrm_rsc_op[@id='SERVICE1_last_0']:  @transition-magic=2:198;35:334:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=34, @rc-code=198, @op-status=2, @exec-time=19986, @queue-time=1
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE1']/lrm_rsc_op[@id='SERVICE1_last_failure_0']:  @operation_key=SERVICE1_stop_0, @operation=stop, @transition-key=35:334:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @transition-magic=2:198;35:334:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=34, @rc-code=198, @op-status=2, @exec-time=19986, @queue-time=1
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/45, version=0.434.60)
Aug 27 08:47:22 [1331] NODE2       crmd:  warning: status_from_rc:    Action 35 (SERVICE1_stop_0) on NODE1 failed (target: 0 vs. rc: 198): Error
Aug 27 08:47:22 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE1_stop_0 (35) confirmed on NODE1 (rc=198, ignoring failure)
Aug 27 08:47:22 [1331] NODE2       crmd:     info: update_failcount:  Updating last failure for SERVICE1 on NODE1 after failed stop: rc=198 (update=INFINITY, time=1598510842)
Aug 27 08:47:22 [1331] NODE2       crmd:     info: process_graph_event:       Detected action (334.35) SERVICE1_stop_0.34=OCF_TIMEOUT: failed
Aug 27 08:47:22 [1331] NODE2       crmd:   notice: run_graph: Transition 334 (Complete=11, Pending=0, Fired=0, Skipped=3, Incomplete=10, Source=/var/lib/pacemaker/pengine/pe-error-171.bz2): Stopped
Aug 27 08:47:22 [1331] NODE2       crmd:     info: do_state_transition:       State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
Aug 27 08:47:22 [1329] NODE2      attrd:     info: attrd_peer_update: Setting last-failure-SERVICE1#stop_0[NODE1]: (null) -> 1598510842 from NODE2
Aug 27 08:47:22 [1329] NODE2      attrd:     info: write_attribute:   Sent CIB request 173 with 2 changes for last-failure-SERVICE1#stop_0 (id n/a, set n/a)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/attrd/173)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.60 2
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.61 (null)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=61
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']:  <transient_attributes id="1"/>
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    ++                                     <instance_attributes id="status-1">
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    ++                                       <nvpair id="status-1-last-failure-SERVICE1.stop_0" name="last-failure-SERVICE1#stop_0" value="1598510842"/>
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    ++                                     </instance_attributes>
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    ++                                   </transient_attributes>
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/attrd/173, version=0.434.61)
Aug 27 08:47:22 [1329] NODE2      attrd:     info: attrd_cib_callback:        CIB update 173 result for last-failure-SERVICE1#stop_0: OK | rc=0
Aug 27 08:47:22 [1329] NODE2      attrd:     info: attrd_cib_callback:        * last-failure-SERVICE1#stop_0[NODE2]=(null)
Aug 27 08:47:22 [1329] NODE2      attrd:     info: attrd_cib_callback:        * last-failure-SERVICE1#stop_0[NODE1]=1598510842
Aug 27 08:47:22 [1330] NODE2    pengine:   notice: unpack_config:     On loss of CCM Quorum: Ignore
Aug 27 08:47:22 [1331] NODE2       crmd:     info: abort_transition_graph:    Transition aborted by transient_attributes.1 'create': Transient attribute change | cib=0.434.61 source=abort_unless_down:356 path=/cib/status/node_state[@id='1'] complete=true
Aug 27 08:47:22 [1330] NODE2    pengine:     info: determine_online_status:   Node NODE1 is online
Aug 27 08:47:22 [1330] NODE2    pengine:     info: determine_online_status:   Node NODE2 is online
Aug 27 08:47:22 [1330] NODE2    pengine:  warning: unpack_rsc_op:     Pretending the failure of SERVICE1_stop_0 (rc=198) on NODE1 succeeded
Aug 27 08:47:22 [1330] NODE2    pengine:  warning: unpack_rsc_op:     Pretending the failure of SERVICE1_stop_0 (rc=198) on NODE1 succeeded
Aug 27 08:47:22 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE9 active on NODE1
Aug 27 08:47:22 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE9 active on NODE1
Aug 27 08:47:22 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE4 active on NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE1 active on NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 1 is already processed
Aug 27 08:47:22 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 2 is already processed
Aug 27 08:47:22 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 1 is already processed
Aug 27 08:47:22 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 2 is already processed
Aug 27 08:47:22 [1330] NODE2    pengine:     info: group_print:        Resource Group: IPV
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:           VIRTUALIP  (ocf::heartbeat:IPaddr2):       Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:           SOURCEIP   (ocf::heartbeat:IPsrcaddr):      Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE1        (systemd:service1):     Stopped (failure ignored)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE2    (systemd:service2): Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE3       (systemd:service3):  Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE4       (systemd:service4):  Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE5      (systemd:service5): Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE6 (systemd:service6):    Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE7  (systemd:service7):      Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE8   (systemd:service8):  Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE9        (systemd:service9):   Started
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:              1 : NODE1
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:              2 : NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE10        (systemd:service10):        Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE11       (systemd:service11):       Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE12   (systemd:service12):       Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE13  (systemd:service13):       Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: clone_print:        Clone Set: SERVICE14-clone [SERVICE14]
Aug 27 08:47:22 [1330] NODE2    pengine:     info: short_print:            Started: [ NODE2 ]
Aug 27 08:47:22 [1330] NODE2    pengine:     info: short_print:            Stopped: [ NODE1 ]
Aug 27 08:47:22 [1330] NODE2    pengine:     info: clone_print:        Clone Set: SERVICE15-clone [SERVICE15]
Aug 27 08:47:22 [1330] NODE2    pengine:     info: short_print:            Started: [ NODE2 ]
Aug 27 08:47:22 [1330] NODE2    pengine:     info: short_print:            Stopped: [ NODE1 ]
Aug 27 08:47:22 [1330] NODE2    pengine:     info: RecurringOp:        Start recurring monitor (5s) for SERVICE1 on NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:    error: native_create_actions:     Resource SERVICE9 is active on 2 nodes (attempting recovery)
Aug 27 08:47:22 [1330] NODE2    pengine:   notice: native_create_actions:     See https://wiki.clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information
Aug 27 08:47:22 [1330] NODE2    pengine:     info: RecurringOp:        Start recurring monitor (5s) for SERVICE9 on NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: RecurringOp:        Start recurring monitor (10s) for SERVICE14:1 on NODE1
Aug 27 08:47:22 [1330] NODE2    pengine:     info: RecurringOp:        Start recurring monitor (10s) for SERVICE15:1 on NODE1
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   VIRTUALIP       (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SOURCEIP        (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:   notice: LogAction:  * Start      SERVICE1                 (            NODE2 )
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE2    (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE3       (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE4       (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE5      (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE6 (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE7  (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE8   (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:   notice: LogAction:  * Move       SERVICE9                 ( NODE1 -> NODE2 )
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE10        (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE11       (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE12   (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE13  (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE14:0        (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:   notice: LogAction:  * Start      SERVICE14:1         (            NODE1 )
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE15:0 (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:   notice: LogAction:  * Start      SERVICE15:1                  (            NODE1 )
Aug 27 08:47:22 [1330] NODE2    pengine:    error: process_pe_message:        Calculated transition 335 (with errors), saving inputs in /var/lib/pacemaker/pengine/pe-error-172.bz2
Aug 27 08:47:22 [1331] NODE2       crmd:     info: handle_response:   pe_calc calculation pe_calc-dc-1598510842-1002 is obsolete
Aug 27 08:47:22 [1330] NODE2    pengine:   notice: unpack_config:     On loss of CCM Quorum: Ignore
Aug 27 08:47:22 [1330] NODE2    pengine:     info: determine_online_status:   Node NODE1 is online
Aug 27 08:47:22 [1330] NODE2    pengine:     info: determine_online_status:   Node NODE2 is online
Aug 27 08:47:22 [1330] NODE2    pengine:  warning: unpack_rsc_op:     Pretending the failure of SERVICE1_stop_0 (rc=198) on NODE1 succeeded
Aug 27 08:47:22 [1330] NODE2    pengine:  warning: unpack_rsc_op:     Pretending the failure of SERVICE1_stop_0 (rc=198) on NODE1 succeeded
Aug 27 08:47:22 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE9 active on NODE1
Aug 27 08:47:22 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE9 active on NODE1
Aug 27 08:47:22 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE4 active on NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE1 active on NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 1 is already processed
Aug 27 08:47:22 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 2 is already processed
Aug 27 08:47:22 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 1 is already processed
Aug 27 08:47:22 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 2 is already processed
Aug 27 08:47:22 [1330] NODE2    pengine:     info: group_print:        Resource Group: IPV
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:           VIRTUALIP  (ocf::heartbeat:IPaddr2):       Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:           SOURCEIP   (ocf::heartbeat:IPsrcaddr):      Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE1        (systemd:service1):     Stopped (failure ignored)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE2    (systemd:service2): Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE3       (systemd:service3):  Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE4       (systemd:service4):  Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE5      (systemd:service5): Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE6 (systemd:service6):    Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE7  (systemd:service7):      Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE8   (systemd:service8):  Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE9        (systemd:service9):   Started
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:              1 : NODE1
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:              2 : NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE10        (systemd:service10):        Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE11       (systemd:service11):       Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE12   (systemd:service12):       Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: common_print:      SERVICE13  (systemd:service13):       Started NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: clone_print:        Clone Set: SERVICE14-clone [SERVICE14]
Aug 27 08:47:22 [1330] NODE2    pengine:     info: short_print:            Started: [ NODE2 ]
Aug 27 08:47:22 [1330] NODE2    pengine:     info: short_print:            Stopped: [ NODE1 ]
Aug 27 08:47:22 [1330] NODE2    pengine:     info: clone_print:        Clone Set: SERVICE15-clone [SERVICE15]
Aug 27 08:47:22 [1330] NODE2    pengine:     info: short_print:            Started: [ NODE2 ]
Aug 27 08:47:22 [1330] NODE2    pengine:     info: short_print:            Stopped: [ NODE1 ]
Aug 27 08:47:22 [1330] NODE2    pengine:     info: RecurringOp:        Start recurring monitor (5s) for SERVICE1 on NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:    error: native_create_actions:     Resource SERVICE9 is active on 2 nodes (attempting recovery)
Aug 27 08:47:22 [1330] NODE2    pengine:   notice: native_create_actions:     See https://wiki.clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information
Aug 27 08:47:22 [1330] NODE2    pengine:     info: RecurringOp:        Start recurring monitor (5s) for SERVICE9 on NODE2
Aug 27 08:47:22 [1330] NODE2    pengine:     info: RecurringOp:        Start recurring monitor (10s) for SERVICE14:1 on NODE1
Aug 27 08:47:22 [1330] NODE2    pengine:     info: RecurringOp:        Start recurring monitor (10s) for SERVICE15:1 on NODE1
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   VIRTUALIP       (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SOURCEIP        (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:   notice: LogAction:  * Start      SERVICE1                 (            NODE2 )
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE2    (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE3       (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE4       (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE5      (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE6 (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE7  (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE8   (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:   notice: LogAction:  * Move       SERVICE9                 ( NODE1 -> NODE2 )
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE10        (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE11       (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE12   (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE13  (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE14:0        (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:   notice: LogAction:  * Start      SERVICE14:1         (            NODE1 )
Aug 27 08:47:22 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE15:0 (Started NODE2)
Aug 27 08:47:22 [1330] NODE2    pengine:   notice: LogAction:  * Start      SERVICE15:1                  (            NODE1 )
Aug 27 08:47:22 [1330] NODE2    pengine:    error: process_pe_message:        Calculated transition 336 (with errors), saving inputs in /var/lib/pacemaker/pengine/pe-error-173.bz2
Aug 27 08:47:22 [1331] NODE2       crmd:     info: do_state_transition:       State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Aug 27 08:47:22 [1331] NODE2       crmd:     info: do_te_invoke:      Processing graph 336 (ref=pe_calc-dc-1598510842-1003) derived from /var/lib/pacemaker/pengine/pe-error-173.bz2
Aug 27 08:47:22 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating start operation SERVICE1_start_0 locally on NODE2 | action 26
Aug 27 08:47:22 [1331] NODE2       crmd:     info: do_lrm_rsc_op:     Performing key=26:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e op=SERVICE1_start_0
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1402)
Aug 27 08:47:22 [1328] NODE2       lrmd:     info: log_execute:       executing - rsc:SERVICE1 action:start call_id:429
Aug 27 08:47:22 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating stop operation SERVICE9_stop_0 locally on NODE2 | action 43
Aug 27 08:47:22 [1328] NODE2       lrmd:     info: cancel_recurring_action:   Cancelling systemd operation SERVICE9_status_5000
Aug 27 08:47:22 [1331] NODE2       crmd:     info: do_lrm_rsc_op:     Performing key=43:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e op=SERVICE9_stop_0
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1403)
Aug 27 08:47:22 [1328] NODE2       lrmd:     info: log_execute:       executing - rsc:SERVICE9 action:stop call_id:431
Aug 27 08:47:22 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating stop operation SERVICE9_stop_0 on NODE1 | action 42
Aug 27 08:47:22 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE15:1_monitor_0 on NODE1 | action 17
Aug 27 08:47:22 [1331] NODE2       crmd:     info: process_lrm_event: Result of monitor operation for SERVICE9 on NODE2: Cancelled | call=400 key=SERVICE9_monitor_5000 confirmed=true
Aug 27 08:47:22 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating start operation SERVICE14:1_start_0 on NODE1 | action 55
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.61 2
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.62 (null)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=62
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='SERVICE1']/lrm_rsc_op[@id='SERVICE1_last_0']:  @operation_key=SERVICE1_start_0, @operation=start, @transition-key=26:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @transition-magic=-1:193;26:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1598510842, @last-rc-change=1598510842, @exec-time=0
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1402, version=0.434.62)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.62 2
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.63 (null)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=63
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='SERVICE9']/lrm_rsc_op[@id='SERVICE9_last_0']:  @operation_key=SERVICE9_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=43:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @transition-magic=-1:193;43:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1598510842, @last-rc-change=1598510842, @
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1403, version=0.434.63)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.63 2
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.64 (null)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=64
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE9']/lrm_rsc_op[@id='SERVICE9_last_0']:  @operation_key=SERVICE9_stop_0, @operation=stop, @transition-key=42:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @transition-magic=-1:193;42:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1598510864, @last-rc-change=1598510864, @exec-time=0
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/46, version=0.434.64)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.64 2
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.65 (null)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=65
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="SERVICE15" type="service15" class="systemd"/>
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                                <lrm_rsc_op id="SERVICE15_last_0" operation_key="SERVICE15_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="17:336:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;17:336:7:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" op-status="-1" interval="0" last-run
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    ++                                                              </lrm_resource>
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/47, version=0.434.65)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.65 2
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.66 (null)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=66
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE14']/lrm_rsc_op[@id='SERVICE14_last_0']:  @operation_key=SERVICE14_start_0, @operation=start, @transition-key=55:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @transition-magic=-1:193;55:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1598510864, @last-rc-change=1598510864, @exec-time=0
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/48, version=0.434.66)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.66 2
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.67 (null)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=67
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE15']/lrm_rsc_op[@id='SERVICE15_last_0']:  @transition-magic=0:7;17:336:7:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=73, @rc-code=7, @op-status=0, @exec-time=34
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/49, version=0.434.67)
Aug 27 08:47:22 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE15_monitor_0 (17) confirmed on NODE1 (rc=7)
Aug 27 08:47:22 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating start operation SERVICE15:1_start_0 on NODE1 | action 63
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.67 2
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.68 (null)
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=68
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE15']/lrm_rsc_op[@id='SERVICE15_last_0']:  @operation_key=SERVICE15_start_0, @operation=start, @transition-key=63:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @transition-magic=-1:193;63:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0
Aug 27 08:47:22 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/50, version=0.434.68)
Aug 27 08:47:22 [1328] NODE2       lrmd:     info: systemd_exec_result:       Call to start passed: /org/freedesktop/systemd1/job/42671
Aug 27 08:47:22 [1328] NODE2       lrmd:     info: systemd_exec_result:       Call to stop passed: /org/freedesktop/systemd1/job/42812
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.68 2
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.69 (null)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=69
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE9']/lrm_rsc_op[@id='SERVICE9_last_0']:  @transition-magic=0:0;42:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=68, @rc-code=0, @op-status=0, @exec-time=2006
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/51, version=0.434.69)
Aug 27 08:47:24 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE9_stop_0 (42) confirmed on NODE1 (rc=0)
Aug 27 08:47:24 [1331] NODE2       crmd:   notice: process_lrm_event: Result of start operation for SERVICE1 on NODE2: 0 (ok) | call=429 key=SERVICE1_start_0 confirmed=true cib-update=1404
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1404)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.69 2
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.70 (null)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=70
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='SERVICE1']/lrm_rsc_op[@id='SERVICE1_last_0']:  @transition-magic=0:0;26:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=429, @rc-code=0, @op-status=0, @exec-time=2063
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1404, version=0.434.70)
Aug 27 08:47:24 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE1_start_0 (26) confirmed on NODE2 (rc=0)
Aug 27 08:47:24 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE1_monitor_5000 locally on NODE2 | action 27
Aug 27 08:47:24 [1331] NODE2       crmd:     info: do_lrm_rsc_op:     Performing key=27:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e op=SERVICE1_monitor_5000
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1405)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.70 2
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.71 (null)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=71
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='SERVICE1']/lrm_rsc_op[@id='SERVICE1_monitor_5000']:  @crm-debug-origin=do_update_resource, @transition-key=27:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @transition-magic=-1:193;27:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1598510844, @exec-time=0, @queue-time=0
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1405, version=0.434.71)
Aug 27 08:47:24 [1331] NODE2       crmd:     info: process_lrm_event: Result of monitor operation for SERVICE1 on NODE2: 0 (ok) | call=432 key=SERVICE1_monitor_5000 confirmed=false cib-update=1406
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1406)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.71 2
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.72 (null)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=72
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='SERVICE1']/lrm_rsc_op[@id='SERVICE1_monitor_5000']:  @transition-magic=0:0;27:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=432, @rc-code=0, @op-status=0, @exec-time=2
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1406, version=0.434.72)
Aug 27 08:47:24 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE1_monitor_5000 (27) confirmed on NODE2 (rc=0)
Aug 27 08:47:24 [1331] NODE2       crmd:   notice: process_lrm_event: Result of stop operation for SERVICE9 on NODE2: 0 (ok) | call=431 key=SERVICE9_stop_0 confirmed=true cib-update=1407
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1407)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.72 2
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.73 (null)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=73
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='SERVICE9']/lrm_rsc_op[@id='SERVICE9_last_0']:  @transition-magic=0:0;43:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=431, @rc-code=0, @op-status=0, @exec-time=2194
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1407, version=0.434.73)
Aug 27 08:47:24 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE9_stop_0 (43) confirmed on NODE2 (rc=0)
Aug 27 08:47:24 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating start operation SERVICE9_start_0 locally on NODE2 | action 44
Aug 27 08:47:24 [1331] NODE2       crmd:     info: do_lrm_rsc_op:     Performing key=44:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e op=SERVICE9_start_0
Aug 27 08:47:24 [1328] NODE2       lrmd:     info: log_execute:       executing - rsc:SERVICE9 action:start call_id:433
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1408)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.73 2
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.74 (null)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=74
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='SERVICE9']/lrm_rsc_op[@id='SERVICE9_last_0']:  @operation_key=SERVICE9_start_0, @operation=start, @transition-key=44:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @transition-magic=-1:193;44:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1598510844, @last-rc-change=1598510844, @exec-time=0
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1408, version=0.434.74)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.74 2
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.75 (null)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=75
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE14']/lrm_rsc_op[@id='SERVICE14_last_0']:  @transition-magic=0:0;55:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=74, @rc-code=0, @op-status=0, @exec-time=2209
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/52, version=0.434.75)
Aug 27 08:47:24 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE14_start_0 (55) confirmed on NODE1 (rc=0)
Aug 27 08:47:24 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE14:1_monitor_10000 on NODE1 | action 56
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.75 2
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.76 (null)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=76
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE14']:  <lrm_rsc_op id="SERVICE14_monitor_10000" operation_key="SERVICE14_monitor_10000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="56:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;56:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" c
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/53, version=0.434.76)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.76 2
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.77 (null)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=77
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE14']/lrm_rsc_op[@id='SERVICE14_monitor_10000']:  @transition-magic=0:0;56:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=76, @rc-code=0, @op-status=0, @exec-time=2
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/54, version=0.434.77)
Aug 27 08:47:24 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE14_monitor_10000 (56) confirmed on NODE1 (rc=0)
Aug 27 08:47:24 [1328] NODE2       lrmd:     info: systemd_exec_result:       Call to start passed: /org/freedesktop/systemd1/job/42813
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.77 2
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.78 (null)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=78
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE15']/lrm_rsc_op[@id='SERVICE15_last_0']:  @transition-magic=0:0;63:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=75, @rc-code=0, @op-status=0, @exec-time=2316
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/55, version=0.434.78)
Aug 27 08:47:24 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE15_start_0 (63) confirmed on NODE1 (rc=0)
Aug 27 08:47:24 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE15:1_monitor_10000 on NODE1 | action 64
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.78 2
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.79 (null)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=79
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE15']:  <lrm_rsc_op id="SERVICE15_monitor_10000" operation_key="SERVICE15_monitor_10000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="64:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" transition-magic="-1:193;64:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e" exit-reason="" on_node="NODE1" call-id="-1" rc-code="193" o
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/56, version=0.434.79)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.79 2
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.80 (null)
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=80
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE15']/lrm_rsc_op[@id='SERVICE15_monitor_10000']:  @transition-magic=0:0;64:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=77, @rc-code=0, @op-status=0, @exec-time=1
Aug 27 08:47:24 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE1/crmd/57, version=0.434.80)
Aug 27 08:47:24 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE15_monitor_10000 (64) confirmed on NODE1 (rc=0)
Aug 27 08:47:26 [1331] NODE2       crmd:   notice: process_lrm_event: Result of start operation for SERVICE9 on NODE2: 0 (ok) | call=433 key=SERVICE9_start_0 confirmed=true cib-update=1409
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1409)
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.80 2
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.81 (null)
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=81
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='SERVICE9']/lrm_rsc_op[@id='SERVICE9_last_0']:  @transition-magic=0:0;44:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=433, @rc-code=0, @op-status=0, @exec-time=2079, @queue-time=1
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1409, version=0.434.81)
Aug 27 08:47:26 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE9_start_0 (44) confirmed on NODE2 (rc=0)
Aug 27 08:47:26 [1331] NODE2       crmd:   notice: te_rsc_command:    Initiating monitor operation SERVICE9_monitor_5000 locally on NODE2 | action 2
Aug 27 08:47:26 [1331] NODE2       crmd:     info: do_lrm_rsc_op:     Performing key=2:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e op=SERVICE9_monitor_5000
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1410)
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.81 2
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.82 (null)
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=82
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='SERVICE9']/lrm_rsc_op[@id='SERVICE9_monitor_5000']:  @crm-debug-origin=do_update_resource, @transition-key=2:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @transition-magic=-1:193;2:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1598510846, @exec-time=0
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1410, version=0.434.82)
Aug 27 08:47:26 [1331] NODE2       crmd:     info: process_lrm_event: Result of monitor operation for SERVICE9 on NODE2: 0 (ok) | call=434 key=SERVICE9_monitor_5000 confirmed=false cib-update=1411
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/crmd/1411)
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.82 2
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.83 (null)
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=83
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='SERVICE9']/lrm_rsc_op[@id='SERVICE9_monitor_5000']:  @transition-magic=0:0;2:336:0:d10dd5e7-af4d-4bba-a226-516824f8f60e, @call-id=434, @rc-code=0, @op-status=0, @exec-time=5
Aug 27 08:47:26 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/crmd/1411, version=0.434.83)
Aug 27 08:47:26 [1331] NODE2       crmd:     info: match_graph_event: Action SERVICE9_monitor_5000 (2) confirmed on NODE2 (rc=0)
Aug 27 08:47:26 [1331] NODE2       crmd:   notice: run_graph: Transition 336 (Complete=15, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-error-173.bz2): Complete
Aug 27 08:47:26 [1331] NODE2       crmd:     info: do_log:    Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
Aug 27 08:47:26 [1331] NODE2       crmd:   notice: do_state_transition:       State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Aug 27 08:47:27 [1329] NODE2      attrd:     info: attrd_peer_update: Setting last-failure-SERVICE1#stop_0[NODE1]: 1598510842 -> (null) from NODE2
Aug 27 08:47:27 [1329] NODE2      attrd:     info: write_attribute:   Sent CIB request 174 with 2 changes for last-failure-SERVICE1#stop_0 (id n/a, set n/a)
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_process_request:       Forwarding cib_modify operation for section status to all (origin=local/attrd/174)
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.83 2
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.84 b548502a016f55db1c8a902a6483c308
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_perform_op:    -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE1']
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=84
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_delete operation for section //node_state[@uname='NODE1'] /lrm/lrm_resources/lrm_resource[@id='SERVICE1']: OK (rc=0, origin=NODE1/crmd/58, version=0.434.83)
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.83 2
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.84 (null)
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_perform_op:    -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-SERVICE1.stop_0']
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=84
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section status: OK (rc=0, origin=NODE2/attrd/174, version=0.434.84)
Aug 27 08:47:27 [1329] NODE2      attrd:     info: attrd_cib_callback:        CIB update 174 result for last-failure-SERVICE1#stop_0: OK | rc=0
Aug 27 08:47:27 [1329] NODE2      attrd:     info: attrd_cib_callback:        * last-failure-SERVICE1#stop_0[NODE2]=(null)
Aug 27 08:47:27 [1329] NODE2      attrd:     info: attrd_cib_callback:        * last-failure-SERVICE1#stop_0[NODE1]=(null)
Aug 27 08:47:27 [1331] NODE2       crmd:     info: abort_transition_graph:    Transition aborted by deletion of nvpair[@id='status-1-last-failure-SERVICE1.stop_0']: Transient attribute change | cib=0.434.84 source=abort_unless_down:370 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-SERVICE1.stop_0'] complete=true
Aug 27 08:47:27 [1331] NODE2       crmd:   notice: do_state_transition:       State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.84 2
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.434.85 (null)
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_perform_op:    -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE1']
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @num_updates=85
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_delete operation for section //node_state[@uname='NODE1'] /lrm/lrm_resources/lrm_resource[@id='SERVICE1']: OK (rc=0, origin=NODE1/crmd/59, version=0.434.85)
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_perform_op:    Diff: --- 0.434.85 2
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_perform_op:    Diff: +++ 0.435.0 (null)
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib:  @epoch=435, @num_updates=0
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_perform_op:    +  /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']:  @value=1598510869
Aug 27 08:47:27 [1326] NODE2        cib:     info: cib_process_request:       Completed cib_modify operation for section crm_config: OK (rc=0, origin=NODE1/crmd/61, version=0.435.0)
Aug 27 08:47:27 [1331] NODE2       crmd:     info: abort_transition_graph:    Transition aborted by deletion of lrm_resource[@id='SERVICE1']: Resource state removal | cib=0.434.85 source=abort_unless_down:370 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SERVICE1'] complete=true
Aug 27 08:47:27 [1331] NODE2       crmd:     info: abort_transition_graph:    Transition aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1598510869: Configuration change | cib=0.435.0 source=te_update_diff_v2:522 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=true
Aug 27 08:47:27 [1330] NODE2    pengine:   notice: unpack_config:     On loss of CCM Quorum: Ignore
Aug 27 08:47:27 [1330] NODE2    pengine:     info: determine_online_status:   Node NODE1 is online
Aug 27 08:47:27 [1330] NODE2    pengine:     info: determine_online_status:   Node NODE2 is online
Aug 27 08:47:27 [1330] NODE2    pengine:  warning: unpack_rsc_op:     Pretending the failure of SERVICE1_stop_0 (rc=198) on NODE1 succeeded
Aug 27 08:47:27 [1330] NODE2    pengine:  warning: unpack_rsc_op:     Pretending the failure of SERVICE1_stop_0 (rc=198) on NODE1 succeeded
Aug 27 08:47:27 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE9 active on NODE1
Aug 27 08:47:27 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE4 active on NODE2
Aug 27 08:47:27 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE1 active on NODE2
Aug 27 08:47:27 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 1 is already processed
Aug 27 08:47:27 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 2 is already processed
Aug 27 08:47:27 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 1 is already processed
Aug 27 08:47:27 [1330] NODE2    pengine:     info: unpack_node_loop:  Node 2 is already processed
Aug 27 08:47:27 [1330] NODE2    pengine:     info: group_print:        Resource Group: IPV
Aug 27 08:47:27 [1330] NODE2    pengine:     info: common_print:           VIRTUALIP  (ocf::heartbeat:IPaddr2):       Started NODE2
Aug 27 08:47:27 [1330] NODE2    pengine:     info: common_print:           SOURCEIP   (ocf::heartbeat:IPsrcaddr):      Started NODE2
Aug 27 08:47:27 [1330] NODE2    pengine:     info: common_print:      SERVICE1        (systemd:service1):     Started NODE2 (failure ignored)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: common_print:      SERVICE2    (systemd:service2): Started NODE2
Aug 27 08:47:27 [1330] NODE2    pengine:     info: common_print:      SERVICE3       (systemd:service3):  Started NODE2
Aug 27 08:47:27 [1330] NODE2    pengine:     info: common_print:      SERVICE4       (systemd:service4):  Started NODE2
Aug 27 08:47:27 [1330] NODE2    pengine:     info: common_print:      SERVICE5      (systemd:service5): Started NODE2
Aug 27 08:47:27 [1330] NODE2    pengine:     info: common_print:      SERVICE6 (systemd:service6):    Started NODE2
Aug 27 08:47:27 [1330] NODE2    pengine:     info: common_print:      SERVICE7  (systemd:service7):      Started NODE2
Aug 27 08:47:27 [1330] NODE2    pengine:     info: common_print:      SERVICE8   (systemd:service8):  Started NODE2
Aug 27 08:47:27 [1330] NODE2    pengine:     info: common_print:      SERVICE9        (systemd:service9):   Started NODE2
Aug 27 08:47:27 [1330] NODE2    pengine:     info: common_print:      SERVICE10        (systemd:service10):        Started NODE2
Aug 27 08:47:27 [1330] NODE2    pengine:     info: common_print:      SERVICE11       (systemd:service11):       Started NODE2
Aug 27 08:47:27 [1330] NODE2    pengine:     info: common_print:      SERVICE12   (systemd:service12):       Started NODE2
Aug 27 08:47:27 [1330] NODE2    pengine:     info: common_print:      SERVICE13  (systemd:service13):       Started NODE2
Aug 27 08:47:27 [1330] NODE2    pengine:     info: clone_print:        Clone Set: SERVICE14-clone [SERVICE14]
Aug 27 08:47:27 [1330] NODE2    pengine:     info: short_print:            Started: [ NODE1 NODE2 ]
Aug 27 08:47:27 [1330] NODE2    pengine:     info: clone_print:        Clone Set: SERVICE15-clone [SERVICE15]
Aug 27 08:47:27 [1330] NODE2    pengine:     info: short_print:            Started: [ NODE1 NODE2 ]
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   VIRTUALIP       (Started NODE2)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SOURCEIP        (Started NODE2)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE1        (Started NODE2)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE2    (Started NODE2)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE3       (Started NODE2)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE4       (Started NODE2)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE5      (Started NODE2)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE6 (Started NODE2)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE7  (Started NODE2)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE8   (Started NODE2)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE9        (Started NODE2)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE10        (Started NODE2)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE11       (Started NODE2)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE12   (Started NODE2)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE13  (Started NODE2)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE14:0        (Started NODE1)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE14:1        (Started NODE2)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE15:0 (Started NODE1)
Aug 27 08:47:27 [1330] NODE2    pengine:     info: LogActions:        Leave   SERVICE15:1 (Started NODE2)

From hunter86_bg at yahoo.com  Thu Aug 27 10:47:11 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 27 Aug 2020 14:47:11 +0000 (UTC)
Subject: [ClusterLabs] Resources restart when a node joins in
In-Reply-To: <DB7PR02MB53716B42197E31D8D15CA0A288550@DB7PR02MB5371.eurprd02.prod.outlook.com>
References: <DB7PR02MB53710C146E5E8F238EA8F84A88540@DB7PR02MB5371.eurprd02.prod.outlook.com>
 <CAPiuu9-RW5AzyKhzNBOcMZhVuAdzCY1nvK9mJmpvzEFCbpW47w@mail.gmail.com>
 <DB7PR02MB53716B42197E31D8D15CA0A288550@DB7PR02MB5371.eurprd02.prod.outlook.com>
Message-ID: <1794918337.6238676.1598539631434@mail.yahoo.com>

Hi?Quentin,

in order to get help it will be easier if you provide both corosync and pacemaker configuration.


Best Regards,
Strahil Nikolov






? ?????????, 27 ?????? 2020 ?., 17:10:01 ???????+3, Citron Vert <citron_vert at hotmail.com> ??????: 






Hi,

Sorry for using this email adress, my name is Quentin. Thank you for your reply.

I have already tried the stickiness solution (with the deprecated? value). I tried the one you gave me, and it does not change anything. 


Resources don't seem to move from node to node (i don't see the changes with crm_mon command).




In the logs i found this line "error: native_create_actions:???? Resource SERVICE1 is active on 2 nodes"

Which led me to contact you to understand and learn a little more about this cluster. And why there are running resources on the passive node.





You will find attached the logs during the reboot of the passive node and my cluster configuration.


I think I'm missing out on something in the configuration / logs that I don't understand..




Thank you in advance for your help,

Quentin





Le 26/08/2020 ? 20:16, Reid Wahl a ?crit?:


>??
>??
> Hi, Citron.
> 
> 
> 
> 
> Based on your description, it sounds like some resources **might** be moving from node 1 to node 2, failing on node 2, and then moving back to node 1. If that's what's happening (and even if it's not), then it's probably smart to set some resource stickiness as a resource default. The below command sets a resource stickiness score of 1.
> 
> 
> 
> 
> 
> ??? # pcs resource defaults resource-stickiness=1
> 
> 
> 
> 
> 
> Also note that the "default-resource-stickiness" cluster property is deprecated and should not be used.
> 
> 
> 
> 
> Finally, an explicit default resource stickiness score of 0 can interfere with the placement of cloned resource instances. If you don't want any stickiness, then it's better to leave stickiness unset. That way, primitives will have a stickiness of 0, but clone instances will have a stickiness of 1.
> 
> 
> 
> 
> 
> If adding stickiness does not resolve the issue, can you share your cluster configuration and some logs that show the issue happening? Off the top of my head I'm not sure why resources would start and stop on node 2 without moving away from node1, unless they're clone instances that are starting and then failing a monitor operation on node 2.
> 
> 
> 
>??
> On Wed, Aug 26, 2020 at 8:42 AM Citron Vert <citron_vert at hotmail.com> wrote:
> 
> 
>>??
>>??
>> Hello,
>> I am contacting you because I have a problem with my cluster and I cannot find (nor understand) any information that can help me.
>> 
>> I have a 2 nodes cluster (pacemaker, corosync, pcs) installed on CentOS 7 with a set of configuration.
>> Everything seems to works fine, but here is what happens:
>> 
>> ????* Node1 and Node2 are running well with Node1 as primary
>> ????* I reboot Node2 wich is passive (no changes on Node1)
>> ????* Node2 comes back in the cluster as passive
>> ????* corosync logs shows resources getting started then stopped on Node2
>> ????* "crm_mon" command shows some ressources on Node1 getting restarted 
>> 
>> 
>> I don't understand how it should work.
>> If a node comes back, and becomes passive (since Node1 is running primary), there is no reason for the resources to be started then stopped on the new passive node ?
>> 
>> 
>> One of my resources becomes unstable because it gets started and then stoped too quickly on Node2, wich seems to make it restart on Node1 without a failover.
>> 
>> I tried several things and solution proposed by different sites and forums but without success.
>> 
>> 
>> 
>> 
>> Is there a way so that the node, which joins the cluster as passive, does not start its own resources ?
>> 
>> 
>> 
>> 
>> thanks in advance
>> 
>> 
>> 
>> 
>> Here are some information just in case :
>> 
>> $?rpm?-qa?|?grep?-E?"corosync|pacemaker|pcs"
>>??corosync-2.4.5-4.el7.x86_64
>>??pacemaker-cli-1.1.21-4.el7.x86_64
>>??pacemaker-1.1.21-4.el7.x86_64
>>??pcs-0.9.168-4.el7.centos.x86_64
>>??corosynclib-2.4.5-4.el7.x86_64
>>??pacemaker-libs-1.1.21-4.el7.x86_64
>>??pacemaker-cluster-libs-1.1.21-4.el7.x86_64
>> 
>> 
>> 
>> 
>> ????????<nvpair?id="cib-bootstrap-options-stonith-enabled"?name="stonith-enabled"?value="false"/>
>> ????????<nvpair?id="cib-bootstrap-options-no-quorum-policy"?name="no-quorum-policy"?value="ignore"/>
>> ????????<nvpair?id="cib-bootstrap-options-dc-deadtime"?name="dc-deadtime"?value="120s"/>
>> ????????<nvpair?id="cib-bootstrap-options-have-watchdog"?name="have-watchdog"?value="false"/>
>> ????????<nvpair?id="cib-bootstrap-options-dc-version"?name="dc-version"?value="1.1.21-4.el7-f14e36fd43"/>
>> ????????<nvpair?id="cib-bootstrap-options-cluster-infrastructure"?name="cluster-infrastructure"?value="corosync"/>
>> ????????<nvpair?id="cib-bootstrap-options-cluster-name"?name="cluster-name"?value="CLUSTER"/>
>> ????????<nvpair?id="cib-bootstrap-options-last-lrm-refresh"?name="last-lrm-refresh"?value="1598446314"/>
>> ????????<nvpair?id="cib-bootstrap-options-default-resource-stickiness"?name="default-resource-stickiness"?value="0"/>
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users
>> 
>> ClusterLabs home: https://www.clusterlabs.org/
>> 
> 
> 
> 
> -- 
> 
>??
>??
>??
>??
>??
>??
>??
>??
>??
>??
>??
>??
>??
>??
>??
> Regards,
> 
> 
> Reid Wahl, RHCA
> 
> 
> Software Maintenance Engineer, Red Hat
> 
> CEE - Platform Support Delivery - ClusterHA
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

_______________________________________________
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

From Ulrich.Windl at rz.uni-regensburg.de  Fri Aug 28 02:07:11 2020
From: Ulrich.Windl at rz.uni-regensburg.de (Ulrich Windl)
Date: Fri, 28 Aug 2020 08:07:11 +0200
Subject: [ClusterLabs] Antw: [EXT] Re: Resources restart when a node joins in
In-Reply-To: <DB7PR02MB53716B42197E31D8D15CA0A288550@DB7PR02MB5371.eurprd02.prod.outlook.com>
References: <DB7PR02MB53710C146E5E8F238EA8F84A88540@DB7PR02MB5371.eurprd02.prod.outlook.com>
 <CAPiuu9-RW5AzyKhzNBOcMZhVuAdzCY1nvK9mJmpvzEFCbpW47w@mail.gmail.com>
 <DB7PR02MB53716B42197E31D8D15CA0A288550@DB7PR02MB5371.eurprd02.prod.outlook.com>
Message-ID: <5F489F0F020000A10003AF3E@gwsmtp.uni-regensburg.de>

>>> Citron Vert <citron_vert at hotmail.com> schrieb am 27.08.2020 um 09:46 in
Nachricht
<DB7PR02MB53716B42197E31D8D15CA0A288550 at DB7PR02MB5371.eurprd02.prod.outlook.com>

> Hi,
> 
> Sorry for using this email adress, my name is Quentin. Thank you for 
> your reply.
> 
> I have already tried the stickiness solution (with the deprecated  
> value). I tried the one you gave me, and it does not change anything.

Hi!

I see all your resources are systemd-based. Personally I thin systemd and
cluster do not play well with each other, as systemd also has kind of recovery
mechanisms that may interfere with the cluster. Also did you make sure to
"diasble" all the systemd services that the cluster controls?

There's an inconsistency:
Aug 27 08:46:16 [1330] NODE2    pengine:     info: determine_op_status:      
Operation monitor found resource SERVICE9 active on NODE1

SERVICE9 isn't listed as running before.

This should be checked:
Aug 27 08:46:55 [1330] NODE2    pengine:     info: native_color:      Resource
SERVICE14:1 cannot run anywhere
Aug 27 08:46:55 [1330] NODE2    pengine:     info: native_color:      Resource
SERVICE15:1 cannot run anywhere

This sounds odd, too:
Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_replace:      
Replaced 0.434.19 with 0.434.19 from NODE2

Aug 27 08:46:59 [1326] NODE2        cib:     info: cib_process_replace:      
Replaced 0.434.19 with 0.434.19 from NODE2

You should check why this failed:
Aug 27 08:47:01 [1331] NODE2       crmd:  warning: status_from_rc:    Action
20 (SERVICE1_monitor_0) on NODE1 failed (target: 7 vs. rc: 0): Error

Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:      SERVICE1
       (systemd:service1):     Started
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:             
1 : NODE1
Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:             
2 : NODE2

Aug 27 08:47:02 [1330] NODE2    pengine:    error: native_create_actions:    
Resource SERVICE1 is active on 2 nodes (attempting recovery)

Then:
Aug 27 08:47:02 [1330] NODE2    pengine:   notice: LogAction:  * Move      
SERVICE1                 ( NODE1 -> NODE2 )

Aug 27 08:47:02 [1331] NODE2       crmd:  warning: status_from_rc:    Action
20 (SERVICE9_monitor_0) on NODE1 failed (target: 7 vs. rc: 0): Error

Aug 27 08:47:04 [1331] NODE2       crmd:     info: match_graph_event: Action
SERVICE1_stop_0 (36) confirmed on NODE2 (rc=0)
Aug 27 08:47:22 [1331] NODE2       crmd:  warning: status_from_rc:    Action
35 (SERVICE1_stop_0) on NODE1 failed (target: 0 vs. rc: 198): Error
Aug 27 08:47:22 [1331] NODE2       crmd:     info: match_graph_event: Action
SERVICE1_stop_0 (35) confirmed on NODE1 (rc=198, ignoring failure)

The above sounds very odd to me.

You should also include the corresponding systemd logs.

The other thing I noticed is that you have colocation, but no ordering. I
guess your VIP should be started before your dependendies, right?

Regards,
Ulrich

> 
> Resources don't seem to move from node to node (i don't see the changes 
> with crm_mon command).
> 
> 
> In the logs i found this line /"error: native_create_actions:     
> Resource SERVICE1 is active on 2 nodes/"
> 
> Which led me to contact you to understand and learn a little more about 
> this cluster. And why there are running resources on the passive node.
> 
> 
> You will find attached the logs during the reboot of the passive node 
> and my cluster configuration.
> 
> I think I'm missing out on something in the configuration / logs that I 
> don't understand..
> 
> 
> Thank you in advance for your help,
> 
> Quentin
> 
> 
> Le 26/08/2020 ? 20:16, Reid Wahl a ?crit :
>> Hi, Citron.
>>
>> Based on your description, it sounds like some resources **might** be 
>> moving from node 1 to node 2, failing on node 2, and then moving back 
>> to node 1. If that's what's happening (and even if it's not), then 
>> it's probably smart to set some resource stickiness as a resource 
>> default. The below command sets a resource stickiness score of 1.
>>
>>     # pcs resource defaults resource-stickiness=1
>>
>> Also note that the "default-resource-stickiness" cluster property is 
>> deprecated and should not be used.
>>
>> Finally, an explicit default resource stickiness score of 0 can 
>> interfere with the placement of cloned resource instances. If you 
>> don't want any stickiness, then it's better to leave stickiness unset. 
>> That way, primitives will have a stickiness of 0, but clone instances 
>> will have a stickiness of 1.
>>
>> If adding stickiness does not resolve the issue, can you share your 
>> cluster configuration and some logs that show the issue happening? Off 
>> the top of my head I'm not sure why resources would start and stop on 
>> node 2 without moving away from node1, unless they're clone instances 
>> that are starting and then failing a monitor operation on node 2.
>>
>> On Wed, Aug 26, 2020 at 8:42 AM Citron Vert <citron_vert at hotmail.com 
>> <mailto:citron_vert at hotmail.com>> wrote:
>>
>>     Hello,
>>     I am contacting you because I have a problem with my cluster and I
>>     cannot find (nor understand) any information that can help me.
>>
>>     I have a 2 nodes cluster (pacemaker, corosync, pcs) installed on
>>     CentOS 7 with a set of configuration.
>>     Everything seems to works fine, but here is what happens:
>>
>>       * Node1 and Node2 are running well with Node1 as primary
>>       * I reboot Node2 wich is passive (no changes on Node1)
>>       * Node2 comes back in the cluster as passive
>>       * corosync logs shows resources getting started then stopped on
>>         Node2
>>       * "crm_mon" command shows some ressources on Node1 getting
>>         restarted
>>
>>     I don't understand how it should work.
>>     If a node comes back, and becomes passive (since Node1 is running
>>     primary), there is no reason for the resources to be started then
>>     stopped on the new passive node ?
>>
>>     One of my resources becomes unstable because it gets started and
>>     then stoped too quickly on Node2, wich seems to make it restart on
>>     Node1 without a failover.
>>
>>     I tried several things and solution proposed by different sites
>>     and forums but without success.
>>
>>
>>     Is there a way so that the node, which joins the cluster as
>>     passive, does not start its own resources ?
>>
>>
>>     thanks in advance
>>
>>
>>     Here are some information just in case :
>>
>>     $ rpm -qa | grep -E "corosync|pacemaker|pcs"
>>     corosync-2.4.5-4.el7.x86_64
>>     pacemaker-cli-1.1.21-4.el7.x86_64
>>     pacemaker-1.1.21-4.el7.x86_64
>>     pcs-0.9.168-4.el7.centos.x86_64
>>     corosynclib-2.4.5-4.el7.x86_64
>>     pacemaker-libs-1.1.21-4.el7.x86_64
>>     pacemaker-cluster-libs-1.1.21-4.el7.x86_64
>>
>>
>>             <nvpair id="cib-bootstrap-options-stonith-enabled" 
> name="stonith-enabled" value="false"/>
>>             <nvpair id="cib-bootstrap-options-no-quorum-policy" 
> name="no-quorum-policy" value="ignore"/>
>>             <nvpair id="cib-bootstrap-options-dc-deadtime"
name="dc-deadtime" 
> value="120s"/>
>>             <nvpair id="cib-bootstrap-options-have-watchdog" 
> name="have-watchdog" value="false"/>
>>             <nvpair id="cib-bootstrap-options-dc-version" name="dc-version"

> value="1.1.21-4.el7-f14e36fd43"/>
>>             <nvpair id="cib-bootstrap-options-cluster-infrastructure" 
> name="cluster-infrastructure" value="corosync"/>
>>             <nvpair id="cib-bootstrap-options-cluster-name" 
> name="cluster-name" value="CLUSTER"/>
>>             <nvpair id="cib-bootstrap-options-last-lrm-refresh" 
> name="last-lrm-refresh" value="1598446314"/>
>>             <nvpair id="cib-bootstrap-options-default-resource-stickiness"

> name="default-resource-stickiness" value="0"/>
>>
>>
>>
>>
>>     _______________________________________________
>>     Manage your subscription:
>>     https://lists.clusterlabs.org/mailman/listinfo/users 
>>
>>     ClusterLabs home: https://www.clusterlabs.org/ 
>>
>>
>>
>> -- 
>> Regards,
>>
>> Reid Wahl, RHCA
>> Software Maintenance Engineer, Red Hat
>> CEE - Platform Support Delivery - ClusterHA




From nwahl at redhat.com  Fri Aug 28 04:27:12 2020
From: nwahl at redhat.com (Reid Wahl)
Date: Fri, 28 Aug 2020 01:27:12 -0700
Subject: [ClusterLabs] Resources restart when a node joins in
In-Reply-To: <DB7PR02MB53713A95BBA282BF56FBF2BA88520@DB7PR02MB5371.eurprd02.prod.outlook.com>
References: <DB7PR02MB53710C146E5E8F238EA8F84A88540@DB7PR02MB5371.eurprd02.prod.outlook.com>
 <CAPiuu9-RW5AzyKhzNBOcMZhVuAdzCY1nvK9mJmpvzEFCbpW47w@mail.gmail.com>
 <DB7PR02MB53716B42197E31D8D15CA0A288550@DB7PR02MB5371.eurprd02.prod.outlook.com>
 <CAPiuu9-smjtSws3NnqT7k6teQ0mZWFxDadbQP5OYcQ9fvwz04g@mail.gmail.com>
 <DB7PR02MB53713A95BBA282BF56FBF2BA88520@DB7PR02MB5371.eurprd02.prod.outlook.com>
Message-ID: <CAPiuu9-J1W9VeAbqGDozdru=2-w2Z8fHSxMqra6wC++9KQT2-A@mail.gmail.com>

No problem! That's what we're here for. I'm glad it's sorted out :)

On Fri, Aug 28, 2020 at 12:27 AM Citron Vert <citron_vert at hotmail.com>
wrote:

> Hi,
>
> You are right, the problems seem to come from some services that are
> started at startup.
>
> My installation script disables all startup options for all services we
> use, that's why I didn't focus on this possibility.
>
> But after a quick investigation, a colleague had the good idea to make a
> "security" script that monitors and starts certain services.
>
>
> Sorry to have contacted you for this little mistake,
>
> Thank you for the help, it was effective
>
> Quentin
>
>
>
> Le 27/08/2020 ? 09:56, Reid Wahl a ?crit :
>
> Hi, Quentin. Thanks for the logs!
>
> I see you highlighted the fact that SERVICE1 was in "Stopping" state on
> both node 1 and node 2 when node 1 was rejoining the cluster. I also noted
> the following later in the logs, as well as some similar messages earlier:
>
> Aug 27 08:47:02 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE1 active on NODE1
> Aug 27 08:47:02 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE1 active on NODE1
> Aug 27 08:47:02 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE4 active on NODE2
> Aug 27 08:47:02 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE1 active on NODE2
> ...
> Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:              1 : NODE1
> Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:              2 : NODE2
> ...
> Aug 27 08:47:02 [1330] NODE2    pengine:    error: native_create_actions:     Resource SERVICE1 is active on 2 nodes (attempting recovery)
> Aug 27 08:47:02 [1330] NODE2    pengine:   notice: native_create_actions:     See https://wiki.clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information
>
>
> Can you make sure that all the cluster-managed systemd services are disabled from starting at boot (i.e., `systemctl is-enabled service1`, and the same for all the others) on both nodes? If they are enabled, disable them.
>
>
> On Thu, Aug 27, 2020 at 12:46 AM Citron Vert <citron_vert at hotmail.com>
> wrote:
>
>> Hi,
>>
>> Sorry for using this email adress, my name is Quentin. Thank you for your
>> reply.
>>
>> I have already tried the stickiness solution (with the deprecated
>> value). I tried the one you gave me, and it does not change anything.
>>
>> Resources don't seem to move from node to node (i don't see the changes
>> with crm_mon command).
>>
>>
>> In the logs i found this line *"error: native_create_actions:
>> Resource SERVICE1 is active on 2 nodes*"
>>
>> Which led me to contact you to understand and learn a little more about
>> this cluster. And why there are running resources on the passive node.
>>
>>
>> You will find attached the logs during the reboot of the passive node and
>> my cluster configuration.
>>
>> I think I'm missing out on something in the configuration / logs that I
>> don't understand..
>>
>>
>> Thank you in advance for your help,
>>
>> Quentin
>>
>>
>> Le 26/08/2020 ? 20:16, Reid Wahl a ?crit :
>>
>> Hi, Citron.
>>
>> Based on your description, it sounds like some resources **might** be
>> moving from node 1 to node 2, failing on node 2, and then moving back to
>> node 1. If that's what's happening (and even if it's not), then it's
>> probably smart to set some resource stickiness as a resource default. The
>> below command sets a resource stickiness score of 1.
>>
>>     # pcs resource defaults resource-stickiness=1
>>
>> Also note that the "default-resource-stickiness" cluster property is
>> deprecated and should not be used.
>>
>> Finally, an explicit default resource stickiness score of 0 can interfere
>> with the placement of cloned resource instances. If you don't want any
>> stickiness, then it's better to leave stickiness unset. That way,
>> primitives will have a stickiness of 0, but clone instances will have a
>> stickiness of 1.
>>
>> If adding stickiness does not resolve the issue, can you share your
>> cluster configuration and some logs that show the issue happening? Off the
>> top of my head I'm not sure why resources would start and stop on node 2
>> without moving away from node1, unless they're clone instances that are
>> starting and then failing a monitor operation on node 2.
>>
>> On Wed, Aug 26, 2020 at 8:42 AM Citron Vert <citron_vert at hotmail.com>
>> wrote:
>>
>>> Hello,
>>> I am contacting you because I have a problem with my cluster and I
>>> cannot find (nor understand) any information that can help me.
>>>
>>> I have a 2 nodes cluster (pacemaker, corosync, pcs) installed on CentOS
>>> 7 with a set of configuration.
>>> Everything seems to works fine, but here is what happens:
>>>
>>>    - Node1 and Node2 are running well with Node1 as primary
>>>    - I reboot Node2 wich is passive (no changes on Node1)
>>>    - Node2 comes back in the cluster as passive
>>>    - corosync logs shows resources getting started then stopped on Node2
>>>    - "crm_mon" command shows some ressources on Node1 getting restarted
>>>
>>> I don't understand how it should work.
>>> If a node comes back, and becomes passive (since Node1 is running
>>> primary), there is no reason for the resources to be started then stopped
>>> on the new passive node ?
>>>
>>> One of my resources becomes unstable because it gets started and then
>>> stoped too quickly on Node2, wich seems to make it restart on Node1 without
>>> a failover.
>>>
>>> I tried several things and solution proposed by different sites and
>>> forums but without success.
>>>
>>>
>>> Is there a way so that the node, which joins the cluster as passive,
>>> does not start its own resources ?
>>>
>>>
>>> thanks in advance
>>>
>>>
>>> Here are some information just in case :
>>> $ rpm -qa | grep -E "corosync|pacemaker|pcs"
>>> corosync-2.4.5-4.el7.x86_64
>>> pacemaker-cli-1.1.21-4.el7.x86_64
>>> pacemaker-1.1.21-4.el7.x86_64
>>> pcs-0.9.168-4.el7.centos.x86_64
>>> corosynclib-2.4.5-4.el7.x86_64
>>> pacemaker-libs-1.1.21-4.el7.x86_64
>>> pacemaker-cluster-libs-1.1.21-4.el7.x86_64
>>>
>>>
>>>         <nvpair id="cib-bootstrap-options-stonith-enabled" name=
>>> "stonith-enabled" value="false"/>
>>>         <nvpair id="cib-bootstrap-options-no-quorum-policy" name=
>>> "no-quorum-policy" value="ignore"/>
>>>         <nvpair id="cib-bootstrap-options-dc-deadtime" name=
>>> "dc-deadtime" value="120s"/>
>>>         <nvpair id="cib-bootstrap-options-have-watchdog" name=
>>> "have-watchdog" value="false"/>
>>>         <nvpair id="cib-bootstrap-options-dc-version" name="dc-version"
>>>  value="1.1.21-4.el7-f14e36fd43"/>
>>>         <nvpair id="cib-bootstrap-options-cluster-infrastructure" name=
>>> "cluster-infrastructure" value="corosync"/>
>>>         <nvpair id="cib-bootstrap-options-cluster-name" name=
>>> "cluster-name" value="CLUSTER"/>
>>>         <nvpair id="cib-bootstrap-options-last-lrm-refresh" name=
>>> "last-lrm-refresh" value="1598446314"/>
>>>         <nvpair id="cib-bootstrap-options-default-resource-stickiness"
>>>  name="default-resource-stickiness" value="0"/>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Manage your subscription:
>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>> ClusterLabs home: https://www.clusterlabs.org/
>>>
>>
>>
>> --
>> Regards,
>>
>> Reid Wahl, RHCA
>> Software Maintenance Engineer, Red Hat
>> CEE - Platform Support Delivery - ClusterHA
>>
>>
>
> --
> Regards,
>
> Reid Wahl, RHCA
> Software Maintenance Engineer, Red Hat
> CEE - Platform Support Delivery - ClusterHA
>
>

-- 
Regards,

Reid Wahl, RHCA
Software Maintenance Engineer, Red Hat
CEE - Platform Support Delivery - ClusterHA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200828/414b3e52/attachment-0001.htm>

From citron_vert at hotmail.com  Fri Aug 28 03:26:41 2020
From: citron_vert at hotmail.com (Citron Vert)
Date: Fri, 28 Aug 2020 09:26:41 +0200
Subject: [ClusterLabs] Resources restart when a node joins in
In-Reply-To: <CAPiuu9-smjtSws3NnqT7k6teQ0mZWFxDadbQP5OYcQ9fvwz04g@mail.gmail.com>
References: <DB7PR02MB53710C146E5E8F238EA8F84A88540@DB7PR02MB5371.eurprd02.prod.outlook.com>
 <CAPiuu9-RW5AzyKhzNBOcMZhVuAdzCY1nvK9mJmpvzEFCbpW47w@mail.gmail.com>
 <DB7PR02MB53716B42197E31D8D15CA0A288550@DB7PR02MB5371.eurprd02.prod.outlook.com>
 <CAPiuu9-smjtSws3NnqT7k6teQ0mZWFxDadbQP5OYcQ9fvwz04g@mail.gmail.com>
Message-ID: <DB7PR02MB53713A95BBA282BF56FBF2BA88520@DB7PR02MB5371.eurprd02.prod.outlook.com>

Hi,

You are right, the problems seem to come from some services that are 
started at startup.

My installation script disables all startup options for all services we 
use, that's why I didn't focus on this possibility.

But after a quick investigation, a colleague had the good idea to make a 
"security" script that monitors and starts certain services.


Sorry to have contacted you for this little mistake,

Thank you for the help, it was effective

Quentin



Le 27/08/2020 ? 09:56, Reid Wahl a ?crit?:
> Hi, Quentin. Thanks for the logs!
>
> I see you highlighted the fact that SERVICE1 was in "Stopping" state 
> on both node 1 and node 2 when node 1 was rejoining the cluster. I 
> also noted the following later in the logs, as well as some similar 
> messages earlier:
>
> Aug 27 08:47:02 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE1 active on NODE1
> Aug 27 08:47:02 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE1 active on NODE1
> Aug 27 08:47:02 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE4 active on NODE2
> Aug 27 08:47:02 [1330] NODE2    pengine:     info: determine_op_status:       Operation monitor found resource SERVICE1 active on NODE2
> ...
> Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:              1 : NODE1
> Aug 27 08:47:02 [1330] NODE2    pengine:     info: common_print:              2 : NODE2
> ...
> Aug 27 08:47:02 [1330] NODE2    pengine:    error: native_create_actions:     Resource SERVICE1 is active on 2 nodes (attempting recovery)
> Aug 27 08:47:02 [1330] NODE2    pengine:   notice: native_create_actions:     Seehttps://wiki.clusterlabs.org/wiki/FAQ#Resource_is_Too_Active  for more information
>
> Can you make sure that all the cluster-managed systemd services are 
> disabled from starting at boot (i.e., `systemctl is-enabled service1`, 
> and the same for all the others) on both nodes? If they are enabled, 
> disable them.
>
> On Thu, Aug 27, 2020 at 12:46 AM Citron Vert <citron_vert at hotmail.com 
> <mailto:citron_vert at hotmail.com>> wrote:
>
>     Hi,
>
>     Sorry for using this email adress, my name is Quentin. Thank you
>     for your reply.
>
>     I have already tried the stickiness solution (with the deprecated?
>     value). I tried the one you gave me, and it does not change anything.
>
>     Resources don't seem to move from node to node (i don't see the
>     changes with crm_mon command).
>
>
>     In the logs i found this line /"error: native_create_actions:????
>     Resource SERVICE1 is active on 2 nodes/"
>
>     Which led me to contact you to understand and learn a little more
>     about this cluster. And why there are running resources on the
>     passive node.
>
>
>     You will find attached the logs during the reboot of the passive
>     node and my cluster configuration.
>
>     I think I'm missing out on something in the configuration / logs
>     that I don't understand..
>
>
>     Thank you in advance for your help,
>
>     Quentin
>
>
>     Le 26/08/2020 ? 20:16, Reid Wahl a ?crit?:
>>     Hi, Citron.
>>
>>     Based on your description, it sounds like some resources
>>     **might** be moving from node 1 to node 2, failing on node 2, and
>>     then moving back to node 1. If that's what's happening (and even
>>     if it's not), then it's probably smart to set some resource
>>     stickiness as a resource default. The below command sets a
>>     resource stickiness score of 1.
>>
>>     ??? # pcs resource defaults resource-stickiness=1
>>
>>     Also note that the "default-resource-stickiness" cluster property
>>     is deprecated and should not be used.
>>
>>     Finally, an explicit default resource stickiness score of 0 can
>>     interfere with the placement of cloned resource instances. If you
>>     don't want any stickiness, then it's better to leave stickiness
>>     unset. That way, primitives will have a stickiness of 0, but
>>     clone instances will have a stickiness of 1.
>>
>>     If adding stickiness does not resolve the issue, can you share
>>     your cluster configuration and some logs that show the issue
>>     happening? Off the top of my head I'm not sure why resources
>>     would start and stop on node 2 without moving away from node1,
>>     unless they're clone instances that are starting and then failing
>>     a monitor operation on node 2.
>>
>>     On Wed, Aug 26, 2020 at 8:42 AM Citron Vert
>>     <citron_vert at hotmail.com <mailto:citron_vert at hotmail.com>> wrote:
>>
>>         Hello,
>>         I am contacting you because I have a problem with my cluster
>>         and I cannot find (nor understand) any information that can
>>         help me.
>>
>>         I have a 2 nodes cluster (pacemaker, corosync, pcs) installed
>>         on CentOS 7 with a set of configuration.
>>         Everything seems to works fine, but here is what happens:
>>
>>           * Node1 and Node2 are running well with Node1 as primary
>>           * I reboot Node2 wich is passive (no changes on Node1)
>>           * Node2 comes back in the cluster as passive
>>           * corosync logs shows resources getting started then
>>             stopped on Node2
>>           * "crm_mon" command shows some ressources on Node1 getting
>>             restarted
>>
>>         I don't understand how it should work.
>>         If a node comes back, and becomes passive (since Node1 is
>>         running primary), there is no reason for the resources to be
>>         started then stopped on the new passive node ?
>>
>>         One of my resources becomes unstable because it gets started
>>         and then stoped too quickly on Node2, wich seems to make it
>>         restart on Node1 without a failover.
>>
>>         I tried several things and solution proposed by different
>>         sites and forums but without success.
>>
>>
>>         Is there a way so that the node, which joins the cluster as
>>         passive, does not start its own resources ?
>>
>>
>>         thanks in advance
>>
>>
>>         Here are some information just in case :
>>
>>         $?rpm?-qa?|?grep?-E "corosync|pacemaker|pcs"
>>         corosync-2.4.5-4.el7.x86_64
>>         pacemaker-cli-1.1.21-4.el7.x86_64
>>         pacemaker-1.1.21-4.el7.x86_64
>>         pcs-0.9.168-4.el7.centos.x86_64
>>         corosynclib-2.4.5-4.el7.x86_64
>>         pacemaker-libs-1.1.21-4.el7.x86_64
>>         pacemaker-cluster-libs-1.1.21-4.el7.x86_64
>>
>>
>>         ????????<nvpair?id="cib-bootstrap-options-stonith-enabled"?name="stonith-enabled"?value="false"/>
>>         ????????<nvpair?id="cib-bootstrap-options-no-quorum-policy"?name="no-quorum-policy"?value="ignore"/>
>>         ????????<nvpair?id="cib-bootstrap-options-dc-deadtime"?name="dc-deadtime"?value="120s"/>
>>         ????????<nvpair?id="cib-bootstrap-options-have-watchdog"?name="have-watchdog"?value="false"/>
>>         ????????<nvpair?id="cib-bootstrap-options-dc-version"?name="dc-version"?value="1.1.21-4.el7-f14e36fd43"/>
>>         ????????<nvpair?id="cib-bootstrap-options-cluster-infrastructure"?name="cluster-infrastructure"?value="corosync"/>
>>         ????????<nvpair?id="cib-bootstrap-options-cluster-name"?name="cluster-name"?value="CLUSTER"/>
>>         ????????<nvpair?id="cib-bootstrap-options-last-lrm-refresh"?name="last-lrm-refresh"?value="1598446314"/>
>>         ????????<nvpair?id="cib-bootstrap-options-default-resource-stickiness"?name="default-resource-stickiness"?value="0"/>
>>
>>
>>
>>
>>         _______________________________________________
>>         Manage your subscription:
>>         https://lists.clusterlabs.org/mailman/listinfo/users
>>
>>         ClusterLabs home: https://www.clusterlabs.org/
>>
>>
>>
>>     -- 
>>     Regards,
>>
>>     Reid Wahl, RHCA
>>     Software Maintenance Engineer, Red Hat
>>     CEE - Platform Support Delivery - ClusterHA
>
>
>
> -- 
> Regards,
>
> Reid Wahl, RHCA
> Software Maintenance Engineer, Red Hat
> CEE - Platform Support Delivery - ClusterHA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200828/5c203d51/attachment-0001.htm>