[ClusterLabs] 'pcs stonith update' takes, then reverts
kgaillot at redhat.com
kgaillot at redhat.com
Mon Jul 26 09:54:15 EDT 2021
On Fri, 2021-07-23 at 21:46 -0400, Digimer wrote:
> After a LOT of hassle, I finally got it updated, but OMG it was
> painful.
>
> I degraded the cluster (unsure if needed), set maintenance mode,
> deleted
> the stonith levels, deleted the stonith devices, recreated them with
> the
> updated values, recreated the stonith levels, and finally disabled
> maintenance mode.
>
> It should not have been this hard, right? Why is heck would it be
> that
> pacemaker kept "rolling back" to old configs? I'd delete the stonith
That is bizarre. It sounds like the CIB changes were taking effect
locally, then being rejected by the rest of the cluster, which would
send the "correct" CIB back to the originator.
The logs of interest would be pacemaker.log from both nodes at the time
you made the first configuration change that failed. I'm guessing the
logs you posted were from after that point?
> levels, delete one PDU stonith resource, delete a second, and
> suddenly
> all the levels were back and the resources I had just removed came
> back
> with the old configs.
>
> Certainly I was doing something wrong, but what?
>
> digimer
>
> On 2021-07-23 8:04 p.m., Digimer wrote:
> > Update;
> >
> > Appears I can't even delete the damn things. They re-appeared
> > after
> > doing a [pcs stonith remove <id>'!.
> >
> > Wow.
> >
> > digimer
> >
> > On 2021-07-23 7:56 p.m., Digimer wrote:
> > > Hi all,
> > >
> > > Got a really odd one here...
> > >
> > > I had a cluster in the lab where it was built and tested. Then
> > > we
> > > deployed it, and I've been trying to update the stonith config.
> > > It
> > > _seems_ to take at first, then it reverts back to the old config.
> > > This
> > > is super annoying, obviously. :)
> > >
> > > So I can confirm that the APC PDUs work;
> > >
> > > ====
> > > [root at an-a02n01 ~]# fence_apc_snmp -a 10.201.2.3 -n 3 -o status
> > > Status: ON
> > > [root at an-a02n01 ~]# fence_apc_snmp -a 10.201.2.4 -n 3 -o status
> > > Status: ON
> > > ====
> > >
> > > Here's the config as-shipped;
> > >
> > > ====
> > > # pcs stonith config apc_snmp_node1_an-pdu03
> > > Resource: apc_snmp_node1_an-pdu03 (class=stonith
> > > type=fence_apc_snmp)
> > > Attributes: ip=10.201.2.1 pcmk_host_list=an-a02n01
> > > pcmk_off_action=reboot port=5
> > > Operations: monitor interval=60
> > > (apc_snmp_node1_an-pdu03-monitor-interval-60)
> > > Target: an-a02n01
> > > Level 1 - ipmilan_node1
> > > Level 2 - apc_snmp_node1_an-pdu03,apc_snmp_node1_an-pdu04
> > > Level 3 - delay_node1
> > > Target: an-a02n02
> > > Level 1 - ipmilan_node2
> > > Level 2 - apc_snmp_node2_an-pdu03,apc_snmp_node2_an-pdu04
> > > Level 3 - delay_node2
> > > ====
> > >
> > > So in this example, I am trying to update 'apc_snmp_node1_an-
> > > pdu03' to
> > > change the IP from 10.201.2.1 -> 10.201.2.3 and to change the
> > > port from
> > > port 5 -> 3.
> > >
> > > ====
> > > # pcs stonith update apc_snmp_node1_an-pdu03 ip=10.201.2.3
> > > pcmk_host_list=an-a02n01 pcmk_off_action=reboot port=3
> > > # pcs stonith config apc_snmp_node1_an-pdu03
> > > Resource: apc_snmp_node1_an-pdu03 (class=stonith
> > > type=fence_apc_snmp)
> > > Attributes: ip=10.201.2.3 pcmk_host_list=an-a02n01
> > > pcmk_off_action=reboot port=3
> > > ====
> > >
> > > As you can see, initially it appears to work. However, after a
> > > minute,
> > > the config reverts;
> > >
> > > ====
> > > # pcs stonith config apc_snmp_node1_an-pdu03
> > > Resource: apc_snmp_node1_an-pdu03 (class=stonith
> > > type=fence_apc_snmp)
> > > Attributes: ip=10.201.2.1 pcmk_host_list=an-a02n01
> > > pcmk_off_action=reboot port=5
> > > ====
> > >
> > > This happens to all four stonith devices (two per node, two
> > > nodes). I've
> > > tried doing a 'pcs stonith disable <id>' for all four devices,
> > > and did a
> > > 'pcs stonith cleanup' to make sure errors were cleared before
> > > updating,
> > > still no luck.
> > >
> > > Any idea what I'm doing wrong?
> > >
> > > The logs from the node I run the update on, followed by the logs
> > > on the
> > > peer:
> > >
> > > digimer
> > >
> > > ====
> > > Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd[121631]:
> > > notice:
> > > Updating all attributes after cib_refresh_notify event
> > > Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: State transition S_IDLE -> S_POLICY_ENGINE
> > > Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: State transition S_ELECTION -> S_INTEGRATION
> > > Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > notice:
> > > Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices)
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n01
> > > because
> > > resource parameters have changed
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:37:55 2021
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02
> > > because
> > > resource parameters have changed
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:37:42 2021
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Actions: Start apc_snmp_node1_an-pdu03
> > > (
> > > an-a02n02 )
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Calculated transition 173, saving inputs in
> > > /var/lib/pacemaker/pengine/pe-input-193.bz2
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Initiating start operation apc_snmp_node1_an-
> > > pdu03_start_0 on
> > > an-a02n02
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd[121631]:
> > > notice:
> > > Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]:
> > > INFINITY
> > > -> (unset)
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd[121631]:
> > > notice:
> > > Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]:
> > > 1627083488 -> (unset)
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd[121631]:
> > > notice:
> > > Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]:
> > > INFINITY
> > > -> (unset)
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd[121631]:
> > > notice:
> > > Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]:
> > > 1627083475 -> (unset)
> > > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Transition 173 aborted by deletion of
> > > lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']:
> > > Resource
> > > operation removal
> > > Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Transition 173 (Complete=3, Pending=0, Fired=0,
> > > Skipped=1,
> > > Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-
> > > 193.bz2): Stopped
> > > Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021
> > > Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021
> > > Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Calculated transition 174, saving inputs in
> > > /var/lib/pacemaker/pengine/pe-input-194.bz2
> > > Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Initiating monitor operation
> > > apc_snmp_node1_an-pdu03_monitor_60000 on an-a02n02
> > > Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Transition 174 (Complete=1, Pending=0, Fired=0,
> > > Skipped=0,
> > > Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-
> > > 194.bz2): Complete
> > > Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: State transition S_TRANSITION_ENGINE -> S_IDLE
> > > Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: State transition S_IDLE -> S_POLICY_ENGINE
> > > Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021
> > > Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021
> > > Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Actions: Stop apc_snmp_node1_an-pdu03
> > > (
> > > an-a02n02 ) due to node availability
> > > Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Calculated transition 175, saving inputs in
> > > /var/lib/pacemaker/pengine/pe-input-195.bz2
> > > Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0
> > > on
> > > an-a02n02
> > > Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Transition 175 (Complete=1, Pending=0, Fired=0,
> > > Skipped=0,
> > > Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-
> > > 195.bz2): Complete
> > > Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: State transition S_TRANSITION_ENGINE -> S_IDLE
> > > Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd[121631]:
> > > notice:
> > > Updating all attributes after cib_refresh_notify event
> > > Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: State transition S_IDLE -> S_POLICY_ENGINE
> > > Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: State transition S_ELECTION -> S_INTEGRATION
> > > Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > notice:
> > > Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices)
> > > Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021
> > > Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021
> > > Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Actions: Start apc_snmp_node1_an-pdu03
> > > (
> > > an-a02n02 )
> > > Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Calculated transition 176, saving inputs in
> > > /var/lib/pacemaker/pengine/pe-input-196.bz2
> > > Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Initiating start operation apc_snmp_node1_an-
> > > pdu03_start_0 on
> > > an-a02n02
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Transition 176 aborted by operation
> > > apc_snmp_node1_an-pdu03_start_0 'modify' on an-a02n02: Event
> > > failed
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Transition 176 action 31 (apc_snmp_node1_an-pdu03_start_0
> > > on
> > > an-a02n02): expected 'ok' but got 'error'
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Transition 176 (Complete=1, Pending=0, Fired=0,
> > > Skipped=0,
> > > Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-
> > > 196.bz2): Complete
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-attrd[121631]:
> > > notice:
> > > Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]:
> > > (unset)
> > > -> INFINITY
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-attrd[121631]:
> > > notice:
> > > Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]:
> > > (unset)
> > > -> 1627083929
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Actions: Recover apc_snmp_node1_an-pdu03
> > > (
> > > an-a02n02 )
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Calculated transition 177, saving inputs in
> > > /var/lib/pacemaker/pengine/pe-input-197.bz2
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-
> > > a02n02 ->
> > > an-a02n01 )
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Calculated transition 178, saving inputs in
> > > /var/lib/pacemaker/pengine/pe-input-198.bz2
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0
> > > on
> > > an-a02n02
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Initiating start operation apc_snmp_node1_an-
> > > pdu03_start_0
> > > locally on an-a02n01
> > > Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Requesting local execution of start operation for
> > > apc_snmp_node1_an-pdu03 on an-a02n01
> > > Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > [142B
> > > blob data]
> > > Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > notice:
> > > fence_apc_snmp_monitor_1[30692] error output [ ]
> > > Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > notice:
> > > fence_apc_snmp_monitor_1[30692] error output [ ]
> > > Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > notice:
> > > fence_apc_snmp_monitor_1[30692] error output [ 2021-07-23
> > > 16:45:35,820
> > > ERROR: Please use '-h' for usage ]
> > > Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > notice:
> > > fence_apc_snmp_monitor_1[30692] error output [ ]
> > > Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > [128B
> > > blob data]
> > > Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > warning: fence_apc_snmp[30692] stderr: [ ]
> > > Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > warning: fence_apc_snmp[30692] stderr: [ ]
> > > Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > warning: fence_apc_snmp[30692] stderr: [ 2021-07-23 16:45:35,820
> > > ERROR:
> > > Please use '-h' for usage ]
> > > Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > warning: fence_apc_snmp[30692] stderr: [ ]
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > [142B
> > > blob data]
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > notice:
> > > fence_apc_snmp_monitor_2[31341] error output [ ]
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > notice:
> > > fence_apc_snmp_monitor_2[31341] error output [ ]
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > notice:
> > > fence_apc_snmp_monitor_2[31341] error output [ 2021-07-23
> > > 16:45:42,979
> > > ERROR: Please use '-h' for usage ]
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > notice:
> > > fence_apc_snmp_monitor_2[31341] error output [ ]
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > [128B
> > > blob data]
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > warning: fence_apc_snmp[31341] stderr: [ ]
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > warning: fence_apc_snmp[31341] stderr: [ ]
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > warning: fence_apc_snmp[31341] stderr: [ 2021-07-23 16:45:42,979
> > > ERROR:
> > > Please use '-h' for usage ]
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > warning: fence_apc_snmp[31341] stderr: [ ]
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]:
> > > notice:
> > > Operation 'monitor' [31341] using apc_snmp_node1_an-pdu03
> > > returned -201
> > > (Generic Pacemaker error)
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Result of start operation for apc_snmp_node1_an-pdu03 on
> > > an-a02n01: error
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Transition 178 aborted by operation
> > > apc_snmp_node1_an-pdu03_start_0 'modify' on an-a02n01: Event
> > > failed
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Transition 178 action 32 (apc_snmp_node1_an-pdu03_start_0
> > > on
> > > an-a02n01): expected 'ok' but got 'error'
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Transition 178 (Complete=2, Pending=0, Fired=0,
> > > Skipped=0,
> > > Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-
> > > 198.bz2): Complete
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-attrd[121631]:
> > > notice:
> > > Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]:
> > > (unset)
> > > -> INFINITY
> > > Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-attrd[121631]:
> > > notice:
> > > Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]:
> > > (unset)
> > > -> 1627083942
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:45:29 2021
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:45:29 2021
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Actions: Recover apc_snmp_node1_an-pdu03
> > > (
> > > an-a02n01 )
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Calculated transition 179, saving inputs in
> > > /var/lib/pacemaker/pengine/pe-input-199.bz2
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:45:29 2021
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:45:29 2021
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Unexpected result (error) was recorded for start of
> > > apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02
> > > after
> > > 1000000 failures (max=1000000)
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Actions: Stop apc_snmp_node1_an-pdu03
> > > (
> > > an-a02n01 ) due to node availability
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > schedulerd[121632]:
> > > notice: Calculated transition 180, saving inputs in
> > > /var/lib/pacemaker/pengine/pe-input-200.bz2
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0
> > > locally
> > > on an-a02n01
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Requesting local execution of stop operation for
> > > apc_snmp_node1_an-pdu03 on an-a02n01
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Result of stop operation for apc_snmp_node1_an-pdu03 on
> > > an-a02n01: ok
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: Transition 180 (Complete=1, Pending=0, Fired=0,
> > > Skipped=0,
> > > Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-
> > > 200.bz2): Complete
> > > Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-
> > > controld[121633]:
> > > notice: State transition S_TRANSITION_ENGINE -> S_IDLE
> > > ====
> > >
> > > ====
> > > Jul 23 16:44:43 an-a02n02.alteeve.com pacemaker-
> > > controld[3296860]:
> > > notice: State transition S_NOT_DC -> S_PENDING
> > > Jul 23 16:44:43 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active
> > > devices)
> > > Jul 23 16:44:43 an-a02n02.alteeve.com pacemaker-
> > > controld[3296860]:
> > > notice: State transition S_PENDING -> S_NOT_DC
> > > Jul 23 16:44:47 an-a02n02.alteeve.com pacemaker-
> > > controld[3296860]:
> > > notice: Requesting local execution of start operation for
> > > apc_snmp_node1_an-pdu03 on an-a02n02
> > > Jul 23 16:44:47 an-a02n02.alteeve.com pacemaker-attrd[3296858]:
> > > notice:
> > > Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]:
> > > INFINITY
> > > -> (unset)
> > > Jul 23 16:44:47 an-a02n02.alteeve.com pacemaker-attrd[3296858]:
> > > notice:
> > > Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]:
> > > 1627083488 -> (unset)
> > > Jul 23 16:44:47 an-a02n02.alteeve.com pacemaker-attrd[3296858]:
> > > notice:
> > > Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]:
> > > INFINITY
> > > -> (unset)
> > > Jul 23 16:44:47 an-a02n02.alteeve.com pacemaker-attrd[3296858]:
> > > notice:
> > > Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]:
> > > 1627083475 -> (unset)
> > > Jul 23 16:44:48 an-a02n02.alteeve.com pacemaker-
> > > controld[3296860]:
> > > notice: Result of start operation for apc_snmp_node1_an-pdu03 on
> > > an-a02n02: ok
> > > Jul 23 16:44:48 an-a02n02.alteeve.com pacemaker-
> > > controld[3296860]:
> > > notice: Requesting local execution of monitor operation for
> > > apc_snmp_node1_an-pdu03 on an-a02n02
> > > Jul 23 16:44:48 an-a02n02.alteeve.com pacemaker-
> > > controld[3296860]:
> > > notice: Result of monitor operation for apc_snmp_node1_an-pdu03
> > > on
> > > an-a02n02: ok
> > > Jul 23 16:45:09 an-a02n02.alteeve.com pacemaker-
> > > controld[3296860]:
> > > notice: Requesting local execution of stop operation for
> > > apc_snmp_node1_an-pdu03 on an-a02n02
> > > Jul 23 16:45:09 an-a02n02.alteeve.com pacemaker-
> > > controld[3296860]:
> > > notice: Result of stop operation for apc_snmp_node1_an-pdu03 on
> > > an-a02n02: ok
> > > Jul 23 16:45:11 an-a02n02.alteeve.com pacemaker-
> > > controld[3296860]:
> > > notice: State transition S_NOT_DC -> S_PENDING
> > > Jul 23 16:45:11 an-a02n02.alteeve.com pacemaker-
> > > controld[3296860]:
> > > notice: State transition S_PENDING -> S_NOT_DC
> > > Jul 23 16:45:11 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active
> > > devices)
> > > Jul 23 16:45:16 an-a02n02.alteeve.com pacemaker-
> > > controld[3296860]:
> > > notice: Requesting local execution of start operation for
> > > apc_snmp_node1_an-pdu03 on an-a02n02
> > > Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > [144B
> > > blob data]
> > > Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > notice: fence_apc_snmp_monitor_1[3377776] error output [ ]
> > > Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > notice: fence_apc_snmp_monitor_1[3377776] error output [ ]
> > > Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > notice: fence_apc_snmp_monitor_1[3377776] error output [ 2021-07-
> > > 23
> > > 16:45:22,463 ERROR: Please use '-h' for usage ]
> > > Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > notice: fence_apc_snmp_monitor_1[3377776] error output [ ]
> > > Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > [130B
> > > blob data]
> > > Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > warning: fence_apc_snmp[3377776] stderr: [ ]
> > > Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > warning: fence_apc_snmp[3377776] stderr: [ ]
> > > Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > warning: fence_apc_snmp[3377776] stderr: [ 2021-07-23
> > > 16:45:22,463
> > > ERROR: Please use '-h' for usage ]
> > > Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > warning: fence_apc_snmp[3377776] stderr: [ ]
> > > Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > [144B
> > > blob data]
> > > Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > notice: fence_apc_snmp_monitor_2[3378394] error output [ ]
> > > Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > notice: fence_apc_snmp_monitor_2[3378394] error output [ ]
> > > Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > notice: fence_apc_snmp_monitor_2[3378394] error output [ 2021-07-
> > > 23
> > > 16:45:29,627 ERROR: Please use '-h' for usage ]
> > > Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > notice: fence_apc_snmp_monitor_2[3378394] error output [ ]
> > > Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > [130B
> > > blob data]
> > > Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > warning: fence_apc_snmp[3378394] stderr: [ ]
> > > Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > warning: fence_apc_snmp[3378394] stderr: [ ]
> > > Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > warning: fence_apc_snmp[3378394] stderr: [ 2021-07-23
> > > 16:45:29,627
> > > ERROR: Please use '-h' for usage ]
> > > Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > warning: fence_apc_snmp[3378394] stderr: [ ]
> > > Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]:
> > > notice: Operation 'monitor' [3378394] using apc_snmp_node1_an-
> > > pdu03
> > > returned -201 (Generic Pacemaker error)
> > > Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-
> > > controld[3296860]:
> > > notice: Result of start operation for apc_snmp_node1_an-pdu03 on
> > > an-a02n02: error
> > > Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-attrd[3296858]:
> > > notice:
> > > Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]:
> > > (unset)
> > > -> INFINITY
> > > Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-attrd[3296858]:
> > > notice:
> > > Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]:
> > > (unset)
> > > -> 1627083929
> > > Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-
> > > controld[3296860]:
> > > notice: Requesting local execution of stop operation for
> > > apc_snmp_node1_an-pdu03 on an-a02n02
> > > Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-
> > > controld[3296860]:
> > > notice: Result of stop operation for apc_snmp_node1_an-pdu03 on
> > > an-a02n02: ok
> > > Jul 23 16:45:42 an-a02n02.alteeve.com pacemaker-attrd[3296858]:
> > > notice:
> > > Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]:
> > > (unset)
> > > -> INFINITY
> > > Jul 23 16:45:42 an-a02n02.alteeve.com pacemaker-attrd[3296858]:
> > > notice:
> > > Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]:
> > > (unset)
> > > -> 1627083942
> > > ====
> > >
>
>
--
Ken Gaillot <kgaillot at redhat.com>
More information about the Users
mailing list