[ClusterLabs] Failed migration causing fencing loop

[Gao,Yan]

On 2022/3/31 9:03, Ulrich Windl wrote:
> Hi!
> 
> I just wanted to point out one thing that hit us with SLES15 SP3:
> Some failed live VM migration causing node fencing resulted in a fencing loop, because of two reasons:
> 
> 1) Pacemaker thinks that even _after_ fencing there is some migration to "clean up". Pacemaker treats the situation as if the VM is running on both nodes, thus (50% chance?) trying to stop the VM on the node that just booted after fencing. That's supid but shouldn't be fatal IF there weren't...
> 
> 2) The stop operation of the VM (that atually isn't running) fails,

AFAICT it could not connect to the hypervisor, but the logic in the RA 
is kind of arguable that the probe (monitor) of the VM returned "not 
running", but the stop right after that returned failure...

OTOH, the point about pacemaker is the stop of the resource on the 
fenced and rejoined node is not really necessary. There has been 
discussions about this here and we are trying to figure out a solution 
for it:

https://github.com/ClusterLabs/pacemaker/pull/2146#discussion_r828204919

For now it requires administrator's intervene if the situation happens:
1) Fix the access to hypervisor before the fenced node rejoins.
2) Manually cleanup the resource, which tells pacemaker it can safely 
forget the historical migrate_to failure.

Regards,
   Yan

> causing a node fence. So the loop is complete.
> 
> Some details (many unrelated messages left out):
> 
> Mar 30 16:06:14 h16 libvirtd[13637]: internal error: libxenlight failed to restore domain 'v15'
> 
> Mar 30 16:06:15 h19 pacemaker-schedulerd[7350]:  warning: Unexpected result (error: v15: live migration to h16 failed: 1) was recorded for migrate_to of prm_xen_v15 on h18 at Mar 30 16:06:13 2022
> 
> Mar 30 16:13:37 h19 pacemaker-schedulerd[7350]:  warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of prm_libvirtd:0 on h18 at Mar 30 16:13:36 2022
> Mar 30 16:13:37 h19 pacemaker-schedulerd[7350]:  warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of prm_libvirtd:0 on h18 at Mar 30 16:13:36 2022
> Mar 30 16:13:37 h19 pacemaker-schedulerd[7350]:  warning: Cluster node h18 will be fenced: prm_libvirtd:0 failed there
> 
> Mar 30 16:19:00 h19 pacemaker-schedulerd[7350]:  warning: Unexpected result (error: v15: live migration to h18 failed: 1) was recorded for migrate_to of prm_xen_v15 on h16 at Mar 29 23:58:40 2022
> Mar 30 16:19:00 h19 pacemaker-schedulerd[7350]:  error: Resource prm_xen_v15 is active on 2 nodes (attempting recovery)
> 
> Mar 30 16:19:00 h19 pacemaker-schedulerd[7350]:  notice:  * Restart    prm_xen_v15              (             h18 )
> 
> Mar 30 16:19:04 h18 VirtualDomain(prm_xen_v15)[8768]: INFO: Virtual domain v15 currently has no state, retrying.
> Mar 30 16:19:05 h18 VirtualDomain(prm_xen_v15)[8787]: INFO: Virtual domain v15 currently has no state, retrying.
> Mar 30 16:19:07 h18 VirtualDomain(prm_xen_v15)[8822]: ERROR: Virtual domain v15 has no state during stop operation, bailing out.
> Mar 30 16:19:07 h18 VirtualDomain(prm_xen_v15)[8836]: INFO: Issuing forced shutdown (destroy) request for domain v15.
> Mar 30 16:19:07 h18 VirtualDomain(prm_xen_v15)[8860]: ERROR: forced stop failed
> 
> Mar 30 16:19:07 h19 pacemaker-controld[7351]:  notice: Transition 124 action 115 (prm_xen_v15_stop_0 on h18): expected 'ok' but got 'error'
> 
> Note: Our cluster nodes start pacemaker during boot. Yesterday I was there when the problem happened. But as we had another boot loop some time ago I wrote a systemd service that counts boots, and if too many happen within a short time, pacemaker will be disabled on that node. As it it set now, the counter is reset if the node is up for at least 15 minutes; if it fails more than 4 times to do so, pacemaker will be disabled. If someone wants to try that or give feedback, drop me a line, so I could provide the RPM (boot-loop-handler-0.0.5-0.0.noarch)...
> 
> Regards,
> Ulrich
> 
> 
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> ClusterLabs home: https://www.clusterlabs.org/
>