[ClusterLabs] Xen Migration/resource cleanup problem in SLES11 SP3

[Dejan Muhamedagic]

Hi,

On Thu, Oct 08, 2015 at 02:29:08PM +0200, Ulrich Windl wrote:
> Hi!
> 
> I'd like to report an "interesting problem" with SLES11 SP3+HAE (latest updates):
> 
> When doing "rcopenais stop" on node "h10" with three Xen-VMs running, the cluster tried to migrate those VMs to other nodes (OK).
> 
> However migration failed on the remote nodes, but the cluster thought migration was successfully. Later the cluster restarted the VMs (BAD).
> 
> Oct  8 13:19:17 h10 Xen(prm_xen_v07)[16537]: INFO: v07: xm migrate to h01 succeeded.
> Oct  8 13:20:38 h01 Xen(prm_xen_v07)[9027]: ERROR: v07: Not active locally, migration failed!

xm did report success in migrate_to, but the overall migration
should've been considered failed, because migrate_from failed. Do
you have a too low timeout? The failure msg is logged 81 second
later, provided the clocks are in sync.

> Oct  8 13:44:53 h01 pengine[18985]:  warning: unpack_rsc_op_failure: Processing failed op migrate_from for prm_xen_v07 on h01: unknown error (1)
> 
> Things are really bad after h10 was rebooted eventually: The cluster restarted the three VMs again, because it thought those VMs were still running on h10! (VERY BAD)
> During startup, the cluster did nor probe the three VMs.

If a node restarted, how could anything think that there was
anything there still running. Strange.

But anyway, the if the migrate_from fails, then the resource
should still be running at the origin host, right?

Thanks,

Dejan

> Oct  8 14:14:20 h01 pengine[18985]:  warning: unpack_rsc_op_failure: Processing failed op migrate_from for prm_xen_v07 on h01: unknown error (1)
> 
> Oct  8 14:14:20 h01 pengine[18985]:   notice: LogActions: Restart prm_xen_v07 (Started h10)
> 
> Oct  8 14:14:20 h01 crmd[18986]:   notice: te_rsc_command: Initiating action 89: stop prm_xen_v07_stop_0 on h01 (local)
> 
> ...
> 
> Regards,
> Ulrich
> 
> 
> 
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org