[ClusterLabs] Live migrate a VM in a cluster group

Jason Gauthier jagauthier at gmail.com
Mon May 28 07:44:27 EDT 2018 

On Mon, May 28, 2018 at 12:03 AM, Andrei Borzenkov <arvidjaar at gmail.com> wrote:
> 28.05.2018 05:50, Jason Gauthier пишет:
>> Greetings,
>>
>>  I've set up a cluster intended for VMs.  I created a VM, and have
>> been pretty pleased with migrating it back and forth between the two
>> nodes.  Now, I am also using the VNC console, which requires listening
>> on port 59xx.  Of course, once the machine is migrated the IP to
>> access the VNC console is different.
>> So, I thought I would be clever and create a cluster IP address.  I
>> did, and as a stand alone resource it migrates between nodes
>> perfectly.  I then put these two primitives into a group.
>>
>> When I try to migrate the group nothing happens.
>> There aren't any cluster errors, and the logs do not give me any kind
>> of indication of error either.
>>
>
> "migrate" is ambiguous here - it may mean moving resource or
> live-migrating VM. Show exact command(s) you use and logs related to
> theses commands.
>
Sure, okay.  The command I am using is this:
# crm resource migrate g_Calibre alpha

The logs from the current owner looks like this:

May 28 07:42:32 [4333] beta        cib:     info: cib_process_request:
 Forwarding cib_delete operation for section constraints to all
(origin=local/crm_resource/3)
May 28 07:42:32 [4333] beta        cib:     info: cib_perform_op:
 Diff: --- 1.189.27 2
May 28 07:42:32 [4333] beta        cib:     info: cib_perform_op:
 Diff: +++ 1.190.0 f3b87f88cebf5470aed9f061e5f564c4
May 28 07:42:32 [4333] beta        cib:     info: cib_perform_op:
 -- /cib/configuration/constraints/rsc_location[@id='cli-prefer-g_Calibre']
May 28 07:42:32 [4333] beta        cib:     info: cib_perform_op:
 +  /cib:  @epoch=190, @num_updates=0
May 28 07:42:32 [4333] beta        cib:     info: cib_process_request:
 Completed cib_delete operation for section constraints: OK (rc=0,
origin=beta/crm_resource/3, version=1.190.0)
May 28 07:42:32 [4334] beta stonith-ng:     info:
update_cib_stonith_devices_v2:        Updating device list from the
cib: delete rsc_location[@id='cli-prefer-g_Calibre']
May 28 07:42:32 [4334] beta stonith-ng:     info: cib_devices_update:
 Updating devices to version 1.190.0
May 28 07:42:32 [4333] beta        cib:     info: cib_process_request:
 Forwarding cib_modify operation for section constraints to all
(origin=local/crm_resource/4)
May 28 07:42:32 [4334] beta stonith-ng:   notice: unpack_config:
 Watchdog will be used via SBD if fencing is required
May 28 07:42:32 [4333] beta        cib:     info: cib_perform_op:
 Diff: --- 1.190.0 2
May 28 07:42:32 [4333] beta        cib:     info: cib_perform_op:
 Diff: +++ 1.191.0 (null)
May 28 07:42:32 [4333] beta        cib:     info: cib_perform_op:
 +  /cib:  @epoch=191
May 28 07:42:32 [4333] beta        cib:     info: cib_perform_op:
 ++ /cib/configuration/constraints:  <rsc_location
id="cli-prefer-g_Calibre" rsc="g_Calibre" role="Started" node="alpha"
score="INFINITY"/>
May 28 07:42:32 [4333] beta        cib:     info: cib_process_request:
 Completed cib_modify operation for section constraints: OK (rc=0,
origin=beta/crm_resource/4, version=1.191.0)
May 28 07:42:32 [4333] beta        cib:     info: cib_file_backup:
 Archived previous version as /var/lib/pacemaker/cib/cib-28.raw
May 28 07:42:32 [4333] beta        cib:     info:
cib_file_write_with_digest:   Wrote version 1.190.0 of the CIB to disk
(digest: 3526dbb181e0e21078f2ff0f421c7031)
May 28 07:42:32 [4333] beta        cib:     info:
cib_file_write_with_digest:   Reading cluster configuration file
/var/lib/pacemaker/cib/cib.WVOQBs (digest:
/var/lib/pacemaker/cib/cib.i4m2ZP)
May 28 07:42:32 [4333] beta        cib:     info: cib_file_backup:
 Archived previous version as /var/lib/pacemaker/cib/cib-29.raw
May 28 07:42:32 [4333] beta        cib:     info:
cib_file_write_with_digest:   Wrote version 1.191.0 of the CIB to disk
(digest: 4c39c8efbd77c9e3ee831a20890f611d)
May 28 07:42:32 [4333] beta        cib:     info:
cib_file_write_with_digest:   Reading cluster configuration file
/var/lib/pacemaker/cib/cib.rxLoCv (digest:
/var/lib/pacemaker/cib/cib.6mH61S)
May 28 07:42:37 [4333] beta        cib:     info: cib_process_ping:
 Reporting our current digest to alpha:
820a81c6a738a338a6dee5f02685d464 for 1.191.0 (0x56502e4a4130 0)


The logs from the other node look like this:
May 28 07:41:23 [5082] alpha        cib:     info: cib_process_ping:
 Reporting our current digest to alpha:
193a1c4a09c2515bb160b2430c40079f for 1.189.27 (0x55de0024d9e0 0)
May 28 07:42:32 [5082] alpha        cib:     info: cib_perform_op:
 Diff: --- 1.189.27 2
May 28 07:42:32 [5082] alpha        cib:     info: cib_perform_op:
 Diff: +++ 1.190.0 f3b87f88cebf5470aed9f061e5f564c4
May 28 07:42:32 [5082] alpha        cib:     info: cib_perform_op:
 -- /cib/configuration/constraints/rsc_location[@id='cli-prefer-g_Calibre']
May 28 07:42:32 [5082] alpha        cib:     info: cib_perform_op:
 +  /cib:  @epoch=190, @num_updates=0
May 28 07:42:32 [5082] alpha        cib:     info:
cib_process_request: Completed cib_delete operation for section
constraints: OK (rc=0, origin=beta/crm_resource/3, version=1.190.0)
May 28 07:42:32 [5087] alpha       crmd:     info:
abort_transition_graph:      Transition aborted by deletion of
rsc_location[@id='cli-prefer-g_Calibre']: Configuration change |
cib=1.190.0 source=te_update_diff:444
path=/cib/configuration/constraints/rsc_location[@id='cli-prefer-g_Calibre']
complete=true
May 28 07:42:32 [5087] alpha       crmd:   notice:
do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE |
input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
May 28 07:42:32 [5083] alpha stonith-ng:     info:
update_cib_stonith_devices_v2:       Updating device list from the
cib: delete rsc_location[@id='cli-prefer-g_Calibre']
May 28 07:42:32 [5083] alpha stonith-ng:     info: cib_devices_update:
 Updating devices to version 1.190.0
May 28 07:42:32 [5083] alpha stonith-ng:   notice: unpack_config:
 Watchdog will be used via SBD if fencing is required
May 28 07:42:32 [5082] alpha        cib:     info: cib_perform_op:
 Diff: --- 1.190.0 2
May 28 07:42:32 [5082] alpha        cib:     info: cib_perform_op:
 Diff: +++ 1.191.0 (null)
May 28 07:42:32 [5082] alpha        cib:     info: cib_perform_op:
 +  /cib:  @epoch=191
May 28 07:42:32 [5082] alpha        cib:     info: cib_perform_op:
 ++ /cib/configuration/constraints:  <rsc_location
id="cli-prefer-g_Calibre" rsc="g_Calibre" role="Started" node="alpha"
score="INFINITY"/>
May 28 07:42:32 [5082] alpha        cib:     info: cib_file_backup:
 Archived previous version as /var/lib/pacemaker/cib/cib-23.raw
May 28 07:42:32 [5082] alpha        cib:     info:
cib_file_write_with_digest:  Wrote version 1.190.0 of the CIB to disk
(digest: 3526dbb181e0e21078f2ff0f421c7031)
May 28 07:42:32 [5082] alpha        cib:     info:
cib_process_request: Completed cib_modify operation for section
constraints: OK (rc=0, origin=beta/crm_resource/4, version=1.191.0)
May 28 07:42:32 [5082] alpha        cib:     info:
cib_file_write_with_digest:  Reading cluster configuration file
/var/lib/pacemaker/cib/cib.p6o9HA (digest:
/var/lib/pacemaker/cib/cib.Scs085)
May 28 07:42:32 [5087] alpha       crmd:     info:
abort_transition_graph:      Transition aborted by
rsc_location.cli-prefer-g_Calibre 'create': Configuration change |
cib=1.191.0 source=te_update_diff:444
path=/cib/configuration/constraints complete=true
May 28 07:42:32 [5086] alpha    pengine:   notice: unpack_config:
 Watchdog will be used via SBD if fencing is required
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_online_status_fencing:     Node beta is active
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_online_status:     Node beta is online
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_online_status_fencing:     Node alpha is active
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_online_status:     Node alpha is online
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_dlm_controld:0
active on beta
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_dlm_controld:0
active on beta
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_fs_gfs2:0
active on beta
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_fs_gfs2:0
active on beta
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_CalibreVNC
active on beta
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_CalibreVNC
active on beta
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_Calibre active
on beta
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_Calibre active
on beta
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_dlm_controld:1
active on alpha
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_dlm_controld:1
active on alpha
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_fs_gfs2:1
active on alpha
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_fs_gfs2:1
active on alpha
May 28 07:42:32 [5086] alpha    pengine:     info: native_print:
 stonith_sbd     (stonith:external/sbd): Started alpha
May 28 07:42:32 [5086] alpha    pengine:     info: clone_print:  Clone
Set: cl_gfs2 [g_gfs2]
May 28 07:42:32 [5086] alpha    pengine:     info: short_print:
Started: [ alpha beta ]
May 28 07:42:32 [5086] alpha    pengine:     info: group_print:
Resource Group: g_Calibre
May 28 07:42:32 [5086] alpha    pengine:     info: native_print:
      p_CalibreVNC       (ocf::heartbeat:IPaddr2):       Started beta
May 28 07:42:32 [5086] alpha    pengine:     info: native_print:
      p_Calibre  (ocf::heartbeat:VirtualDomain): Started beta
May 28 07:42:32 [5086] alpha    pengine:     info: LogActions:  Leave
 stonith_sbd     (Started alpha)
May 28 07:42:32 [5086] alpha    pengine:     info: LogActions:  Leave
 p_dlm_controld:0        (Started beta)
May 28 07:42:32 [5086] alpha    pengine:     info: LogActions:  Leave
 p_fs_gfs2:0     (Started beta)
May 28 07:42:32 [5086] alpha    pengine:     info: LogActions:  Leave
 p_dlm_controld:1        (Started alpha)
May 28 07:42:32 [5086] alpha    pengine:     info: LogActions:  Leave
 p_fs_gfs2:1     (Started alpha)
May 28 07:42:32 [5086] alpha    pengine:     info: LogActions:  Leave
 p_CalibreVNC    (Started beta)
May 28 07:42:32 [5086] alpha    pengine:     info: LogActions:  Leave
 p_Calibre       (Started beta)
May 28 07:42:32 [5086] alpha    pengine:   notice: process_pe_message:
 Calculated transition 136, saving inputs in
/var/lib/pacemaker/pengine/pe-input-326.bz2
May 28 07:42:32 [5087] alpha       crmd:     info: handle_response:
 pe_calc calculation pe_calc-dc-1527507752-483 is obsolete
May 28 07:42:32 [5082] alpha        cib:     info: cib_file_backup:
 Archived previous version as /var/lib/pacemaker/cib/cib-24.raw
May 28 07:42:32 [5082] alpha        cib:     info:
cib_file_write_with_digest:  Wrote version 1.191.0 of the CIB to disk
(digest: 4c39c8efbd77c9e3ee831a20890f611d)
May 28 07:42:32 [5082] alpha        cib:     info:
cib_file_write_with_digest:  Reading cluster configuration file
/var/lib/pacemaker/cib/cib.LfyZBD (digest:
/var/lib/pacemaker/cib/cib.G10d48)
May 28 07:42:32 [5086] alpha    pengine:   notice: unpack_config:
 Watchdog will be used via SBD if fencing is required
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_online_status_fencing:     Node beta is active
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_online_status:     Node beta is online
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_online_status_fencing:     Node alpha is active
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_online_status:     Node alpha is online
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_dlm_controld:0
active on beta
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_dlm_controld:0
active on beta
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_fs_gfs2:0
active on beta
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_fs_gfs2:0
active on beta
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_CalibreVNC
active on beta
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_CalibreVNC
active on beta
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_Calibre active
on beta
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_Calibre active
on beta
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_dlm_controld:1
active on alpha
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_dlm_controld:1
active on alpha
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_fs_gfs2:1
active on alpha
May 28 07:42:32 [5086] alpha    pengine:     info:
determine_op_status: Operation monitor found resource p_fs_gfs2:1
active on alpha
May 28 07:42:32 [5086] alpha    pengine:     info: native_print:
 stonith_sbd     (stonith:external/sbd): Started alpha
May 28 07:42:32 [5086] alpha    pengine:     info: clone_print:  Clone
Set: cl_gfs2 [g_gfs2]
May 28 07:42:32 [5086] alpha    pengine:     info: short_print:
Started: [ alpha beta ]
May 28 07:42:32 [5086] alpha    pengine:     info: group_print:
Resource Group: g_Calibre
May 28 07:42:32 [5086] alpha    pengine:     info: native_print:
      p_CalibreVNC       (ocf::heartbeat:IPaddr2):       Started beta
May 28 07:42:32 [5086] alpha    pengine:     info: native_print:
      p_Calibre  (ocf::heartbeat:VirtualDomain): Started beta
May 28 07:42:32 [5086] alpha    pengine:     info: LogActions:  Leave
 stonith_sbd     (Started alpha)
May 28 07:42:32 [5086] alpha    pengine:     info: LogActions:  Leave
 p_dlm_controld:0        (Started beta)
May 28 07:42:32 [5086] alpha    pengine:     info: LogActions:  Leave
 p_fs_gfs2:0     (Started beta)
May 28 07:42:32 [5086] alpha    pengine:     info: LogActions:  Leave
 p_dlm_controld:1        (Started alpha)
May 28 07:42:32 [5086] alpha    pengine:     info: LogActions:  Leave
 p_fs_gfs2:1     (Started alpha)
May 28 07:42:32 [5086] alpha    pengine:     info: LogActions:  Leave
 p_CalibreVNC    (Started beta)
May 28 07:42:32 [5086] alpha    pengine:     info: LogActions:  Leave
 p_Calibre       (Started beta)
May 28 07:42:32 [5086] alpha    pengine:   notice: process_pe_message:
 Calculated transition 137, saving inputs in
/var/lib/pacemaker/pengine/pe-input-327.bz2
May 28 07:42:32 [5087] alpha       crmd:     info:
do_state_transition: State transition S_POLICY_ENGINE ->
S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE
origin=handle_response
May 28 07:42:32 [5087] alpha       crmd:     info: do_te_invoke:
 Processing graph 137 (ref=pe_calc-dc-1527507752-484) derived from
/var/lib/pacemaker/pengine/pe-input-327.bz2
May 28 07:42:32 [5087] alpha       crmd:   notice: run_graph:
Transition 137 (Complete=0, Pending=0, Fired=0, Skipped=0,
Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-327.bz2):
Complete
May 28 07:42:32 [5087] alpha       crmd:     info: do_log:      Input
I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
May 28 07:42:32 [5087] alpha       crmd:   notice:
do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE |
input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
May 28 07:42:37 [5082] alpha        cib:     info: cib_process_ping:
 Reporting our current digest to alpha:
820a81c6a738a338a6dee5f02685d464 for 1.191.0 (0x55ddffde8700 0)



I do see something I did not see before that is probably relevant, but
still not clear as to what is happening.
May 28 07:42:32 [5087] alpha       crmd:     info:
abort_transition_graph:      Transition aborted by
rsc_location.cli-prefer-g_Calibre 'create': Configuration change |
cib=1.191.0 source=te_update_diff:444
path=/cib/configuration/constraints complete=true

Thanks,



>> I'm wondering if this is even something I should expect to work. I
>> would certainly like it to.
>> Here's the relevant config:
>>
>> primitive p_Calibre VirtualDomain \
>>         params config="/vms/Calibre/Calibre.xml"
>> hypervisor="qemu:///system" migration_transport=ssh \
>>         meta allow-migrate=true \
>>         op start timeout=120s interval=0 \
>>         op stop timeout=120s interval=0 \
>>         op monitor timeout=30 interval=10 depth=0 \
>>         utilization cpu=2 hv_memory=8192
>> primitive p_CalibreVNC IPaddr2 \
>>         params ip=192.168.10.10 cidr_netmask=24 nic=br10 \
>>         op monitor interval=10s
>>
>> group g_Calibre p_CalibreVNC p_Calibre \
>>         meta target-role=Started
>>
>> location cli-prefer-g_Calibre g_Calibre role=Started inf: alpha
>> location cli-prefer-p_Calibre p_Calibre role=Started inf: beta
>> location cli-prefer-p_CalibreVNC p_CalibreVNC role=Started inf: beta
>>
>> Any help is appreciated.
>>
>> Jason
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

More information about the Users mailing list