[Pacemaker] iscsi migration to slow (disk errors) what to do ...

[Florian Haas]

On 06/14/2011 01:38 PM, Jelle de Jong wrote:
> migration test by doing crm node standby on active target
> # crm configure show
> http://paste.debian.net/119832/
> I already had to tune the ocf:heartbeat:iSCSILogicalUnit timeout to 80s.

That's entirely possible, if your KVM hosts are producing a big I/O
backlog (but see my comment about cache="none" below).

> # repeating error message during migration until migration completes
> ERROR: Called "tgtadm --lld iscsi --op delete --mode logicalunit --tid 1
> --lun=1"
> ERROR: Exit code 22
> ERROR: Command output: "tgtadm: this logical unit is still active"

If that message repeats a few time and then still succeeds, that's OK. I
need to downgrade that message to a warning, though.

> # disk erros during iscsi/drbd migration on kvm host system
> http://paste.debian.net/119830/

You need to either use portblock (check the guide I mentioned in my 2/24
message), or move the IP address to the end of the resource group.

> # lvm logical volume is damaged after this...
> 
> # the kvm guest system was running bonnie++ -d /tmp/bonnie/ -n 128
> # and the guest reported disk errors and bonnie crashed
> # dmesg:
> http://paste.debian.net/119831/
> 
> Other kvm guest running mysql got corrupted databases.
> 
> However no more read-only file-systems on all kvm guests and the file
> system damage was recoverable instead of complete destruction after
> running fsck in previous tests...

Sure all your KVM block devices are using cache="none"?

You may also want to take a look at this guide:

http://www.linbit.com/en/education/tech-guides/highly-available-virtualization-with-kvm-iscsi-pacemaker/

> Please advice :) A migraion of the iscsi/drbd target should be possible
> on a busy system without damage to the guests?

Yes, and we know it's possible. That's what we demonstrated here:

http://linuxconfau.blip.tv/file/4719948/

Florian


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 262 bytes
Desc: OpenPGP digital signature
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20110614/92575891/attachment-0003.sig>