[ClusterLabs] Need help for "libvirtd[16755]: resource busy: Lockspace resource '...' is not locked"

[Ulrich Windl]

Hi!

I reconfigured my cluster to let it control virtlockd (instead of just "enable" it in systemd). However I still have problems I don't understand:
When live-migrating a Xen PV I still get these messages:
Jan 25 12:38:06 h18 virtlockd[42724]: libvirt version: 6.0.0
Jan 25 12:38:06 h18 virtlockd[42724]: hostname: h18
Jan 25 12:38:06 h18 virtlockd[42724]: resource busy: Lockspace resource '1076b120cb84515c54de74dc163c1a3af2832d51feb0b020dd070d3ed5c96128' is not locked
Jan 25 12:38:06 h18 libvirtd[42689]: libvirt version: 6.0.0
Jan 25 12:38:06 h18 libvirtd[42689]: hostname: h18
Jan 25 12:38:06 h18 libvirtd[42689]: resource busy: Lockspace resource '1076b120cb84515c54de74dc163c1a3af2832d51feb0b020dd070d3ed5c96128' is not locked
Jan 25 12:38:06 h18 libvirtd[42689]: Unable to release lease on test-jeos1
Jan 25 12:38:06 h18 VirtualDomain(prm_xen_test-jeos1)[47902]: INFO: test-jeos1: live migration to h16 succeeded.

When I inspect my nodes with "lsof -M /var/lib/libvirt/lockd", I see that for each PVM running on a node there is a virtlockd having open a file like this:
h16:~ # lsof -M /var/lib/libvirt/lockd
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
virtlockd 26349 root   12uw  REG   9,10        0 122900 /var/lib/libvirt/lockd/files/4c6bebd1f4bc581255b422a65d317f31deef91f777e51ba0daf04419dda7ade5
virtlockd 26349 root   13uw  REG   9,10        0 122901 /var/lib/libvirt/lockd/files/f9d587c61002c7480f8b86116eb4f7dfa210e52af7e944762f58c2c2f89a6865
virtlockd 26349 root   14uw  REG   9,10        0 122902 /var/lib/libvirt/lockd/files/1076b120cb84515c54de74dc163c1a3af2832d51feb0b020dd070d3ed5c96128
h16:~ # virsh list
 Id   Name         State
----------------------------
 0    Domain-0     running
 6    test-jeos3   running
 7    test-jeos4   running
 8    test-jeos1   running

h18:/etc/libvirt # lsof -M /var/lib/libvirt/lockd
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
virtlockd 42724 root   12uw  REG   9,10        0 124945 /var/lib/libvirt/lockd/files/7cc124bc29ec975f78295c33a67159a35c941cbc2f513c637d7285db82d1f13f
ph18:/etc/libvirt # virsh list
 Id   Name         State
----------------------------
 0    Domain-0     running
 7    test-jeos2   running

h18:/etc/libvirt # ll /var/lib/libvirt/lockd/files/
total 0
-rw------- 1 root root 0 Jan 25 12:38 1076b120cb84515c54de74dc163c1a3af2832d51feb0b020dd070d3ed5c96128
-rw------- 1 root root 0 Jan 25 12:08 4c6bebd1f4bc581255b422a65d317f31deef91f777e51ba0daf04419dda7ade5
-rw------- 1 root root 0 Jan 25 12:06 7cc124bc29ec975f78295c33a67159a35c941cbc2f513c637d7285db82d1f13f
-rw------- 1 root root 0 Jan 25 12:08 f9d587c61002c7480f8b86116eb4f7dfa210e52af7e944762f58c2c2f89a6865

So all files that exist are open by at most one node when not migrating. And I see a corresponding lock in /proc/locks:
12: POSIX  ADVISORY  WRITE 26349 09:0a:122902 0 0
13: POSIX  ADVISORY  WRITE 26349 09:0a:122901 0 0
14: POSIX  ADVISORY  WRITE 26349 09:0a:122900 0 0

24: POSIX  ADVISORY  WRITE 42724 09:0a:124945 0 0

So I wonder: Do I have a configuration problem still, or do I have a software problem (SLES15 SP2)?

The cluster config is basically:
primitive prm_lockspace_ocfs2 Filesystem ...
primitive prm_lockspace_raid_md10 Raid1 ...
primitive prm_virtlockd systemd:virtlockd.socket ...
clone cln_lockspace_ocfs2 prm_lockspace_ocfs2 meta interleave=true
clone cln_lockspace_raid_md10 prm_lockspace_raid_md10 meta interleave=true
clone cln_virtlockd prm_virtlockd meta interleave=true
colocation col_lockspace_ocfs2__DLM inf: cln_lockspace_ocfs2 cln_DLM
colocation col_lockspace_ocfs2__raid_md10 inf: cln_lockspace_ocfs2 cln_lockspace_raid_md10
colocation col_virtlockd__lockspace_fs inf: cln_virtlockd cln_lockspace_ocfs2
colocation col_vm__virtlockd inf: ( prm_xen_test-jeos1 prm_xen_test-jeos2 prm_xen_test-jeos3 prm_xen_test-jeos4 ) cln_virtlockd
order ord_DLM__lockspace_ocfs2 Mandatory: cln_DLM cln_lockspace_ocfs2
order ord_lockspace_fs__virtlockd Mandatory: cln_lockspace_ocfs2 cln_virtlockd
order ord_raid_md10__lockspace_ocfs2 Mandatory: cln_lockspace_raid_md10 cln_lockspace_ocfs2
order ord_virtlockd__vm Mandatory: cln_virtlockd ( prm_xen_test-jeos1 prm_xen_test-jeos2 prm_xen_test-jeos3 prm_xen_test-jeos4 )

Should I put another or mor libvirt processes under cluster control?
libvirtd.service
libvirtd-tls.socket
libvirtd-ro.socket
libvirtd.socket
virtlogd.socket

Regards,
Ulrich