[ClusterLabs] Mirrored cLVM/Xen PVM Performance question for block device

[Ulrich Windl]

Hi!

I have a performance question regarding delay for reading blocks in a PV Xen VM.
Forst a little background: Originally to monitor NFS outages, I developed a tool "iotwatch" (short: IOTW) that reads the first block of a block device or file (or anything you can open() and read() with Direct I/O). The tool samples the target at a rather high rate (like 5s), keeping statistics that are queried at a lower rate (like 5 min).

A wrapper around the tool is used as monitoring plugin, and the outoput looks like this:
/dev/sys/var: alpha=0.01, count=75(120/120), last=0.0011, avg=0.00423/0.00264/0\
.00427, min=0.00052(0.00052/0.00084), max=0.02465(0.02465/0.02062), variance=0.\
00005(0.00003)|last=0.0011;;;0 exp_avg=0.00427;;;0 emin=0.00084;;;0 emax=0.0206\
2;;;0 davg=0.00264;;;0 dstd_dev=0.00617;;;0

A short explanation what these numbers mean:
"alpha" is the weight used for exponential averaging (e.g. for "exp_avg"). "count" is the number of samples since last read and the number of samples in the sampling queue (e.g. 120 valid samples ot of a maximum of 120). The values "avg" is average, "min" is the minimum", "max" is the maximum, "variance" is what it says, and "last" is the last sampling value.
In text output there are three numbers instead of just one, meaning (the indicated value, the average of the value within the sampling queue, and the exponentially averaged value). This is mostly for debugging. The performance data output has just one of those values, selectable via command-line option. Also the statistics can be (in this case they are) reset after it was read, so min and max will start anew...

OK, that was a rather long story before presenting the details:

A VM has its root disk on a mirrored LV (cLVM) presented as "phy:", and inside the VM the disk is partitioned like this:
Device     Boot  Start      End  Sectors  Size Id Type
/dev/xvdb1 *      2048   411647   409600  200M 83 Linux
/dev/xvdb2      411648 83886079 83474432 39.8G  5 Extended
/dev/xvdb5      413696 83886079 83472384 39.8G 8e Linux LVM

xvdb5 is a PV for the sys VG, like this:
  opt  sys -wi-ao----   4.00g                                                   
  root sys -wi-ao----   8.00g                                                   
  srv  sys -wi-ao----   4.00g                                                   
  swap sys -wi-ao----   2.00g                                                   
  tmp  sys -wi-ao---- 512.00m                                                   
  var  sys -wi-ao----   6.00g                                                   

LV var is mounted on /var as ext3 (acl,user_xattr). The timing threads runs with prio -80 (nice 0) at SCHED_RR, so I guess other processes won't disturb the measurements much. I see no other threads using a real-time scheduling policy in the VM; system tasks seem to run at prio 0 with some negative nice value instead...
(On the xen host corosync, DLM and OCFS2 runs with prio -2)

Now the story: The performance of the root disk inside the VM (IOTW-PV) has a typical read delay of less than 2ms with peaks below 40ms (A comparable local disk in bare metal) would have less than 0.2ms delay with peaks below 7ms).  However when timing the var LV (IOTW_FS), the average is below 4ms with peaks up to 80ms.

The storage system behind is a FC-based 3PAR StorServ with all SSDs and the service time for reads is (according to the storage system'S own perfomance monitor (SSMC)) significantly below 0.25ms at the same time interval.

So I wonder: How can LVM in the VM add another 40ms peak to the base timing? The other thing that puzzles  me is this: While the timing for the root disk is basically good with very few peaks, the timing of the LV has mainly three levels: First, most common level is good performance. the next level is like 20ms (more), and the third level are peaks of another 20 or 40 ms.

Is there any explanation for this? The VM is SLES12 SP5, while the Xen Host is still SLES11 SP4.

At the moment I'm thinking how to implement VM disks in a way that is efficient while supporting live migration of VMs.
In the past we were using filesystem images stored in OCFS2 which itself was put in a mirrored cLVM LV. Performance was rather poor, so I skipped the OCFS2 layer and created a separate LV for each VM. Unfortunately mirroring all VM images to different storage systems is an absolute requirement.

I'd be glad to get some insights.

Regards,
Ulrich



-------------- next part --------------
A non-text attachment was scrubbed...
Name: IOTW_FS.png
Type: image/png
Size: 97372 bytes
Desc: not available
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200520/6fd03b0c/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IOTW-PV.png
Type: image/png
Size: 101961 bytes
Desc: not available
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200520/6fd03b0c/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SSMC-Perf.PNG
Type: image/png
Size: 61135 bytes
Desc: not available
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20200520/6fd03b0c/attachment-0005.png>