[ClusterLabs] Performance of a mirrored LV (cLVM) with OCFS: Attempt to monitor it

[Ulrich Windl]

Hi!

I am developing some monitoring program that does direct I/O (read) on files and devices to measure the end-to-end delay. End-to-end delay, because I think thatis the only value the user is really interested in; especially when you start hunting for problems.

We use a combination of mirrored cLVM LUN (both mirror legs on a FC-SAN) and OCFS2 to provide a shared filesystem for Xen-VM images (PV guests).

The problem all started when I used another monitor (written in Perl) that does a flock() with a 3-second timeout in a Xen-VM to monitor a local LDAP server: At some times of the day the lock times out, even if the file is only locked for a fragment of a second. As (to my understanding) flock() does no real I/O, the rest of the system must be very slow. So I started to dig...

I added a monitor for both LVM-Legs, the LV itself (the only LV in the VG), and the OCFS2 filesystem on top of the LV (Networking is all in a LAN, in case that should be significant).

Before I start, let me explain the measurement components:
I keep the last n samples in a queue, where the folowing "d=dynamic" values are derived from:
davg = Average from the delay values in the queue
dstd_dev = Standard deviation for the delay values in the queue
last = last delay value measured
I also have two "quick attack" exponentially averaged values (e=exponential):
emin = smallest value found in the queue (new minimum is immediately used, then exponential average updates are used)
emax = largest value found in the queue  (new maximum is immediately used, then exponential average updates are used)

This way, I can make rather fast samples (like every 3 seconds) without killing the performance of Nagios that polls my monitor every few minutes. All measurements were done with reading 512 bytes every 3 seconds in a thread with FIFO realtime priority 79 and "IOPRIO_PRIO_VALUE(IOPRIO_CLASS_RT, 7)" (in case the system is very busy).

So what I found out is this: The legs of the mirrored LV work rather well (davg < 1.5ms (), emax < 12 ms (96ms)), but the cLVM LV is rather slow (davg < 56ms, emax < 1.5s!). So cLVM adds an extra delay of at least 1.4 seconds at peak. OCFS has davg < 54ms and emax < 1.5s, so it does NOT add any significant time to the LV timing.

cLVM has never made a good impression regarding performance, so I wonder if there's anything we could do to improve the4 performance. I suspect that one VM paging heavily on OCFS2 kills the performance of the whole cluster (that hosts Xen PV guests only). Anyone with deeper insights?

I'll add sample graphs for the described configuration (all measurements taken at one node of the 3-node cluster), apologizing for the poor selection of colurs (I didn't tune the defaults). You can see that everything vanishes under the emax.

For reference I'll also attach a graph (measuring NFS delay) using a static min/max, to demonstrate why I changed to a "dynamic/exponential" min/max ;-)

And before you ask: "IOTW" stands for "I/O Timing Watch"...

Regards,
Ulrich



-------------- next part --------------
A non-text attachment was scrubbed...
Name: IOTW-cLVM-Leg2.png
Type: image/png
Size: 34927 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20160425/c67f574f/attachment-0015.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IOTW-NFS-Test.png
Type: image/png
Size: 32908 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20160425/c67f574f/attachment-0016.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IOTW-cLVM-OCFS2.png
Type: image/png
Size: 36401 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20160425/c67f574f/attachment-0017.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IOTW-cLVM-Leg1.png
Type: image/png
Size: 42825 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20160425/c67f574f/attachment-0018.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IOTW-cLVM-xen-LV.png
Type: image/png
Size: 37418 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20160425/c67f574f/attachment-0019.png>