[ClusterLabs] SAN with drbd and pacemaker

Fri Sep 18 11:02:12 UTC 2015

On 09/18/2015 09:28 AM, Marco Marino wrote:
> Can you explain me this? 16 volumes?

With 20 disk of 4TB you have a total capacity of 80TB. If you run all of
them as RAID6 then you have a total of 72TB.

If you ask your controller to create a 8TB volume, this volume is spread
across all the 20 disk. As 2 stripes are used for parity, you have
20-2=18 data stripes per volume. This makes each stripe 444G big,
leaving 3500G free for other volumes.

If you fill up the remaining 3500G with volumes the same way, you get 8
additional volumes (OK, the last volume is <8TB then).

In total you have 9 volumes then, each disk has data/parity on all of
these volumes.

9x8=72, voila!

If a disk error appear and the controller marks the disk dead then all 9
volumes are affected.

With 20 6TB/8TB drives, you just get more 8TB volumes using this way.

What would of course reduce the risk is to always use <20 disk in one
raid6 volume, so not each disk serves all volumes.

Another issue is about performance, not every RAID controller performs
best with 20 drives. Adaptec recommends an odd number of drives, with 7
or 9 drives performs best AFAIK.

So you could make volume 1 on disks 1-9, volume 2 on disk 2-10, volume 3
on disk 3-11 etc. etc.

Or consider using some combination of RAID6 and RAID1, but this gives
you way less available disk size (and no, I have no calculation handy on
the chance for failure for RAID6 vs. RAID15 vs. RAID16)

greetings kai

> 
> Thank you
> 
> 
> 
> 2015-09-17 15:54 GMT+02:00 Kai Dupke <kdupke at suse.com>:
> 
>> On 09/17/2015 09:44 AM, Marco Marino wrote:
>>> Hi, I have 2 servers supermicro lsi 2108 with many disks (80TB) and I'm
>>> trying to build a SAN with drbd and pacemaker. I'm studying, but I have
>> no
>>> experience on large array of disks with drbd and pacemaker, so I have
>> some
>>> questions:
>>>
>>> I'm using MegaRAID Storage Manager to create virtual drives. Each virtual
>>> drive is a device on linux (eg /dev/sdb, /dev/sdc.....), so my first
>>> question is: it's a good idea to create virtual drive of 8 TB (max)? I'm
>>> thinking to rebuild array time in case of disk failure (about 1 day for 8
>>
>> It depends on your disks and RAID level. If one disk fails the content
>> of this disk has to be recreated by either copying (all RAID levels with
>> some RAID 1 included) or calculating (all with no RAID1 included), in
>> the later case all disks get really stressed.
>>
>> If you run 20x4TB disks as RAID6, then an 8TB volume is only ~500G per
>> disk. However, if one disk fails, then all the other 15 volumes this
>> disk handles are broken, too. (BTW, most raid controller can handle
>> multiple stripes per disk, but usually only a handful) In such case the
>> complete 4TB of the broken disk has to be recovered, affecting all 16
>> volumes.
>>
>> On the other side, if you use 4x5x4TB as 4x 12TB RAID6, a broken disk
>> only affects one of 4 volumes - but at the cost of more disks needed.
>>
>> You can do the similar calculation based on RAID16/15.
>>
>> The only reason I see to create small slices is to make them fit on
>> smaller replacement disks, which might be more easily available/payable
>> at time of error (but now we are entering a more low cost area where
>> usually SAN and DRBD do not take place).
>>
>> greetings
>> Kai Dupke
>> Senior Product Manager
>> Server Product Line
>> --
>> Sell not virtue to purchase wealth, nor liberty to purchase power.
>> Phone:  +49-(0)5102-9310828     Mail: kdupke at suse.com
>> Mobile: +49-(0)173-5876766      WWW:  www.suse.com
>>
>> SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
>> GF:Felix Imendörffer,Jane Smithard,Graham Norton,HRB 21284 (AG Nürnberg)
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> http://clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
> 
> 
> 
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 

Kai Dupke
Senior Product Manager
Server Product Line
-- 
Sell not virtue to purchase wealth, nor liberty to purchase power.
Phone:  +49-(0)5102-9310828     Mail: kdupke at suse.com
Mobile: +49-(0)173-5876766      WWW:  www.suse.com

SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
GF:Felix Imendörffer,Jane Smithard,Graham Norton,HRB 21284 (AG Nürnberg)