[Pacemaker] [Openais] Linux HA on debian sparc

Steven Dake sdake at redhat.com
Thu Jun 2 19:43:58 UTC 2011


On 06/01/2011 11:05 PM, william felipe_welter wrote:
> I recompile my kernel without hugetlb .. and the result are the same..
> 
> My test program still resulting:
> PATH=/dev/shm/teste123XXXXXX
> page size=20000
> fd=3
> ADDR_ORIG:0xe000a000  ADDR:0xffffffff
> Erro
> 
> And Pacemaker still resulting because the mmap error:
> Could not initialize Cluster Configuration Database API instance error 2
> 

Give the patch I posted recently a spin - corosync WFM with this patch
on sparc64 with hugetlb set.  Please report back results.

Regards
-steve

> For make sure that i have disable the hugetlb there is my /proc/meminfo:
> MemTotal:       33093488 kB
> MemFree:        32855616 kB
> Buffers:            5600 kB
> Cached:            53480 kB
> SwapCached:            0 kB
> Active:            45768 kB
> Inactive:          28104 kB
> Active(anon):      18024 kB
> Inactive(anon):     1560 kB
> Active(file):      27744 kB
> Inactive(file):    26544 kB
> Unevictable:           0 kB
> Mlocked:               0 kB
> SwapTotal:       6104680 kB
> SwapFree:        6104680 kB
> Dirty:                 0 kB
> Writeback:             0 kB
> AnonPages:         14936 kB
> Mapped:             7736 kB
> Shmem:              4624 kB
> Slab:              39184 kB
> SReclaimable:      10088 kB
> SUnreclaim:        29096 kB
> KernelStack:        7088 kB
> PageTables:         1160 kB
> Quicklists:        17664 kB
> NFS_Unstable:          0 kB
> Bounce:                0 kB
> WritebackTmp:          0 kB
> CommitLimit:    22651424 kB
> Committed_AS:     519368 kB
> VmallocTotal:   1069547520 kB
> VmallocUsed:       11064 kB
> VmallocChunk:   1069529616 kB
> 
> 
> 2011/6/1 Steven Dake <sdake at redhat.com>:
>> On 06/01/2011 07:42 AM, william felipe_welter wrote:
>>> Steven,
>>>
>>> cat /proc/meminfo
>>> ...
>>> HugePages_Total:       0
>>> HugePages_Free:        0
>>> HugePages_Rsvd:        0
>>> HugePages_Surp:        0
>>> Hugepagesize:       4096 kB
>>> ...
>>>
>>
>> It definitely requires a kernel compile and setting the config option to
>> off.  I don't know the debian way of doing this.
>>
>> The only reason you may need this option is if you have very large
>> memory sizes, such as 48GB or more.
>>
>> Regards
>> -steve
>>
>>> Its 4MB..
>>>
>>> How can i disable hugetlb ? ( passing CONFIG_HUGETLBFS=n at boot to
>>> kernel ?)
>>>
>>> 2011/6/1 Steven Dake <sdake at redhat.com <mailto:sdake at redhat.com>>
>>>
>>>     On 06/01/2011 01:05 AM, Steven Dake wrote:
>>>     > On 05/31/2011 09:44 PM, Angus Salkeld wrote:
>>>     >> On Tue, May 31, 2011 at 11:52:48PM -0300, william felipe_welter
>>>     wrote:
>>>     >>> Angus,
>>>     >>>
>>>     >>> I make some test program (based on the code coreipcc.c) and i
>>>     now i sure
>>>     >>> that are problems with the mmap systems call on sparc..
>>>     >>>
>>>     >>> Source code of my test program:
>>>     >>>
>>>     >>> #include <stdlib.h>
>>>     >>> #include <sys/mman.h>
>>>     >>> #include <stdio.h>
>>>     >>>
>>>     >>> #define PATH_MAX  36
>>>     >>>
>>>     >>> int main()
>>>     >>> {
>>>     >>>
>>>     >>> int32_t fd;
>>>     >>> void *addr_orig;
>>>     >>> void *addr;
>>>     >>> char path[PATH_MAX];
>>>     >>> const char *file = "teste123XXXXXX";
>>>     >>> size_t bytes=10024;
>>>     >>>
>>>     >>> snprintf (path, PATH_MAX, "/dev/shm/%s", file);
>>>     >>> printf("PATH=%s\n",path);
>>>     >>>
>>>     >>> fd = mkstemp (path);
>>>     >>> printf("fd=%d \n",fd);
>>>     >>>
>>>     >>>
>>>     >>> addr_orig = mmap (NULL, bytes, PROT_NONE,
>>>     >>>               MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
>>>     >>>
>>>     >>>
>>>     >>> addr = mmap (addr_orig, bytes, PROT_READ | PROT_WRITE,
>>>     >>>               MAP_FIXED | MAP_SHARED, fd, 0);
>>>     >>>
>>>     >>> printf("ADDR_ORIG:%p  ADDR:%p\n",addr_orig,addr);
>>>     >>>
>>>     >>>
>>>     >>>   if (addr != addr_orig) {
>>>     >>>                printf("Erro");
>>>     >>>         }
>>>     >>> }
>>>     >>>
>>>     >>> Results on x86:
>>>     >>> PATH=/dev/shm/teste123XXXXXX
>>>     >>> fd=3
>>>     >>> ADDR_ORIG:0x7f867d8e6000  ADDR:0x7f867d8e6000
>>>     >>>
>>>     >>> Results on sparc:
>>>     >>> PATH=/dev/shm/teste123XXXXXX
>>>     >>> fd=3
>>>     >>> ADDR_ORIG:0xf7f72000  ADDR:0xffffffff
>>>     >>
>>>     >> Note: 0xffffffff == MAP_FAILED
>>>     >>
>>>     >> (from man mmap)
>>>     >> RETURN VALUE
>>>     >>        On success, mmap() returns a pointer to the mapped area.  On
>>>     >>        error, the value MAP_FAILED (that is, (void *) -1) is
>>>     returned,
>>>     >>        and errno is  set appropriately.
>>>     >>
>>>     >>>
>>>     >>>
>>>     >>> But im wondering if is really needed to call mmap 2 times ?
>>>      What are the
>>>     >>> reason to call the mmap 2 times, on the second time using the
>>>     address of the
>>>     >>> first?
>>>     >>>
>>>     >>>
>>>     >> Well there are 3 calls to mmap()
>>>     >> 1) one to allocate 2 * what you need (in pages)
>>>     >> 2) maps the first half of the mem to a real file
>>>     >> 3) maps the second half of the mem to the same file
>>>     >>
>>>     >> The point is when you write to an address over the end of the
>>>     >> first half of memory it is taken care of the the third mmap which
>>>     maps
>>>     >> the address back to the top of the file for you. This means you
>>>     >> don't have to worry about ringbuffer wrapping which can be a
>>>     headache.
>>>     >>
>>>     >> -Angus
>>>     >>
>>>     >
>>>     > interesting this mmap operation doesn't work on sparc linux.
>>>     >
>>>     > Not sure how I can help here - Next step would be a follow up with the
>>>     > sparc linux mailing list.  I'll do that and cc you on the message
>>>     - see
>>>     > if we get any response.
>>>     >
>>>     > http://vger.kernel.org/vger-lists.html
>>>     >
>>>     >>>
>>>     >>>
>>>     >>>
>>>     >>>
>>>     >>> 2011/5/31 Angus Salkeld <asalkeld at redhat.com
>>>     <mailto:asalkeld at redhat.com>>
>>>     >>>
>>>     >>>> On Tue, May 31, 2011 at 06:25:56PM -0300, william felipe_welter
>>>     wrote:
>>>     >>>>> Thanks Steven,
>>>     >>>>>
>>>     >>>>> Now im try to run on the MCP:
>>>     >>>>> - Uninstall the pacemaker 1.0
>>>     >>>>> - Compile and install 1.1
>>>     >>>>>
>>>     >>>>> But now i have problems to initialize the pacemakerd: Could not
>>>     >>>> initialize
>>>     >>>>> Cluster Configuration Database API instance error 2
>>>     >>>>> Debbuging with gdb i see that the error are on the confdb.. most
>>>     >>>> specificaly
>>>     >>>>> the errors start on coreipcc.c  at line:
>>>     >>>>>
>>>     >>>>>
>>>     >>>>> 448        if (addr != addr_orig) {
>>>     >>>>> 449                goto error_close_unlink;  <- enter here
>>>     >>>>> 450       }
>>>     >>>>>
>>>     >>>>> Some ideia about  what can cause this  ?
>>>     >>>>>
>>>     >>>>
>>>     >>>> I tried porting a ringbuffer (www.libqb.org
>>>     <http://www.libqb.org>) to sparc and had the same
>>>     >>>> failure.
>>>     >>>> There are 3 mmap() calls and on sparc the third one keeps failing.
>>>     >>>>
>>>     >>>> This is a common way of creating a ring buffer, see:
>>>     >>>>
>>>     http://en.wikipedia.org/wiki/Circular_buffer#Exemplary_POSIX_Implementation
>>>     >>>>
>>>     >>>> I couldn't get it working in the short time I tried. It's probably
>>>     >>>> worth looking at the clib implementation to see why it's failing
>>>     >>>> (I didn't get to that).
>>>     >>>>
>>>     >>>> -Angus
>>>     >>>>
>>>
>>>     Note, we sorted this out we believe.  Your kernel has hugetlb enabled,
>>>     probably with 4MB pages.  This requires corosync to allocate 4MB pages.
>>>
>>>     Can you verify your hugetlb settings?
>>>
>>>     If you can turn this option off, you should have atleast a working
>>>     corosync.
>>>
>>>     Regards
>>>     -steve
>>>     >>>>
>>>     >>>> _______________________________________________
>>>     >>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>     <mailto:Pacemaker at oss.clusterlabs.org>
>>>     >>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>     >>>>
>>>     >>>> Project Home: http://www.clusterlabs.org
>>>     >>>> Getting started:
>>>     http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>     >>>> Bugs:
>>>     >>>>
>>>     http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>>>     >>>>
>>>     >>>
>>>     >>>
>>>     >>>
>>>     >>> --
>>>     >>> William Felipe Welter
>>>     >>> ------------------------------
>>>     >>> Consultor em Tecnologias Livres
>>>     >>> william.welter at 4linux.com.br <mailto:william.welter at 4linux.com.br>
>>>     >>> www.4linux.com.br <http://www.4linux.com.br>
>>>     >>
>>>     >>> _______________________________________________
>>>     >>> Openais mailing list
>>>     >>> Openais at lists.linux-foundation.org
>>>     <mailto:Openais at lists.linux-foundation.org>
>>>     >>> https://lists.linux-foundation.org/mailman/listinfo/openais
>>>     >>
>>>     >>
>>>     >> _______________________________________________
>>>     >> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>     <mailto:Pacemaker at oss.clusterlabs.org>
>>>     >> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>     >>
>>>     >> Project Home: http://www.clusterlabs.org
>>>     >> Getting started:
>>>     http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>     >> Bugs:
>>>     http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>>>     >
>>>     > _______________________________________________
>>>     > Openais mailing list
>>>     > Openais at lists.linux-foundation.org
>>>     <mailto:Openais at lists.linux-foundation.org>
>>>     > https://lists.linux-foundation.org/mailman/listinfo/openais
>>>
>>>
>>>     _______________________________________________
>>>     Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>     <mailto:Pacemaker at oss.clusterlabs.org>
>>>     http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>>     Project Home: http://www.clusterlabs.org
>>>     Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>     Bugs:
>>>     http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>>>
>>>
>>>
>>>
>>> --
>>> William Felipe Welter
>>> ------------------------------
>>> Consultor em Tecnologias Livres
>>> william.welter at 4linux.com.br <mailto:william.welter at 4linux.com.br>
>>> www.4linux.com.br <http://www.4linux.com.br>
>>>
>>>
>>>
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>>
>>
> 
> 
> 





More information about the Pacemaker mailing list