[Pacemaker] [Openais] Linux HA on debian sparc
Steven Dake
sdake at redhat.com
Thu Jun 2 19:43:58 UTC 2011
On 06/01/2011 11:05 PM, william felipe_welter wrote:
> I recompile my kernel without hugetlb .. and the result are the same..
>
> My test program still resulting:
> PATH=/dev/shm/teste123XXXXXX
> page size=20000
> fd=3
> ADDR_ORIG:0xe000a000 ADDR:0xffffffff
> Erro
>
> And Pacemaker still resulting because the mmap error:
> Could not initialize Cluster Configuration Database API instance error 2
>
Give the patch I posted recently a spin - corosync WFM with this patch
on sparc64 with hugetlb set. Please report back results.
Regards
-steve
> For make sure that i have disable the hugetlb there is my /proc/meminfo:
> MemTotal: 33093488 kB
> MemFree: 32855616 kB
> Buffers: 5600 kB
> Cached: 53480 kB
> SwapCached: 0 kB
> Active: 45768 kB
> Inactive: 28104 kB
> Active(anon): 18024 kB
> Inactive(anon): 1560 kB
> Active(file): 27744 kB
> Inactive(file): 26544 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 6104680 kB
> SwapFree: 6104680 kB
> Dirty: 0 kB
> Writeback: 0 kB
> AnonPages: 14936 kB
> Mapped: 7736 kB
> Shmem: 4624 kB
> Slab: 39184 kB
> SReclaimable: 10088 kB
> SUnreclaim: 29096 kB
> KernelStack: 7088 kB
> PageTables: 1160 kB
> Quicklists: 17664 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 22651424 kB
> Committed_AS: 519368 kB
> VmallocTotal: 1069547520 kB
> VmallocUsed: 11064 kB
> VmallocChunk: 1069529616 kB
>
>
> 2011/6/1 Steven Dake <sdake at redhat.com>:
>> On 06/01/2011 07:42 AM, william felipe_welter wrote:
>>> Steven,
>>>
>>> cat /proc/meminfo
>>> ...
>>> HugePages_Total: 0
>>> HugePages_Free: 0
>>> HugePages_Rsvd: 0
>>> HugePages_Surp: 0
>>> Hugepagesize: 4096 kB
>>> ...
>>>
>>
>> It definitely requires a kernel compile and setting the config option to
>> off. I don't know the debian way of doing this.
>>
>> The only reason you may need this option is if you have very large
>> memory sizes, such as 48GB or more.
>>
>> Regards
>> -steve
>>
>>> Its 4MB..
>>>
>>> How can i disable hugetlb ? ( passing CONFIG_HUGETLBFS=n at boot to
>>> kernel ?)
>>>
>>> 2011/6/1 Steven Dake <sdake at redhat.com <mailto:sdake at redhat.com>>
>>>
>>> On 06/01/2011 01:05 AM, Steven Dake wrote:
>>> > On 05/31/2011 09:44 PM, Angus Salkeld wrote:
>>> >> On Tue, May 31, 2011 at 11:52:48PM -0300, william felipe_welter
>>> wrote:
>>> >>> Angus,
>>> >>>
>>> >>> I make some test program (based on the code coreipcc.c) and i
>>> now i sure
>>> >>> that are problems with the mmap systems call on sparc..
>>> >>>
>>> >>> Source code of my test program:
>>> >>>
>>> >>> #include <stdlib.h>
>>> >>> #include <sys/mman.h>
>>> >>> #include <stdio.h>
>>> >>>
>>> >>> #define PATH_MAX 36
>>> >>>
>>> >>> int main()
>>> >>> {
>>> >>>
>>> >>> int32_t fd;
>>> >>> void *addr_orig;
>>> >>> void *addr;
>>> >>> char path[PATH_MAX];
>>> >>> const char *file = "teste123XXXXXX";
>>> >>> size_t bytes=10024;
>>> >>>
>>> >>> snprintf (path, PATH_MAX, "/dev/shm/%s", file);
>>> >>> printf("PATH=%s\n",path);
>>> >>>
>>> >>> fd = mkstemp (path);
>>> >>> printf("fd=%d \n",fd);
>>> >>>
>>> >>>
>>> >>> addr_orig = mmap (NULL, bytes, PROT_NONE,
>>> >>> MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
>>> >>>
>>> >>>
>>> >>> addr = mmap (addr_orig, bytes, PROT_READ | PROT_WRITE,
>>> >>> MAP_FIXED | MAP_SHARED, fd, 0);
>>> >>>
>>> >>> printf("ADDR_ORIG:%p ADDR:%p\n",addr_orig,addr);
>>> >>>
>>> >>>
>>> >>> if (addr != addr_orig) {
>>> >>> printf("Erro");
>>> >>> }
>>> >>> }
>>> >>>
>>> >>> Results on x86:
>>> >>> PATH=/dev/shm/teste123XXXXXX
>>> >>> fd=3
>>> >>> ADDR_ORIG:0x7f867d8e6000 ADDR:0x7f867d8e6000
>>> >>>
>>> >>> Results on sparc:
>>> >>> PATH=/dev/shm/teste123XXXXXX
>>> >>> fd=3
>>> >>> ADDR_ORIG:0xf7f72000 ADDR:0xffffffff
>>> >>
>>> >> Note: 0xffffffff == MAP_FAILED
>>> >>
>>> >> (from man mmap)
>>> >> RETURN VALUE
>>> >> On success, mmap() returns a pointer to the mapped area. On
>>> >> error, the value MAP_FAILED (that is, (void *) -1) is
>>> returned,
>>> >> and errno is set appropriately.
>>> >>
>>> >>>
>>> >>>
>>> >>> But im wondering if is really needed to call mmap 2 times ?
>>> What are the
>>> >>> reason to call the mmap 2 times, on the second time using the
>>> address of the
>>> >>> first?
>>> >>>
>>> >>>
>>> >> Well there are 3 calls to mmap()
>>> >> 1) one to allocate 2 * what you need (in pages)
>>> >> 2) maps the first half of the mem to a real file
>>> >> 3) maps the second half of the mem to the same file
>>> >>
>>> >> The point is when you write to an address over the end of the
>>> >> first half of memory it is taken care of the the third mmap which
>>> maps
>>> >> the address back to the top of the file for you. This means you
>>> >> don't have to worry about ringbuffer wrapping which can be a
>>> headache.
>>> >>
>>> >> -Angus
>>> >>
>>> >
>>> > interesting this mmap operation doesn't work on sparc linux.
>>> >
>>> > Not sure how I can help here - Next step would be a follow up with the
>>> > sparc linux mailing list. I'll do that and cc you on the message
>>> - see
>>> > if we get any response.
>>> >
>>> > http://vger.kernel.org/vger-lists.html
>>> >
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> 2011/5/31 Angus Salkeld <asalkeld at redhat.com
>>> <mailto:asalkeld at redhat.com>>
>>> >>>
>>> >>>> On Tue, May 31, 2011 at 06:25:56PM -0300, william felipe_welter
>>> wrote:
>>> >>>>> Thanks Steven,
>>> >>>>>
>>> >>>>> Now im try to run on the MCP:
>>> >>>>> - Uninstall the pacemaker 1.0
>>> >>>>> - Compile and install 1.1
>>> >>>>>
>>> >>>>> But now i have problems to initialize the pacemakerd: Could not
>>> >>>> initialize
>>> >>>>> Cluster Configuration Database API instance error 2
>>> >>>>> Debbuging with gdb i see that the error are on the confdb.. most
>>> >>>> specificaly
>>> >>>>> the errors start on coreipcc.c at line:
>>> >>>>>
>>> >>>>>
>>> >>>>> 448 if (addr != addr_orig) {
>>> >>>>> 449 goto error_close_unlink; <- enter here
>>> >>>>> 450 }
>>> >>>>>
>>> >>>>> Some ideia about what can cause this ?
>>> >>>>>
>>> >>>>
>>> >>>> I tried porting a ringbuffer (www.libqb.org
>>> <http://www.libqb.org>) to sparc and had the same
>>> >>>> failure.
>>> >>>> There are 3 mmap() calls and on sparc the third one keeps failing.
>>> >>>>
>>> >>>> This is a common way of creating a ring buffer, see:
>>> >>>>
>>> http://en.wikipedia.org/wiki/Circular_buffer#Exemplary_POSIX_Implementation
>>> >>>>
>>> >>>> I couldn't get it working in the short time I tried. It's probably
>>> >>>> worth looking at the clib implementation to see why it's failing
>>> >>>> (I didn't get to that).
>>> >>>>
>>> >>>> -Angus
>>> >>>>
>>>
>>> Note, we sorted this out we believe. Your kernel has hugetlb enabled,
>>> probably with 4MB pages. This requires corosync to allocate 4MB pages.
>>>
>>> Can you verify your hugetlb settings?
>>>
>>> If you can turn this option off, you should have atleast a working
>>> corosync.
>>>
>>> Regards
>>> -steve
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> <mailto:Pacemaker at oss.clusterlabs.org>
>>> >>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>> >>>>
>>> >>>> Project Home: http://www.clusterlabs.org
>>> >>>> Getting started:
>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> >>>> Bugs:
>>> >>>>
>>> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> William Felipe Welter
>>> >>> ------------------------------
>>> >>> Consultor em Tecnologias Livres
>>> >>> william.welter at 4linux.com.br <mailto:william.welter at 4linux.com.br>
>>> >>> www.4linux.com.br <http://www.4linux.com.br>
>>> >>
>>> >>> _______________________________________________
>>> >>> Openais mailing list
>>> >>> Openais at lists.linux-foundation.org
>>> <mailto:Openais at lists.linux-foundation.org>
>>> >>> https://lists.linux-foundation.org/mailman/listinfo/openais
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> <mailto:Pacemaker at oss.clusterlabs.org>
>>> >> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>> >>
>>> >> Project Home: http://www.clusterlabs.org
>>> >> Getting started:
>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> >> Bugs:
>>> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>>> >
>>> > _______________________________________________
>>> > Openais mailing list
>>> > Openais at lists.linux-foundation.org
>>> <mailto:Openais at lists.linux-foundation.org>
>>> > https://lists.linux-foundation.org/mailman/listinfo/openais
>>>
>>>
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> <mailto:Pacemaker at oss.clusterlabs.org>
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs:
>>> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>>>
>>>
>>>
>>>
>>> --
>>> William Felipe Welter
>>> ------------------------------
>>> Consultor em Tecnologias Livres
>>> william.welter at 4linux.com.br <mailto:william.welter at 4linux.com.br>
>>> www.4linux.com.br <http://www.4linux.com.br>
>>>
>>>
>>>
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>>
>>
>
>
>
More information about the Pacemaker
mailing list