[ClusterLabs] Pacemaker detail log directory permissions

Tue Apr 30 07:55:18 EDT 2019

On 30/04/19 07:55 +0200, Ulrich Windl wrote:
>>>> Jan Pokorný <jpokorny at redhat.com> schrieb am 29.04.2019 um 17:22
>>>> in Nachricht <20190429152200.GA19102 at redhat.com>:
>> On 29/04/19 14:58 +0200, Jan Pokorný wrote:
>>> On 29/04/19 08:20 +0200, Ulrich Windl wrote:
>>>>>>> Jan Pokorný <jpokorny at redhat.com> schrieb am 25.04.2019 um 18:49
>>>>>>> in Nachricht <20190425164946.GF23995 at redhat.com>:
>>>>> I think the prime and foremost use case is that half of the actual
>>>>> pacemaker daemons run as hacluster:haclient themselves, and it's
>>>>> preferred for them to be not completely muted about what they do,
>>>>> correct? :‑)
>>>> 
>>>> I interesting aspect: So the commands write directly to one file
>>>> concurrently?  How's synchronization done. It's especially
>>>> interesting with multiple threads.
>>> 
>>> You don't believe your libc (fprintf in particular) to abide the
>>> POSIX constraints, do you? :-)
>>> 
>>>>   All functions that reference (FILE *) objects, except those with
>>>>   names ending in _unlocked, shall behave as if they use flockfile()
>>>>   and funlockfile() internally to obtain ownership of these (FILE *)
>>>>   objects.
>>> 
>>> 
> [https://pubs.opengroup.org/onlinepubs/9699919799/functions/flockfile.html]
>>> 
>>>> Wouldn't it be much easier to use some socket-bases syslog like
>>>> service that serializes all the messages to one file. Then the
>>>> clients just need write access to the socket, not to the files.
> 
> I agree that multiple threads in one thread have no problem using
> printf(), but (at least in the buffered case) if multiple processes
> write to the same file, that type of locking doesn't help much IMHO.

Oops, you are right, I made a logical shortcut connecting flockfile(3)
and flock(1), which was entirely unbacked.  You are correct it would
matter only amongst the threads, not otherwise unsynchronized processes.
Sorry about the noise :-/

Shamefully, this rather important (nobody wants garbled log messages)
aspect is in no way documented in libqb's context (it does not do any
explicit locking on its own), especially since the length of the logged
messages can go above the default of 512 B (in the upcoming libqb 2,
IIUIC) ... and luckily, I was steering the direction to still stay
modest and cap that on 4 kiB, even if for other reasons:

https://github.com/ClusterLabs/libqb/pull/292#issuecomment-361745575

which still might be within Linux + typical FSs (ext4) boundaries
to guarantee atomicity of an append (or maybe not even that, it all
seems a gray area of any guarantees provided by the underlying system,
inputs from the experts welcome).  Anyway, ISTM that we should at the
very least increase the buffer size for block buffering to the
configured one (up to 4 kiB as mentioned) if BUFSIZ would be less,
to prevent tainting this expected atomicity from the get-go.

Anyway, correctness of the message durability is still dependent
on the syslog implementation to behave correctly if syslog logging
sink is enabled (holds for both corosync and pacemaker daemons),
so file-based logging can still be considered optional, and e.g.
for pacemaker, it should be possible to turn it off altogether in
the future as mentioned (already possible with corosync, IIRC).

Lowest level I/O a very complex and difficult area to reason about,
sadly, we didn't do our homework at the very beginning, so problems
are likely to snowball (incl. brain farts like the one I did above
... it might have been avoided if the contraints of libqb logging
usage were documented properly).

> (In clasical syslog there's _one_ process that receives the messages
> and writes them to files. I can only remember HP-UX kernel messages
> that were synchronized byte-wise, so if multiple cores emitted the
> same message concurrently you go some funny messages like
> "OpOpeeniingng"... ;_)

Truly nasty to not have even somewhat atomic blocks (see block
based buffering, which is a default for fprintf et al. unless the
output stream equals stdout) that gets interleaved.

>> FWIW, have just realized that by virtue of this, we have actually
>> silently alleviated a hidden lock contention (against the realtime
>> corosync process! but allegedly, it is not too verbose if nothing
>> interesting happens unless set to be more verbose) since
>> Pacemaker-2.0.0:l
>> 
>> [...]
>> 

Already disproved, thanks to Ulrich, sorry.

>> But we can do better for those who would prefer to serialize
>> multiple (per daemon) file outputs by hand (if ever at all, complete
>> serialization can be found in the syslog sink, perhaps) as briefly
>> touched on earlier in this thread[1].
>> 
>> [1] https://lists.clusterlabs.org/pipermail/users/2019-April/025709.html 

That would be a low-effort assuring the atomicity (except for
spurious duplicate instances of the same daemon, but that shall be
reduced to some extent with the fix of the recent important-ranked CVE).

-- 
Jan (Poki)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20190430/cf4bee48/attachment.sig>