[ClusterLabs] Issue with DRBD + a systemd resource

[Jan Pokorný]

On 14/12/17 20:59 +0300, Andrei Borzenkov wrote:
> 14.12.2017 19:25, Jan Pokorný пишет:
>> On 14/12/17 10:49 -0500, Julien Semaan wrote:
>>> Great success!
>>> 
>>> Adding the following line to /usr/lib/systemd/system/pacemaker.service did
>>> it:
>>> After=dbus.service
>> 
>> Note, this is not a proper way for overriding the systemd unit files,
>> which is rather along the lines:
>> - make a copy to /etc/systemd/system directory
>> - edit this copy
>> - issue systemctl daemon-reload so that this is taken into at all
>>   (which also happens automatically after reboot)
>> 
>> Otherwise your changes are lost say after reinstalling pacemaker.
>> 
> 
> That's not the right way to do it either unless you have really ancient
> systemd version. The right way is to create configuration drop-in for
> your additional settings only:
> 
> mkdir /etc/systemd/system/pacemaker.service.d
> echo "[Unit]
> After=dbus.service" >
> /etc/systemd/system/pacemaker.service.d/after-dbus.conf
> systemctl daemon-reload

That's not what systemd.unit(5) says, it still spells out the former
first, then states the latter as an alternative.

I am not sure what's clearly better or trendy now, but good that
you pointed the alternative out so that people can get unbiased
view :-)

> You can also use "systemctl edit pacemaker.service" which under the hood
> does the same.

Never used that, good to know.

> And on SUSE you can add this directory to csync2 configuration to ensure
> each node has the same settings (do not know what is the equivalent on RH).

Customizable multisync for basicm, eventual unification of the set
of files across the nodes  ... admittedly there's currently a gap.

>> (FTR, in the same vein, I am proposing similar scheme for resource
>> agents: https://github.com/ClusterLabs/OCF-spec/issues/5)
>> 
>>> Now, the question is, should the unit file shipped in the RPM be adjusted
>>> (currently using CentOS 7), if so, is this the best place to get the message
>>> going, or should I post this to a specific BTS ?
>> 
>> Pacemaker maintainers are reading this so will take care of this
>> for upstream, you may then want to give a heads up to particular
>> downstream, giving them the right pointers.
>> 
>> Anyway, the change is seemingly straightfoward, but few things
>> should be answered/investigated first:
>> - After=dbus.service or rather After=dbus.socket (or both)?
> 
> It must be dbus.service. On startup it does not really matter but on
> shutdown D-Bus daemon may still be stopped before pacemaker without
> explicit dependency on dbus.service.

Ah, how could I miss that, thanks for clarification.

> Note that systemd has private D-Bus socket which is used when D-Bus
> daemon is not available. systemctl (should) transparently use it when it
> cannot connect to system D-Bus daemon.

Ah, that would explain my observations.

>> - mere After=, or combined with other relationship modifying
>>   directives
>>   - Wants=, Requires= ... what's the expected behaviour when
>>     DBus crashes (or is it merely hypothetical because then
>>     whole systemd is in trouble then?)
> 
> If I correctly understand the problem, pacemaker itself does not depend
> on or need D-Bus; it only needs it when there are resources of class
> systemd. Crash and restart of D-Bus daemon kills existing D-Bus
> connections; does pacemaker (lrmd) has any long-lived connections or it
> connects on demand?

Not sure if it's absolutely robust, but pacemaker checks the state of
cached connection prior to triggering new operations through that,
reconnecting when needed.

> In the latter case D-Bus daemon restart should not really affect it
> unless it happens exactly when it is performing operation on
> resource.
> 
> Note that dbus.service is by far not the only one additional dependency
> you may need. E.g. any NFS mount resource requires that
> pacemaker.service is ordered after remote-fs.target, otherwise NFS
> mounts may be forcibly teared down before pacemaker has chance to
> cleanly stop them and all dependent resources.

All good points.

> It would actually be interesting to have framework that allows
> dynamically add dependencies which are required by individual RA.

There is, actually:

https://github.com/ClusterLabs/resource-agents/pull/974/files
+
https://github.com/ClusterLabs/pacemaker/commit/06e2e269091ba69e699301d8c86c58ef94809be0#diff-3586c09cb77fc693548c38cc8906291bR17

(with the impulse being https://bugzilla.redhat.com/1316130)

>>   - should DBus reloads somehow affect pacemaker?
> 
> D-Bus restart is considered too harmful and explicitly not supported
> even though it should not affect pacemaker itself (see above).

Came to the same conclusion (see the parallel response to myself).

>> - (insert some other questions that are not currently
>>   coming to mind of rather inexpert person)
>> 
> 
> systemd is the whole new can of worms. Just look at disastrous effect
> innocent, well established setting of SBD_DELAY_START="true" may have.

-- 
Jan (Poki)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20171214/b33177d1/attachment-0002.sig>