[ClusterLabs] Issue with DRBD + a systemd resource

[Ken Gaillot]

On Thu, 2017-12-14 at 20:59 +0300, Andrei Borzenkov wrote:
> 14.12.2017 19:25, Jan Pokorný пишет:
> > On 14/12/17 10:49 -0500, Julien Semaan wrote:
> > > Great success!
> > > 
> > > Adding the following line to
> > > /usr/lib/systemd/system/pacemaker.service did
> > > it:
> > > After=dbus.service
> > 
> > Note, this is not a proper way for overriding the systemd unit
> > files,
> > which is rather along the lines:
> > - make a copy to /etc/systemd/system directory
> > - edit this copy
> > - issue systemctl daemon-reload so that this is taken into at all
> >   (which also happens automatically after reboot)
> > 
> > Otherwise your changes are lost say after reinstalling pacemaker.
> > 
> 
> That's not the right way to do it either unless you have really
> ancient
> systemd version. The right way is to create configuration drop-in for
> your additional settings only:
> 
> mkdir /etc/systemd/system/pacemaker.service.d
> echo "[Unit]
> After=dbus.service" >
> /etc/systemd/system/pacemaker.service.d/after-dbus.conf
> systemctl daemon-reload
> 
> You can also use "systemctl edit pacemaker.service" which under the
> hood
> does the same.
> 
> And on SUSE you can add this directory to csync2 configuration to
> ensure
> each node has the same settings (do not know what is the equivalent
> on RH).
> 
> > (FTR, in the same vein, I am proposing similar scheme for resource
> > agents: https://github.com/ClusterLabs/OCF-spec/issues/5)
> > 
> > > Now, the question is, should the unit file shipped in the RPM be
> > > adjusted
> > > (currently using CentOS 7), if so, is this the best place to get
> > > the message
> > > going, or should I post this to a specific BTS ?
> > 
> > Pacemaker maintainers are reading this so will take care of this
> > for upstream, you may then want to give a heads up to particular
> > downstream, giving them the right pointers.

We actually used to have After/Requires=dbus.service in the unit file,
but a systemd developer who reviewed our unit file told us that it was
redundant, so we removed it in 1.1.17. :-(

Apparently that was mistaken, and we need to put it back. I'll make
sure it gets into 2.0, and probably backport to the 1.1 branch as well.

> > 
> > Anyway, the change is seemingly straightfoward, but few things
> > should be answered/investigated first:
> > - After=dbus.service or rather After=dbus.socket (or both)?
> 
> It must be dbus.service. On startup it does not really matter but on
> shutdown D-Bus daemon may still be stopped before pacemaker without
> explicit dependency on dbus.service.
> 
> Note that systemd has private D-Bus socket which is used when D-Bus
> daemon is not available. systemctl (should) transparently use it when
> it
> cannot connect to system D-Bus daemon.
> 
> > - mere After=, or combined with other relationship modifying
> >   directives
> >   - Wants=, Requires= ... what's the expected behaviour when
> >     DBus crashes (or is it merely hypothetical because then
> >     whole systemd is in trouble then?)
> 
> If I correctly understand the problem, pacemaker itself does not
> depend
> on or need D-Bus; it only needs it when there are resources of class
> systemd. Crash and restart of D-Bus daemon kills existing D-Bus
> connections; does pacemaker (lrmd) has any long-lived connections or
> it
> connects on demand? In the latter case D-Bus daemon restart should
> not
> really affect it unless it happens exactly when it is performing
> operation on resource.

Pacemaker does maintain a long-lived connection to DBus for efficiency,
but it is able to detect connection failures and reconnect.

> Note that dbus.service is by far not the only one additional
> dependency
> you may need. E.g. any NFS mount resource requires that
> pacemaker.service is ordered after remote-fs.target, otherwise NFS
> mounts may be forcibly teared down before pacemaker has chance to
> cleanly stop them and all dependent resources.
> 
> It would actually be interesting to have framework that allows
> dynamically add dependencies which are required by individual RA.

It would :-)

We added it in 1.1.17. Pacemaker now is ordered after resource-agents-
deps.target, which resource-agents ships by default as a no-op. It is
intended for site-specific requirements that are not themselves managed
by pacemaker yet are required by resources managed by pacemaker.
Sysadmins can add unit overrides to add their local requirements to the
target.

dbus.service, by contrast, is required for any systemd-class resource,
so it makes sense to have that explicit dependency in the pacemaker
unit file. Certainly, it would work as a work-around in the meantime to
add dbus to the resource-agents-deps target.

> 
> >   - should DBus reloads somehow affect pacemaker?
> 
> D-Bus restart is considered too harmful and explicitly not supported
> even though it should not affect pacemaker itself (see above).
> 
> > - (insert some other questions that are not currently
> >   coming to mind of rather inexpert person)
> > 
> 
> systemd is the whole new can of worms. Just look at disastrous effect
> innocent, well established setting of SBD_DELAY_START="true" may
> have.
-- 
Ken Gaillot <kgaillot at redhat.com>