[ClusterLabs] drbd could not start by pacemaker. strange limited root privileges?

[Ken Gaillot]

On Thu, 2019-05-23 at 13:21 +0200, László Neduki wrote:
> Hi,
> 
> (
> I sent a similar question from an other acount 3 days ago, but: 
> - I do not see it on the list. Maybe I should not see my own email? 

A DRBD message from govomevo at gmail.com did make it to the list a week
ago. You should get your own emails from the list server, though your
own mail server or client might filter them.

> So I created a new account
> - I have additional infos (but no solution), so I rewrite the
> question
> )
> 
> pacemaker cannot start drbd9 resources. As I see, root has very
> limited privileges in the drbd resource agent, when it run by the
> pacemaker. I downloaded the latest pacemaker this week, and I
> compiled drbd9 rpms also. I hope, You can help me, I do not find the
> cause of this behaviour. Please see the below test cases:

I'm not a DRBD expert, but given the symptoms you describe, my first
thoughts would be that either the ocf:linbit:drbd agent you're using
isn't the right version for your DRBD version, or something like
SELinux is restricting access.

> 1. When I create Pacemaker DRBD resource I get errors
> # pcs resource create DrbdDB ocf:linbit:drbd drbd_resource=drbd_db op
> monitor interval=60s meta notify=true
> # pcs resource master DrbdDBClone DrbdDB master-max=1 master-node-
> max=1 clone-node-max=1 notify=true
> # pcs constraint location DrbdDBClone prefers node1=INFINITY
> # pcs cluster stop --all; pcs cluster start --all; pcs status
> 
> Failed Actions:
> * DrbdDB_monitor_0 on node1 'not installed' (5): call=6,
> status=complete, exitreason='DRBD kernel (module) not available?',
>     last-rc-change='Thu May 23 09:54:09 2019', queued=0ms, exec=58ms
> * DrbdDB_monitor_0 on node2 'not installed' (5): call=6,
> status=complete, exitreason='DRBD kernel (module) not available?',
>     last-rc-change='Thu May 23 10:00:22 2019', queued=0ms, exec=71ms
> 
> 2. when I try to start drbd_db by drbdadm directly, it works well:
> # modprobe drbd #on each node
> # drbdadm up drbd_db #on each node
> # drbdadm primary drbd_db
> # drbdadm status 
> it shows drbd_db is UpToDate on each node
> I also can promote and mount filesystem well
> 
> 3. When I use debug-start, it works fine (so the resource syntax
> sould be correct)
> # drbdadm status
> No currently configured DRBD found.
> # pcs resource debug-start DrbdDBMaster
> Error: unable to debug-start a master, try the master's resource:
> DrbdDB
> # pcs resource debug-start DrbdDB #on each node
> Operation start for DrbdDB:0 (ocf:linbit:drbd) returned: 'ok' (0)
> # drbdadm status
> it shows drbd_db is UpToDate on each node
> 
> 4. Pacemaker handle other resources well . If I set auto_promote=yes,
> and I start (but not promote) the drbd_db by drbdadm, then pacemaker
> can create filesystem on it well, and also the appserver, database
> resources. 
> 
> 5. The strangest behaviour for me. Root have very limited privileges
> whitin the drbd resource agent. If I write this line to the
> srbd_start() method of  /usr/lib/ocf/resource.d/linbit/drbd
> 
> ocf_log err "lados " $(whoami) $( ls -l /home/opc/tmp/modprobe2.trace
> ) $( do_cmd touch /home/opc/tmp/modprobe2.trace )
> 
> I got theese messeges in log, when I start the cluster
> 
> # tail -f /var/log/cluster/corosync.log | grep -A 8 -B 3 -i lados
> 
> ...
> May 21 15:35:12  drbd(DrbdDB)[31649]:    ERROR: lados  root
> May 21 15:35:12 [31309] node1       lrmd:   notice:
> operation_finished:    DrbdDB_start_0:31649:stderr [ ls: cannot
> access /home/opc/tmp/modprobe2.trace: Permission denied ]
> May 21 15:35:12 [31309] node1       lrmd:   notice:
> operation_finished:    DrbdFra_start_0:31649:stderr [ touch: cannot
> touch '/home/opc/tmp/modprobe2.trace': Permission denied ]
> ...
> and also, when I try to strace the "modprobe -s drbd `$DRBDADM sh-
> mod-parms`" in drbd resource agent, I only see 1 line in the
> /root/modprobe2.trace. This meens for me:
> - root cannot trace the calls in drbdadm (even if root can strace
> drbdadm outside of pacemaker well)
> - root can write into files his own directory
> (/root/modprobe2.trace) 
> 
> 6. Opposit of previous test
> root has these privileges outside from pacamaker
> 
> # sudo su -
> # touch /home/opc/tmp/modprobe2.trace
> # ls -l /home/opc/tmp/modprobe2.trace
> -rw-r--r--. 1 root root 0 May 21 15:44 /home/opc/tmp/modprobe2.trace
> 
> 
> Thanks: lados.
> 
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> ClusterLabs home: https://www.clusterlabs.org/
-- 
Ken Gaillot <kgaillot at redhat.com>