[ClusterLabs] resource agent OCF_HEARTBEAT_GALERA issue/broken - ?
Reid Wahl
nwahl at redhat.com
Wed Jul 27 19:35:27 EDT 2022
On Wed, Jul 27, 2022 at 4:33 PM Reid Wahl <nwahl at redhat.com> wrote:
>
> On Wed, Jul 27, 2022 at 2:08 AM lejeczek via Users
> <users at clusterlabs.org> wrote:
> >
> >
> >
> > On 26/07/2022 20:56, Reid Wahl wrote:
> > > On Tue, Jul 26, 2022 at 4:21 AM lejeczek via Users
> > > <users at clusterlabs.org> wrote:
> > >> Hi guys
> > >>
> > >> I set up a clone of a new instance of mariadb galera - which otherwise,
> > >> outside of pcs works - but I see something weird.
> > >>
> > >> Firstly cluster claims it's all good:
> > >>
> > >> -> $ pcs status --full
> > >> ...
> > >>
> > >> * Clone Set: mariadb-apps-clone [mariadb-apps] (promotable):
> > >> * mariadb-apps (ocf::heartbeat:galera): Master
> > >> sucker.internal.ccn
> > >> * mariadb-apps (ocf::heartbeat:galera): Master
> > >> drunk.internal.ccn
> > > Clearly the problem is that your server is drunk.
> > >
> > >> but that mariadb is _not_ started actually.
> > >>
> > >> In clone's attr I set:
> > >>
> > >> config=/apps/etc/mariadb-server.cnf
> > >>
> > >> I also for peace of mind set:
> > >>
> > >> datadir=/apps/mysql/data
> > >>
> > >> even tough '/apps/etc/mariadb-server.cnf' declares that & other bits -
> > >> again, works outside of pcs.
> > >>
> > >> Then I see in pacemaker logs:
> > >>
> > >> notice: mariadb-apps_start_0 at drunk.internal.ccn output [ 220726 11:56:13
> > >> mysqld_safe Logging to '/tmp/tmp.On5VnzOyaF'.\n220726 11:56:13
> > >> mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql\n ]
> > >>
> > >> .. and I think what the F?
> > >>
> > >> resource-agents-4.9.0-22.el8.x86_64
> > >>
> > >> All thoughts share are much appreciated.
> > >>
> > >> many thanks, L.
> > >>
> > > How do you start mariadb outside of pacemaker's control?
> > simply by:
> > -> $ sudo -u mysql /usr/libexec/mysqld
> > --defaults-file=/apps/etc/mariadb-server.cnf
> > --wsrep-new-clusterrep
>
> Got it. FWIW, `--wsrep-new-clusterrep` doesn't appear to be a valid
> option (no search results and nothing in the help output). Maybe this
> is a typo in the email; `--wsrep-new-cluster` is valid.
>
> > >
> > > It seems that *something* is running, based on the "Starting" message
> > > and the fact that the resources are still in Started state...
> > Yes, a "regular" instance of galera (which outside of
> > pacemaker works with "regular" systemd) is running as ofc
> > one resource already.
> >
> > >
> > > The logic to start and promote the galera resource is contained within
> > > /usr/lib/ocf/resource.d/heartbeat/galera and
> > > /usr/lib/ocf/lib/heartbeat/mysql-common.sh. I encourage you to inspect
> > > those for any relevant differences between your own startup method and
> > > that of the resource agent.
> > >
> > > As one example, note that the resource agent uses mysqld_safe by
> > > default. This is configurable via the `binary` option for the
> > > resource. Be sure that you've looked at all the available options
> > > (`pcs resource describe ocf:heartbeat:galera`) and configured any of
> > > them that you need. You've definitely at least started that process
> > > with config and datadir.
> > log snippet I pasted was not to emphasize 'mysqld_safe' but
> > the fact that resource does:
> > ...
> > Starting mysqld daemon with databases from /var/lib/mysql
> > ...
> >
> > Yes, I did check man pages for options and if if you suggest
> > checking source code then I say that's case for a bug report.
>
> For something like pacemaker, I wouldn't point a user to the code.
> galera and mysql-common.sh are shell scripts, so they're easier to
> interpret. The idea was "maybe you can find a discrepancy between how
> the scripts are starting mysql, and how you're manually starting
> mysql."
>
> Filing a bug might be reasonable. I don't have much hands-on
> experience with mysql or with this resource agent.
>
> > I tried my best to make it clear, doubt can do that better
> > but I'll re-try:
> > a) bits needed to run a "new/second" instance are all in
> > 'mariadb-server.cnf' and galera works outside of pacemaker
> > (cmd above)
> > b) if I use attrs: 'datadir' & 'config' yet resource tells
> > me "..daemon with databases from /var/lib/mysql..." then..
> > people looking into the code should perhaps be you (as
> > Redhat employee) and/or other authors/developers - if I
> > filed it as BZ - no?
> >
> > It should be easily reproducible, have:
> > 1) a galera (as a 2nd/3rd/etc. instance, outside & with
> > different settings from "standard" OS's rpm installation)
> > work & tested
> > 2) set up a resource:
> > -> $ pcs resource create mariadb-apps ocf:heartbeat:galera
> > cluster_host_map="drunk.internal.ccn:10.0.1.6;sucker.internal.ccn:10.0.1.7"
> > config="/apps/etc/mariadb-server.cnf"
> > wsrep_cluster_address="gcomm://10.0.1.6:5567,10.0.1.7:5567"
> > user=mysql group=mysql check_user="pacemaker"
> > check_passwd="pacemaker#9897" op monitor OCF_CHECK_LEVEL="0"
> > timeout="30s" interval="20s" op monitor role="Master"
> > OCF_CHECK_LEVEL="0" timeout="30s" interval="10s" op monitor
> > role="Slave" OCF_CHECK_LEVEL="0" timeout="30s"
> > interval="30s" promotable promoted-max=2 meta
> > failure-timeout=30s
>
> The datadir option isn't specified in the resource create command
> above. If datadir not explicitly set, then the resource agent uses
> /var/lib/mysql by default:
> ```
> mysql_common_start()
> {
> local mysql_extra_params="$1"
> local pid
>
> $SU - $OCF_RESKEY_user -s /bin/sh -c \
> "${OCF_RESKEY_binary} --defaults-file=$OCF_RESKEY_config \
> --pid-file=$OCF_RESKEY_pid \
> --socket=$OCF_RESKEY_socket \
> --datadir=$OCF_RESKEY_datadir \
> --log-error=$OCF_RESKEY_log \
> $OCF_RESKEY_additional_parameters \
> $mysql_extra_params >/dev/null 2>&1" &
> pid=$!
> ```
>
> where OCF_RESKEY_datadir is set to /var/lib/mysql if you don't set it yourself.
>
> As mentioned above, I'm not very familiar with mysql, but I would
> expect the CLI --datadir=/var/lib/mysql option to override anything in
> the defaults-file.
To be clear: I know you already mentioned setting
datadir=/apps/mysql/data. Just wondering if that was missed somewhere
since it's not part of the given command.
> >
> > and you should get (give version & evn is the same) see what
> > I see - weird "misbehavior"
> >
> > many thanks, L.
> >
> > _______________________________________________
> > Manage your subscription:
> > https://lists.clusterlabs.org/mailman/listinfo/users
> >
> > ClusterLabs home: https://www.clusterlabs.org/
>
>
>
> --
> Regards,
>
> Reid Wahl (He/Him)
> Senior Software Engineer, Red Hat
> RHEL High Availability - Pacemaker
--
Regards,
Reid Wahl (He/Him)
Senior Software Engineer, Red Hat
RHEL High Availability - Pacemaker
More information about the Users
mailing list