[Pacemaker] DRBD Split Brain after each reboot
andschais at gmail.com
andschais at gmail.com
Tue Dec 22 08:26:44 UTC 2009
Hi Michel,
Yes, I have try with a simpler configuration, I follow this steps:
1) Master/Slave ocf:linbit:drbd RA + ocf:heartbeat:Filesystem RA-> shutdown
-r now -> Ok, no split brain
2) ..+ ocf:heartbeat:IPaddr2 RA -> shutdown -r now -> Ok
3) ..+ heartbeat:drbdlinks RA -> shutdown -r now -> Ok
4) ..+ ocf:heartbeat:pgsql RA -> shutdown -r now -> Ok
5) ..+ ocf:custom:Asterisk RA -> shutdown -r now -> Ok
6) ..+ ocf:heartbeat:apache RA -> shutdown -r now -> Ok
7) ..+ lsb:postfix RA -> shutdown -r now -> Ok
8) ..+ lsb:dhcp3-server -> shutdown -r now -> Ok
9) ..+ lsb:lsb:atftpd -> shutdown -r now -> FAIL, Split brain
At this point I get the first split brain, after a lot of google search I
finally add a start-delay of ten seconds to Start and Promote operations for
drbd RA. After that I reboot a couple times and everything works fine, no
more split brain.
10) ..+ ocf:custom:JBoss RA -> shutdown -r now -> FAIL, Split brain
With this resource enabled I always get a split brain after "normal" reboot.
I tryed to
increase start-delay time on both.start and promote operation to 40 seconds,
that time is more than required to stop JBoss.
If I remove start-delay , I can see in secondary logs:
Dec 21 23:38:02 secondary drbd[17758]: DEBUG: r0: Calling drbdadm -c
/etc/drbd.conf primary r0
Dec 21 23:38:03 secondary lrmd: [19818]: info: RA output:
(drbd:0:promote:stderr) 0: State change failed: (-1) Multiple primaries not
allowed by config
Dec 21 23:38:03 secondary lrmd: [19818]: info: RA output:
(drbd:0:promote:stderr) Command 'drbdsetup 0 primary' terminated with exit
code 11
Dec 21 23:38:03 secondary drbd[17758]: ERROR: r0: Called drbdadm -c
/etc/drbd.conf primary r0
Dec 21 23:38:03 secondary lrmd: [19818]: info: RA output:
(drbd:0:promote:stderr) 2009/12/21_23:38:03 ERROR: r0: Called drbdadm -c
/etc/drbd.conf primary r0
Dec 21 23:38:03 secondary drbd[17758]: ERROR: r0: Exit code 11
Dec 21 23:38:03 secondary lrmd: [19818]: info: RA output:
(drbd:0:promote:stderr) 2009/12/21_23:38:03 ERROR: r0: Exit code 11
Dec 21 23:38:03 secondary drbd[17758]: ERROR: r0: Command output:
Dec 21 23:38:03 secondary lrmd: [19818]: info: RA output:
(drbd:0:promote:stderr) 2009/12/21_23:38:03 ERROR: r0: Command output:
Dec 21 23:38:03 secondary lrmd: [19818]: info: RA output:
(drbd:0:promote:stdout)
Dec 21 23:38:03 secondary drbd[17758]: DEBUG: r0: Calling drbdadm -c
/etc/drbd.conf primary r0
Dec 21 23:38:04 secondary kernel: [215495.004740] tg3: eth1: Link is down.
Dec 21 23:38:04 secondary lrmd: [19818]: info: RA output:
(drbd:0:promote:stderr) 0: State change failed: (-1) Multiple primaries not
allowed by config
Dec 21 23:38:04 secondary lrmd: [19818]: info: RA output:
(drbd:0:promote:stderr) Command 'drbdsetup 0 primary' terminated with exit
code 11
Dec 21 23:38:04 secondary drbd[17758]: ERROR: r0: Called drbdadm -c
/etc/drbd.conf primary r0
Dec 21 23:38:04 secondary lrmd: [19818]: info: RA output:
(drbd:0:promote:stderr) 2009/12/21_23:38:04 ERROR: r0: Called drbdadm -c
/etc/drbd.conf primary r0
Dec 21 23:38:04 secondary drbd[17758]: ERROR: r0: Exit code 11
Dec 21 23:38:04 secondary lrmd: [19818]: info: RA output:
(drbd:0:promote:stderr) 2009/12/21_23:38:04 ERROR: r0: Exit code 11
Dec 21 23:38:04 secondary drbd[17758]: ERROR: r0: Command output:
Dec 21 23:38:04 secondary lrmd: [19818]: info: RA output:
(drbd:0:promote:stderr) 2009/12/21_23:38:04 ERROR: r0: Command output:
Dec 21 23:38:04 secondary lrmd: [19818]: info: RA output:
(drbd:0:promote:stdout)
Dec 21 23:38:05 secondary drbd[17758]: DEBUG: r0: Calling drbdadm -c
/etc/drbd.conf primary r0
Dec 21 23:38:06 secondary kernel: [215496.595684] block drbd0: PingAck did
not arrive in time.
Dec 21 23:38:06 secondary kernel: [215496.602119] tg3: eth1: Link is up at
100 Mbps, full duplex.
Dec 21 23:38:06 secondary kernel: [215496.602122] tg3: eth1: Flow control is
off for TX and off for RX.
Dec 21 23:38:06 secondary kernel: [215496.638538] block drbd0: peer( Primary
-> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown
)
Dec 21 23:38:06 secondary kernel: [215496.638547] block drbd0: asender
terminated
Dec 21 23:38:06 secondary kernel: [215496.638550] block drbd0: Terminating
asender thread
Dec 21 23:38:06 secondary kernel: [215496.638589] block drbd0: short read
expecting header on sock: r=-512
Dec 21 23:38:06 secondary kernel: [215496.697734] block drbd0: Connection
closed
Dec 21 23:38:06 secondary kernel: [215496.697734] block drbd0: conn(
NetworkFailure -> Unconnected )
Dec 21 23:38:06 secondary kernel: [215496.697734] block drbd0: receiver
terminated
Dec 21 23:38:06 secondary kernel: [215496.697734] block drbd0: Restarting
receiver thread
Dec 21 23:38:06 secondary kernel: [215496.697734] block drbd0: receiver
(re)started
Dec 21 23:38:06 secondary kernel: [215496.697734] block drbd0: helper
command: /sbin/drbdadm fence-peer minor-0
Dec 21 23:38:07 secondary crm-fence-peer.sh[17839]: invoked for r0
Means that I'm having communication failure due to network goes down too
fast or that secondary wants to be Master before primary can be slave?...or
both? :-)
Thanks for help!!
Andres
2009/12/21 andschais at gmail.com <andschais at gmail.com>
> Hi all,
>
> I'm getting troubles with a Pacemaker+DRBD 2 nodes cluster. I am trying to
> solve it for about a week, I really need help!!!
> If I disconnect power cord the failover works great, resources migrate to
> secondary node and back to primary when I turn it on.
> But when turn off primary node with a "shutdown -r now" command, I always
> finish with a split brian. That's not all, If a put just a few resources
> (for example: virtual IP, DRBD, Apache and PostgreSQL) split brain does not
> take place, but at the moment I put 8 or 9 resources (specially when one of
> those resources is JBoss AS) I always get split brain...
> Can someone give me some hints?
>
> My systems are:
>
> OS: Debian Lenny 2.6.26-2-686
> Corosync 1.1.2
> DRBD 8.3.6
>
> And my configuration files are:
>
> /etc/corosync/corosync.conf
>
> # Please read the openais.conf.5 manual page
> totem {
> version: 2
> # How long before declaring a token lost (ms)
> token: 3000
> # How many token retransmits before forming a new configuration
> token_retransmits_before_loss_const: 10
> # How long to wait for join messages in the membership protocol
> (ms)
> join: 60
> # How long to wait for consensus to be achieved before starting a
> new round of membership configuration (ms)
> consensus: 1500
> # Turn off the virtual synchrony filter
> vsftype: none
> # Number of messages that may be sent by one processor on receipt
> of the token
> max_messages: 20
> # Limit generated nodeids to 31-bits (positive signed integers)
> clear_node_high_bit: yes
> # Disable encryption
> secauth: on
> # How many threads to use for encryption/decryption
> threads: 0
> # Optionally assign a fixed node id (integer)
> # nodeid: 1234
> # This specifies the mode of redundant ring, which may be none,
> active, or passive.
> rrp_mode: passive
> interface {
> # The following values need to be set based on your
> environment
> ringnumber: 0
> bindnetaddr: 172.16.1.0
> mcastaddr: 226.94.1.1
> mcastport: 5405
> }
> interface {
> # The following values need to be set based on your
> environment
> ringnumber: 1
> bindnetaddr: 10.186.68.0
> mcastaddr: 226.94.2.1
> mcastport: 5405
> }
> }
> amf {
> mode: disabled
> }
> service {
> # Load the Pacemaker Cluster Resource Manager
> ver: 0
> name: pacemaker
> }
> aisexec {
> user: root
> group: root
> }
> logging {
> to_stderr: yes
> debug: on
> timestamp: on
> to_file: yes
> logfile: /var/log/corosync.log
> to_syslog: no
> syslog_facility: daemon
> }
> }
>
>
> /etc/drbd.conf
>
> global {
> usage-count yes;
> }
> common {
> syncer { rate 33M; }
> }
> resource r0 {
> protocol C;
> handlers {
> pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh;
> /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
> reboot -f";
> pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh;
> /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
> reboot -f";
> local-io-error "/usr/lib/drbd/notify-io-error.sh;
> /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ;
> halt -f";
> fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
> after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
> outdate-peer "/usr/lib/drbd/outdate-peer.sh";
> split-brain "/usr/lib/drbd/notify-split-brain.sh root at localhost";
> }
> startup {
> degr-wfc-timeout 30;
> wfc-timeout 30;
> }
> disk {
> fencing resource-only;
> on-io-error detach;
> }
> net {
> after-sb-0pri disconnect;
> after-sb-1pri disconnect;
> after-sb-2pri disconnect;
> rr-conflict disconnect;
> }
>
> on primary {
> device /dev/drbd0;
> disk /dev/vg00/drbd;
> address 172.16.1.1:7788;
> meta-disk internal;
> }
> on secondary {
> device /dev/drbd0;
> disk /dev/vg00/drbd;
> address 172.16.1.2:7788;
> meta-disk internal;
> }
> }
>
>
> and my crm config
>
> <configuration>
> <crm_config>
> <cluster_property_set id="cib-bootstrap-options">
> <nvpair id="cib-bootstrap-options-no-quorum-policy"
> name="no-quorum-policy" value="ignore"/>
> <nvpair id="cib-bootstrap-options-stonith-enabled"
> name="stonith-enabled" value="false"/>
> <nvpair id="cib-bootstrap-options-expected-quorum-votes"
> name="expected-quorum-votes" value="2"/>
> <nvpair id="cib-bootstrap-options-last-lrm-refresh"
> name="last-lrm-refresh" value="1261424411"/>
> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version"
> value="1.0.6-cebe2b6ff49b36b29a3bd7ada1c4701c7470febe"/>
> <nvpair id="cib-bootstrap-options-cluster-infrastructure"
> name="cluster-infrastructure" value="openais"/>
> </cluster_property_set>
> </crm_config>
> <nodes>
> <node uname="primary" type="normal" id="primary">
> <instance_attributes id="nodes-primary">
> <nvpair name="standby" id="nodes-primary-standby" value="off"/>
> </instance_attributes>
> </node>
> <node uname="secondary" type="normal" id="secondary">
> <instance_attributes id="nodes-secondary">
> <nvpair name="standby" id="nodes-secondary-standby" value="off"/>
> </instance_attributes>
> </node>
> </nodes>
> <resources>
> <master id="ms-drbd">
> <meta_attributes id="ms-drbd-meta_attributes">
> <nvpair id="ms-drbd-meta_attributes-master-max" name="master-max"
> value="1"/>
> <nvpair id="ms-drbd-meta_attributes-master-node-max"
> name="master-node-max" value="1"/>
> <nvpair id="ms-drbd-meta_attributes-clone-max" name="clone-max"
> value="2"/>
> <nvpair id="ms-drbd-meta_attributes-clone-node-max"
> name="clone-node-max" value="1"/>
> <nvpair id="ms-drbd-meta_attributes-notify" name="notify"
> value="true"/>
> <nvpair id="ms-drbd-meta_attributes-globally-unique"
> name="globally-unique" value="false"/>
> <nvpair name="target-role"
> id="ms-drbd-meta_attributes-target-role" value="Started"/>
> </meta_attributes>
> <primitive class="ocf" id="drbd" provider="linbit" type="drbd">
> <instance_attributes id="drbd-instance_attributes">
> <nvpair id="drbd-instance_attributes-drbd_resource"
> name="drbd_resource" value="r0"/>
> </instance_attributes>
> <operations>
> <op id="drbd-monitor-59s" interval="59s" name="monitor"
> role="Master" timeout="30s"/>
> <op id="drbd-monitor-60s" interval="60s" name="monitor"
> role="Slave" timeout="30s"/>
> <op id="drbd-start-0s" interval="0s" name="start"
> start-delay="10s"/>
> <op id="drbd-promote-0s" interval="0s" name="promote"
> start-delay="10s"/>
> </operations>
> </primitive>
> </master>
> <group id="p-group">
> <primitive class="ocf" id="fs" provider="heartbeat"
> type="Filesystem">
> <instance_attributes id="fs-instance_attributes">
> <nvpair id="fs-instance_attributes-fstype" name="fstype"
> value="ext3"/>
> <nvpair id="fs-instance_attributes-directory" name="directory"
> value="/drbd"/>
> <nvpair id="fs-instance_attributes-device" name="device"
> value="/dev/drbd0"/>
> </instance_attributes>
> <meta_attributes id="fs-meta_attributes">
> <nvpair id="fs-meta_attributes-is-managed" name="is-managed"
> value="true"/>
> </meta_attributes>
> </primitive>
> <primitive class="ocf" id="ip" provider="heartbeat" type="IPaddr2">
> <instance_attributes id="ip-instance_attributes">
> <nvpair id="ip-instance_attributes-ip" name="ip"
> value="10.186.68.1"/>
> <nvpair id="ip-instance_attributes-broadcast" name="broadcast"
> value="10.186.68.127"/>
> <nvpair id="ip-instance_attributes-cidr_netmask"
> name="cidr_netmask" value="25"/>
> </instance_attributes>
> <operations>
> <op id="ip-monitor-10s" interval="10s" name="monitor"/>
> </operations>
> </primitive>
> <primitive class="heartbeat" id="drbdlinks" type="drbdlinks">
> <operations>
> <op id="drbdlinks-monitor-60s" interval="60s" name="monitor"/>
> </operations>
> </primitive>
> <primitive class="ocf" id="postgresql" provider="heartbeat"
> type="pgsql">
> <instance_attributes id="postgresql-instance_attributes">
> <nvpair id="postgresql-instance_attributes-pgctl" name="pgctl"
> value="/usr/lib/postgresql/8.3/bin/pg_ctl"/>
> <nvpair id="postgresql-instance_attributes-psql" name="psql"
> value="/usr/bin/psql"/>
> <nvpair id="postgresql-instance_attributes-pgdata"
> name="pgdata" value="/var/lib/postgresql/8.3/main"/>
> <nvpair id="postgresql-instance_attributes-pgdba" name="pgdba"
> value="postgres"/>
> <nvpair id="postgresql-instance_attributes-pgdb" name="pgdb"
> value="postgres"/>
> <nvpair id="postgresql-instance_attributes-logfile"
> name="logfile" value="/var/log/postgresql/postgresql-8.3-main.log"/>
> </instance_attributes>
> <operations>
> <op id="postgresql-monitor-60s" interval="60s" name="monitor"
> timeout="30s"/>
> </operations>
> </primitive>
> <primitive class="ocf" id="asterisk" provider="custom"
> type="Asterisk">
> <operations>
> <op id="asterisk-monitor-60s" interval="60s" name="monitor"
> start-delay="30s" timeout="30s"/>
> </operations>
> </primitive>
> <primitive class="lsb" id="postfix" type="postfix"/>
> <primitive class="ocf" id="apache2" provider="heartbeat"
> type="apache">
> <instance_attributes id="apache2-instance_attributes">
> <nvpair id="apache2-instance_attributes-configfile"
> name="configfile" value="/etc/apache2/apache2.conf"/>
> </instance_attributes>
> <operations>
> <op id="apache2-monitor-60s" interval="60s" name="monitor"/>
> </operations>
> </primitive>
> <primitive class="lsb" id="dhcp" type="dhcp3-server"/>
> <primitive class="ocf" id="jboss" provider="custom" type="JBoss">
> <instance_attributes id="jboss-instance_attributes">
> <nvpair id="jboss-instance_attributes-java_home"
> name="java_home" value="/opt/java/"/>
> <nvpair id="jboss-instance_attributes-jboss_home"
> name="jboss_home" value="/opt/jboss"/>
> </instance_attributes>
> <operations>
> <op id="jboss-monitor-60s" interval="60s" name="monitor"
> start-delay="100s" timeout="30s"/>
> <op id="jboss-start-0s" interval="0s" name="start"
> timeout="99s"/>
> </operations>
> </primitive>
> </group>
> </resources>
> <constraints>
> <rsc_colocation id="p-group-on-ms-drbd" rsc="p-group"
> score="INFINITY" with-rsc="ms-drbd" with-rsc-role="Master"/>
> <rsc_location id="ms-drbd-master-on-primary" rsc="ms-drbd">
> <rule id="ms-drbd-master-on-primary-rule" role="Master"
> score="100">
> <expression attribute="#uname"
> id="ms-drbd-master-on-primary-expression" operation="eq" value="primary"/>
> </rule>
> </rsc_location>
> <rsc_order first="ms-drbd" first-action="promote"
> id="ms-drbd-before-group" score="INFINITY" then="p-group"
> then-action="start"/>
> </constraints>
> <rsc_defaults/>
> <op_defaults/>
> </configuration>
>
> Thanks in advance.
> Andres.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20091222/a546cc80/attachment-0002.htm>
More information about the Pacemaker
mailing list