[Pacemaker] Debian + Pacemaker 1.0.9 + DRBD 8.3.7 with dual primary
Lutz Reinhardt
l.reinhardt at gmx.net
Fri Dec 2 02:55:53 EST 2011
Am 02.12.2011 00:32, schrieb Andreas Kurz:
> Hello Lutz,
>
> On 12/01/2011 01:26 PM, Lutz Reinhardt wrote:
>> hi
>>
>> use a simple config:
>>
>> node node1
>> node node2 \
>> attributes standby="off"
>> primitive res_drbd_cluster_ocfs ocf:linbit:drbd \
>> params drbd_resource="cluster-ocfs" \
>> op monitor interval="20" role="Master" timeout="20" \
>> op monitor interval="30" role="Slave" timeout="20"
>> ms ms_drbd_cluster_ocfs res_drbd_cluster_ocfs \
>> meta resource-stickiness="100" master-max="2" notify="true"
>> interleave="true" target-role="Started"
>> property $id="cib-bootstrap-options" \
>> dc-version="1.0.9-74392a28b7f31d7ddc86689598bd23114f58978b" \
>> cluster-infrastructure="openais" \
>> expected-quorum-votes="2" \
>> no-quorum-policy="ignore" \
>> stonith-enabled="false" \
>> default-resource-stickiness="1000" \
>> last-lrm-refresh="1322660182"
>>
>> now it works fine but one thing happens: when i stop
>> "ms_drbd_cluster_ocfs" and start it again, i see every time a fresh
>> resync of one drbd node. it's not happened when i bring up the drbd
>> device first before i start it with "start ms_drbd_cluster_ocfs" - then
>> it works fine!?
>>
>> whats wrong?
>
> First of all you must use fencing for a dual-primary setup ... as you
> have no fencing configured in Pacemaker I conclude (please correct me if
> I'm wrong) that you also skip this part in DRBD config.
>
> Then please share the result of "drbdadm dump all" and "cat /proc/drbd"
> when your cluster is up. You said you see a "fresh" resync ... you mean
> a full resync? Please also provide kernel logs for this.
yes i mean a full resync and i use fencing (see below):
# drbdadm dump all
global {
minor-count 32;
}
common {
protocol C;
net {
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
}
disk {
on-io-error pass_on;
fencing resource-only;
}
syncer {
rate 12800;
}
startup {
become-primary-on both;
}
handlers {
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh;
/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh;
/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh;
/usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger
; halt -f";
fence-peer "/usr/lib/drbd/crm-fence-peer.sh --timeout 30
--dc-timeout 40";
after-resync-target /usr/lib/drbd/crm-unfence-peer.sh;
}
}
resource cluster-ocfs {
protocol C;
floating ipv4 10.60.60.22:7802 {
device minor 0;
disk /dev/xencluster/cluster-ocfs;
meta-disk internal;
}
floating ipv4 10.60.60.23:7802 {
device minor 0;
disk /dev/xencluster/cluster-ocfs;
meta-disk internal;
}
}
resource test-drbd {
protocol C;
floating ipv4 10.60.60.22:7804 {
device minor 31;
disk /dev/xencluster/test-drbd;
meta-disk internal;
}
floating ipv4 10.60.60.23:7804 {
device minor 31;
disk /dev/xencluster/test-drbd;
meta-disk internal;
}
}
# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757
0: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r----
ns:0 nr:0 dw:0 dr:200 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
31: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r----
ns:0 nr:0 dw:0 dr:152 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
stop ms_drbd_cluster_ocfs:
node1:
Dec 2 08:46:55 node1 kernel: [143347.351875] block drbd0: role( Primary
-> Secondary )
Dec 2 08:46:55 node1 kernel: [143347.363566] block drbd0: peer( Primary
-> Secondary )
Dec 2 08:46:55 node1 kernel: [143347.526187] block drbd0: sock was shut
down by peer
Dec 2 08:46:55 node1 kernel: [143347.526192] block drbd0: peer(
Secondary -> Unknown ) conn( Connected -> Disconnecting ) pdsk( UpToDate
-> DUnknown )
Dec 2 08:46:55 node1 kernel: [143347.526247] block drbd0: asender
terminated
Dec 2 08:46:55 node1 kernel: [143347.526250] block drbd0: Terminating
drbd0_asender
Dec 2 08:46:55 node1 kernel: [143347.526376] block drbd0: Connection closed
Dec 2 08:46:55 node1 kernel: [143347.526384] block drbd0: conn(
Disconnecting -> StandAlone )
Dec 2 08:46:55 node1 kernel: [143347.526403] block drbd0: receiver
terminated
Dec 2 08:46:55 node1 kernel: [143347.526406] block drbd0: Terminating
drbd0_receiver
Dec 2 08:46:55 node1 kernel: [143347.526426] block drbd0: disk(
UpToDate -> Diskless )
Dec 2 08:46:55 node1 kernel: [143347.526476] block drbd0:
drbd_bm_resize called with capacity == 0
Dec 2 08:46:55 node1 kernel: [143347.526482] block drbd0: worker terminated
Dec 2 08:46:55 node1 kernel: [143347.526486] block drbd0: Terminating
drbd0_worker
node2:
Dec 2 08:46:55 node2 kernel: [143573.833171] block drbd0: peer( Primary
-> Secondary ) pdsk( Inconsistent -> UpToDate )
Dec 2 08:46:55 node2 kernel: [143573.843876] block drbd0: role( Primary
-> Secondary )
Dec 2 08:46:55 node2 kernel: [143574.007033] block drbd0: peer(
Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate ->
DUnknown )
Dec 2 08:46:55 node2 kernel: [143574.007077] block drbd0: asender
terminated
Dec 2 08:46:55 node2 kernel: [143574.007082] block drbd0: Terminating
drbd0_asender
Dec 2 08:46:55 node2 kernel: [143574.007285] block drbd0: Connection closed
Dec 2 08:46:55 node2 kernel: [143574.007290] block drbd0: conn(
TearDown -> Unconnected )
Dec 2 08:46:55 node2 kernel: [143574.007296] block drbd0: receiver
terminated
Dec 2 08:46:55 node2 kernel: [143574.007298] block drbd0: Restarting
drbd0_receiver
Dec 2 08:46:55 node2 kernel: [143574.007300] block drbd0: receiver
(re)started
Dec 2 08:46:55 node2 kernel: [143574.007304] block drbd0: conn(
Unconnected -> WFConnection )
Dec 2 08:46:55 node2 kernel: [143574.018028] block drbd0: conn(
WFConnection -> Disconnecting )
Dec 2 08:46:55 node2 kernel: [143574.018043] block drbd0: Discarding
network configuration.
Dec 2 08:46:55 node2 kernel: [143574.018163] block drbd0: Connection closed
Dec 2 08:46:55 node2 kernel: [143574.018180] block drbd0: conn(
Disconnecting -> StandAlone )
Dec 2 08:46:55 node2 kernel: [143574.018302] block drbd0: receiver
terminated
Dec 2 08:46:55 node2 kernel: [143574.018306] block drbd0: Terminating
drbd0_receiver
Dec 2 08:46:55 node2 kernel: [143574.018392] block drbd0: disk(
UpToDate -> Diskless )
Dec 2 08:46:55 node2 kernel: [143574.018501] block drbd0:
drbd_bm_resize called with capacity == 0
Dec 2 08:46:55 node2 kernel: [143574.018508] block drbd0: worker terminated
Dec 2 08:46:55 node2 kernel: [143574.018510] block drbd0: Terminating
drbd0_worker
start ms_drbd_cluster_ocfs:
node1:
Dec 2 08:47:14 node1 kernel: [143365.606992] block drbd0: Starting
worker thread (from cqueue [1898])
Dec 2 08:47:14 node1 kernel: [143365.607353] block drbd0: disk(
Diskless -> Attaching )
Dec 2 08:47:14 node1 kernel: [143365.625044] block drbd0: No usable
activity log found.
Dec 2 08:47:14 node1 kernel: [143365.625050] block drbd0: Method to
ensure write ordering: barrier
Dec 2 08:47:14 node1 kernel: [143365.625056] block drbd0: Backing
device's merge_bvec_fn() = ffffffffa00f2c89
Dec 2 08:47:14 node1 kernel: [143365.625059] block drbd0:
max_segment_size ( = BIO size ) = 4096
Dec 2 08:47:14 node1 kernel: [143365.625064] block drbd0:
drbd_bm_resize called with capacity == 2097016
Dec 2 08:47:14 node1 kernel: [143365.625078] block drbd0: resync
bitmap: bits=262127 words=4096
Dec 2 08:47:14 node1 kernel: [143365.625082] block drbd0: size = 1024
MB (1048508 KB)
Dec 2 08:47:14 node1 kernel: [143365.628586] block drbd0: recounting of
set bits took additional 0 jiffies
Dec 2 08:47:14 node1 kernel: [143365.628590] block drbd0: 0 KB (0 bits)
marked out-of-sync by on disk bit-map.
Dec 2 08:47:14 node1 kernel: [143365.628596] block drbd0: disk(
Attaching -> Consistent )
Dec 2 08:47:14 node1 kernel: [143365.633831] block drbd0: conn(
StandAlone -> Unconnected )
Dec 2 08:47:14 node1 kernel: [143365.633854] block drbd0: Starting
receiver thread (from drbd0_worker [14778])
Dec 2 08:47:14 node1 kernel: [143365.634352] block drbd0: receiver
(re)started
Dec 2 08:47:14 node1 kernel: [143365.634359] block drbd0: conn(
Unconnected -> WFConnection )
Dec 2 08:47:14 node1 kernel: [143365.850121] block drbd0: helper
command: /sbin/drbdadm fence-peer minor-0
Dec 2 08:47:14 node1 kernel: [143365.931181] block drbd0: Handshake
successful: Agreed network protocol version 91
Dec 2 08:47:15 node1 crm-fence-peer.sh[14894]: invoked for cluster-ocfs
Dec 2 08:47:45 node1 crm-fence-peer.sh[14894]: Call cib_create failed
(-21): The object already exists
Dec 2 08:47:45 node1 crm-fence-peer.sh[14894]: <failed>
Dec 2 08:47:45 node1 crm-fence-peer.sh[14894]: <failed_update
id="drbd-fence-by-handler-cluster-ocfs-ms_drbd_cluster_ocfs"
object_type="rsc_location" operation="cib_create" reason="The object
already exists">
Dec 2 08:47:45 node1 crm-fence-peer.sh[14894]: <rsc_location
rsc="ms_drbd_cluster_ocfs"
id="drbd-fence-by-handler-cluster-ocfs-ms_drbd_cluster_ocfs">
Dec 2 08:47:45 node1 crm-fence-peer.sh[14894]: <rule
role="Master" score="-INFINITY"
id="drbd-fence-by-handler-cluster-ocfs-rule-ms_drbd_cluster_ocfs">
Dec 2 08:47:45 node1 crm-fence-peer.sh[14894]: <expression
attribute="#uname" operation="ne" value="node1"
id="drbd-fence-by-handler-cluster-ocfs-expr-ms_drbd_cluster_ocfs"/>
Dec 2 08:47:45 node1 crm-fence-peer.sh[14894]: </rule>
Dec 2 08:47:45 node1 crm-fence-peer.sh[14894]: </rsc_location>
Dec 2 08:47:45 node1 crm-fence-peer.sh[14894]: </failed_update>
Dec 2 08:47:45 node1 crm-fence-peer.sh[14894]: </failed>
Dec 2 08:47:45 node1 crm-fence-peer.sh[14894]: WARNING DATA INTEGRITY
at RISK: could not place the fencing constraint!
Dec 2 08:47:45 node1 kernel: [143397.494413] block drbd0: helper
command: /sbin/drbdadm fence-peer minor-0 exit code 1 (0x100)
Dec 2 08:47:45 node1 kernel: [143397.494676] block drbd0: conn(
WFConnection -> WFReportParams )
Dec 2 08:47:45 node1 kernel: [143397.494841] block drbd0: Starting
asender thread (from drbd0_receiver [14787])
Dec 2 08:47:45 node1 kernel: [143397.494951] block drbd0:
data-integrity-alg: <not-used>
Dec 2 08:47:45 node1 kernel: [143397.494964] block drbd0:
drbd_sync_handshake:
Dec 2 08:47:45 node1 kernel: [143397.494967] block drbd0: self
C25D6F4D22879C5C:0000000000000000:BBD6AD66F9958D13:FE0A927040B5EE53
bits:0 flags:0
Dec 2 08:47:45 node1 kernel: [143397.494971] block drbd0: peer
D97B1C1FD6F6A891:C25D6F4D22879C5C:BBD6AD66F9958D13:FE0A927040B5EE53
bits:0 flags:0
Dec 2 08:47:45 node1 kernel: [143397.494977] block drbd0: conn(
WFReportParams -> NetworkFailure )
Dec 2 08:47:45 node1 kernel: [143397.494980] block drbd0:
uuid_compare()=-1 by rule 50
Dec 2 08:47:45 node1 kernel: [143397.494989] block drbd0: asender
terminated
Dec 2 08:47:45 node1 kernel: [143397.494990] block drbd0:
drbd_sync_handshake:
Dec 2 08:47:45 node1 kernel: [143397.494994] block drbd0: self
C25D6F4D22879C5C:0000000000000000:BBD6AD66F9958D13:FE0A927040B5EE53
bits:0 flags:0
Dec 2 08:47:45 node1 kernel: [143397.494998] block drbd0: peer
D97B1C1FD6F6A891:C25D6F4D22879C5C:BBD6AD66F9958D13:FE0A927040B5EE53
bits:0 flags:0
Dec 2 08:47:45 node1 kernel: [143397.495001] block drbd0:
uuid_compare()=-1 by rule 50
Dec 2 08:47:45 node1 kernel: [143397.495004] block drbd0: Terminating
drbd0_asender
Dec 2 08:47:45 node1 kernel: [143397.495029] block drbd0: Connection closed
Dec 2 08:47:45 node1 kernel: [143397.495066] block drbd0: conn(
NetworkFailure -> Unconnected )
Dec 2 08:47:45 node1 kernel: [143397.495071] block drbd0: receiver
terminated
Dec 2 08:47:45 node1 kernel: [143397.495073] block drbd0: Restarting
drbd0_receiver
Dec 2 08:47:45 node1 kernel: [143397.495076] block drbd0: receiver
(re)started
Dec 2 08:47:45 node1 kernel: [143397.495082] block drbd0: conn(
Unconnected -> WFConnection )
Dec 2 08:47:46 node1 kernel: [143397.512326] block drbd0: disk(
Consistent -> Outdated )
Dec 2 08:47:46 node1 kernel: [143397.598232] block drbd0: Handshake
successful: Agreed network protocol version 91
Dec 2 08:47:46 node1 kernel: [143397.598241] block drbd0: conn(
WFConnection -> WFReportParams )
Dec 2 08:47:46 node1 kernel: [143397.598264] block drbd0: Starting
asender thread (from drbd0_receiver [14787])
Dec 2 08:47:46 node1 kernel: [143397.598415] block drbd0:
data-integrity-alg: <not-used>
Dec 2 08:47:46 node1 kernel: [143397.598428] block drbd0:
drbd_sync_handshake:
Dec 2 08:47:46 node1 kernel: [143397.598432] block drbd0: self
C25D6F4D22879C5C:0000000000000000:BBD6AD66F9958D13:FE0A927040B5EE53
bits:0 flags:0
Dec 2 08:47:46 node1 kernel: [143397.598437] block drbd0: peer
D97B1C1FD6F6A891:C25D6F4D22879C5C:BBD6AD66F9958D13:FE0A927040B5EE53
bits:0 flags:0
Dec 2 08:47:46 node1 kernel: [143397.598440] block drbd0:
uuid_compare()=-1 by rule 50
Dec 2 08:47:46 node1 kernel: [143397.598446] block drbd0: peer( Unknown
-> Primary ) conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown ->
UpToDate )
Dec 2 08:47:46 node1 kernel: [143397.599801] block drbd0: conn(
WFBitMapT -> WFSyncUUID )
Dec 2 08:47:46 node1 kernel: [143397.601341] block drbd0: helper
command: /sbin/drbdadm before-resync-target minor-0
Dec 2 08:47:46 node1 kernel: [143397.603220] block drbd0: helper
command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)
Dec 2 08:47:46 node1 kernel: [143397.603227] block drbd0: conn(
WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )
Dec 2 08:47:46 node1 kernel: [143397.603235] block drbd0: Began resync
as SyncTarget (will sync 0 KB [0 bits set]).
Dec 2 08:47:46 node1 kernel: [143398.149623] block drbd0: peer( Primary
-> Unknown ) conn( SyncTarget -> Disconnecting ) pdsk( UpToDate ->
DUnknown )
Dec 2 08:47:46 node1 kernel: [143398.149814] block drbd0: asender
terminated
Dec 2 08:47:46 node1 kernel: [143398.149822] block drbd0: Terminating
drbd0_asender
Dec 2 08:47:46 node1 kernel: [143398.150000] block drbd0: Connection closed
Dec 2 08:47:46 node1 kernel: [143398.150033] block drbd0: conn(
Disconnecting -> StandAlone )
Dec 2 08:47:46 node1 kernel: [143398.150279] block drbd0: receiver
terminated
Dec 2 08:47:46 node1 kernel: [143398.150284] block drbd0: Terminating
drbd0_receiver
Dec 2 08:47:46 node1 kernel: [143398.150345] block drbd0: disk(
Inconsistent -> Diskless )
Dec 2 08:47:46 node1 kernel: [143398.150451] block drbd0:
drbd_bm_resize called with capacity == 0
Dec 2 08:47:46 node1 kernel: [143398.150459] block drbd0: worker terminated
Dec 2 08:47:46 node1 kernel: [143398.150462] block drbd0: Terminating
drbd0_worker
Dec 2 08:47:46 node1 kernel: [143398.257848] block drbd0: Starting
worker thread (from cqueue [1898])
Dec 2 08:47:46 node1 kernel: [143398.258149] block drbd0: disk(
Diskless -> Attaching )
Dec 2 08:47:46 node1 kernel: [143398.266813] block drbd0: No usable
activity log found.
Dec 2 08:47:46 node1 kernel: [143398.266819] block drbd0: Method to
ensure write ordering: barrier
Dec 2 08:47:46 node1 kernel: [143398.266825] block drbd0: Backing
device's merge_bvec_fn() = ffffffffa00f2c89
Dec 2 08:47:46 node1 kernel: [143398.266828] block drbd0:
max_segment_size ( = BIO size ) = 4096
Dec 2 08:47:46 node1 kernel: [143398.266834] block drbd0:
drbd_bm_resize called with capacity == 2097016
Dec 2 08:47:46 node1 kernel: [143398.266847] block drbd0: resync
bitmap: bits=262127 words=4096
Dec 2 08:47:46 node1 kernel: [143398.266852] block drbd0: size = 1024
MB (1048508 KB)
Dec 2 08:47:46 node1 kernel: [143398.267799] block drbd0: recounting of
set bits took additional 0 jiffies
Dec 2 08:47:46 node1 kernel: [143398.267803] block drbd0: 0 KB (0 bits)
marked out-of-sync by on disk bit-map.
Dec 2 08:47:46 node1 kernel: [143398.267809] block drbd0: disk(
Attaching -> Inconsistent )
Dec 2 08:47:46 node1 kernel: [143398.272213] block drbd0: conn(
StandAlone -> Unconnected )
Dec 2 08:47:46 node1 kernel: [143398.272230] block drbd0: Starting
receiver thread (from drbd0_worker [15524])
Dec 2 08:47:46 node1 kernel: [143398.272260] block drbd0: receiver
(re)started
Dec 2 08:47:46 node1 kernel: [143398.272265] block drbd0: conn(
Unconnected -> WFConnection )
Dec 2 08:47:47 node1 kernel: [143398.572843] block drbd0: Handshake
successful: Agreed network protocol version 91
Dec 2 08:47:47 node1 kernel: [143398.572853] block drbd0: conn(
WFConnection -> WFReportParams )
Dec 2 08:47:47 node1 kernel: [143398.573065] block drbd0: Starting
asender thread (from drbd0_receiver [15535])
Dec 2 08:47:48 node1 kernel: [143399.548019] block drbd0:
data-integrity-alg: <not-used>
Dec 2 08:47:48 node1 kernel: [143399.548191] block drbd0:
drbd_sync_handshake:
Dec 2 08:47:48 node1 kernel: [143399.548197] block drbd0: self
ADA4AB4A18C972CC:0000000000000000:BBD6AD66F9958D13:FE0A927040B5EE53
bits:0 flags:0
Dec 2 08:47:48 node1 kernel: [143399.548204] block drbd0: peer
DF81B74C806482F7:D97B1C1FD6F6A890:ADA4AB4A18C972CD:C25D6F4D22879C5C
bits:0 flags:0
Dec 2 08:47:48 node1 kernel: [143399.548209] block drbd0:
uuid_compare()=-2 by rule 60
Dec 2 08:47:48 node1 kernel: [143399.548212] block drbd0: Becoming sync
target due to disk states.
Dec 2 08:47:48 node1 kernel: [143399.548215] block drbd0: Writing the
whole bitmap, full sync required after drbd_sync_handshake.
Dec 2 08:47:48 node1 kernel: [143399.548986] block drbd0: 1024 MB
(262127 bits) marked out-of-sync by on disk bit-map.
Dec 2 08:47:48 node1 kernel: [143399.549157] block drbd0: peer( Unknown
-> Primary ) conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown ->
UpToDate )
Dec 2 08:47:48 node1 kernel: [143399.551323] block drbd0: conn(
WFBitMapT -> WFSyncUUID )
Dec 2 08:47:48 node1 kernel: [143399.552817] block drbd0: helper
command: /sbin/drbdadm before-resync-target minor-0
Dec 2 08:47:48 node1 kernel: [143399.558452] block drbd0: helper
command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)
Dec 2 08:47:48 node1 kernel: [143399.558459] block drbd0: conn(
WFSyncUUID -> SyncTarget )
Dec 2 08:47:48 node1 kernel: [143399.558466] block drbd0: Began resync
as SyncTarget (will sync 1048508 KB [262127 bits set]).
node2:
Dec 2 08:47:14 node2 kernel: [143592.106933] block drbd0: Starting
worker thread (from cqueue [1840])
Dec 2 08:47:14 node2 kernel: [143592.107136] block drbd0: disk(
Diskless -> Attaching )
Dec 2 08:47:14 node2 kernel: [143592.117429] block drbd0: No usable
activity log found.
Dec 2 08:47:14 node2 kernel: [143592.117434] block drbd0: Method to
ensure write ordering: barrier
Dec 2 08:47:14 node2 kernel: [143592.117440] block drbd0: Backing
device's merge_bvec_fn() = ffffffffa010fc89
Dec 2 08:47:14 node2 kernel: [143592.117444] block drbd0:
max_segment_size ( = BIO size ) = 4096
Dec 2 08:47:14 node2 kernel: [143592.117449] block drbd0:
drbd_bm_resize called with capacity == 2097016
Dec 2 08:47:14 node2 kernel: [143592.117464] block drbd0: resync
bitmap: bits=262127 words=4096
Dec 2 08:47:14 node2 kernel: [143592.117472] block drbd0: size = 1024
MB (1048508 KB)
Dec 2 08:47:14 node2 kernel: [143592.123062] block drbd0: recounting of
set bits took additional 0 jiffies
Dec 2 08:47:14 node2 kernel: [143592.123067] block drbd0: 0 KB (0 bits)
marked out-of-sync by on disk bit-map.
Dec 2 08:47:14 node2 kernel: [143592.123074] block drbd0: disk(
Attaching -> Consistent )
Dec 2 08:47:14 node2 kernel: [143592.127792] block drbd0: conn(
StandAlone -> Unconnected )
Dec 2 08:47:14 node2 kernel: [143592.127810] block drbd0: Starting
receiver thread (from drbd0_worker [15110])
Dec 2 08:47:14 node2 kernel: [143592.127846] block drbd0: receiver
(re)started
Dec 2 08:47:14 node2 kernel: [143592.127851] block drbd0: conn(
Unconnected -> WFConnection )
Dec 2 08:47:14 node2 kernel: [143592.361076] block drbd0: helper
command: /sbin/drbdadm fence-peer minor-0
Dec 2 08:47:14 node2 kernel: [143592.440790] block drbd0: Handshake
successful: Agreed network protocol version 91
Dec 2 08:47:15 node2 crm-fence-peer.sh[15223]: invoked for cluster-ocfs
Dec 2 08:47:45 node2 crm-fence-peer.sh[15223]: INFO peer is reachable,
my disk is Consistent: placed constraint
'drbd-fence-by-handler-cluster-ocfs-ms_drbd_cluster_ocfs'
Dec 2 08:47:45 node2 kernel: [143623.333511] block drbd0: helper
command: /sbin/drbdadm fence-peer minor-0 exit code 4 (0x400)
Dec 2 08:47:45 node2 kernel: [143623.333516] block drbd0: fence-peer
helper returned 4 (peer was fenced)
Dec 2 08:47:45 node2 kernel: [143623.333524] block drbd0: role(
Secondary -> Primary ) disk( Consistent -> UpToDate ) pdsk( DUnknown ->
Outdated )
Dec 2 08:47:45 node2 kernel: [143623.333830] block drbd0: Creating new
current UUID
Dec 2 08:47:45 node2 kernel: [143623.334207] block drbd0: conn(
WFConnection -> WFReportParams )
Dec 2 08:47:45 node2 kernel: [143623.334226] block drbd0: Starting
asender thread (from drbd0_receiver [15121])
Dec 2 08:47:45 node2 kernel: [143623.833678] block drbd0: conn(
WFReportParams -> NetworkFailure )
Dec 2 08:47:45 node2 kernel: [143623.833687] block drbd0: asender
terminated
Dec 2 08:47:45 node2 kernel: [143623.833691] block drbd0: Terminating
drbd0_asender
Dec 2 08:47:45 node2 kernel: [143623.834043] block drbd0: Connection closed
Dec 2 08:47:45 node2 kernel: [143623.834059] block drbd0: conn(
NetworkFailure -> Unconnected )
Dec 2 08:47:45 node2 kernel: [143623.834065] block drbd0: receiver
terminated
Dec 2 08:47:45 node2 kernel: [143623.834067] block drbd0: Restarting
drbd0_receiver
Dec 2 08:47:45 node2 kernel: [143623.834070] block drbd0: receiver
(re)started
Dec 2 08:47:45 node2 kernel: [143623.834075] block drbd0: conn(
Unconnected -> WFConnection )
Dec 2 08:47:46 node2 kernel: [143624.157421] block drbd0: Handshake
successful: Agreed network protocol version 91
Dec 2 08:47:46 node2 kernel: [143624.157429] block drbd0: conn(
WFConnection -> WFReportParams )
Dec 2 08:47:46 node2 kernel: [143624.157585] block drbd0: Starting
asender thread (from drbd0_receiver [15121])
Dec 2 08:47:46 node2 kernel: [143624.157728] block drbd0:
data-integrity-alg: <not-used>
Dec 2 08:47:46 node2 kernel: [143624.157743] block drbd0:
drbd_sync_handshake:
Dec 2 08:47:46 node2 kernel: [143624.157747] block drbd0: self
D97B1C1FD6F6A891:C25D6F4D22879C5C:BBD6AD66F9958D13:FE0A927040B5EE53
bits:0 flags:0
Dec 2 08:47:46 node2 kernel: [143624.157751] block drbd0: peer
C25D6F4D22879C5C:0000000000000000:BBD6AD66F9958D13:FE0A927040B5EE53
bits:0 flags:0
Dec 2 08:47:46 node2 kernel: [143624.157755] block drbd0:
uuid_compare()=1 by rule 70
Dec 2 08:47:46 node2 kernel: [143624.157759] block drbd0: peer( Unknown
-> Secondary ) conn( WFReportParams -> WFBitMapS )
Dec 2 08:47:46 node2 kernel: [143624.160572] block drbd0: conn(
WFBitMapS -> SyncSource ) pdsk( Outdated -> Inconsistent )
Dec 2 08:47:46 node2 kernel: [143624.160580] block drbd0: Began resync
as SyncSource (will sync 0 KB [0 bits set]).
Dec 2 08:47:46 node2 kernel: [143624.249923] block drbd0: role( Primary
-> Secondary )
Dec 2 08:47:46 node2 kernel: [143624.709497] block drbd0: Resync done
(total 1 sec; paused 0 sec; 0 K/sec)
Dec 2 08:47:46 node2 kernel: [143624.709507] block drbd0: conn(
SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )
Dec 2 08:47:46 node2 kernel: [143624.709530] block drbd0: peer(
Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate ->
DUnknown )
Dec 2 08:47:46 node2 kernel: [143624.709633] block drbd0: conn(
TearDown -> Disconnecting )
Dec 2 08:47:46 node2 kernel: [143624.709920] block drbd0: asender
terminated
Dec 2 08:47:46 node2 kernel: [143624.709928] block drbd0: Terminating
drbd0_asender
Dec 2 08:47:46 node2 kernel: [143624.710210] block drbd0: Connection closed
Dec 2 08:47:46 node2 kernel: [143624.710222] block drbd0: conn(
Disconnecting -> StandAlone )
Dec 2 08:47:46 node2 kernel: [143624.710405] block drbd0: receiver
terminated
Dec 2 08:47:46 node2 kernel: [143624.710409] block drbd0: Terminating
drbd0_receiver
Dec 2 08:47:46 node2 kernel: [143624.710469] block drbd0: disk(
UpToDate -> Diskless )
Dec 2 08:47:46 node2 kernel: [143624.710645] block drbd0:
drbd_bm_resize called with capacity == 0
Dec 2 08:47:46 node2 kernel: [143624.710653] block drbd0: worker terminated
Dec 2 08:47:46 node2 kernel: [143624.710657] block drbd0: Terminating
drbd0_worker
Dec 2 08:47:46 node2 kernel: [143624.819951] block drbd0: Starting
worker thread (from cqueue [1840])
Dec 2 08:47:46 node2 kernel: [143624.820182] block drbd0: disk(
Diskless -> Attaching )
Dec 2 08:47:46 node2 kernel: [143624.821156] block drbd0: No usable
activity log found.
Dec 2 08:47:46 node2 kernel: [143624.821162] block drbd0: Method to
ensure write ordering: barrier
Dec 2 08:47:46 node2 kernel: [143624.821168] block drbd0: Backing
device's merge_bvec_fn() = ffffffffa010fc89
Dec 2 08:47:46 node2 kernel: [143624.821171] block drbd0:
max_segment_size ( = BIO size ) = 4096
Dec 2 08:47:46 node2 kernel: [143624.821176] block drbd0:
drbd_bm_resize called with capacity == 2097016
Dec 2 08:47:46 node2 kernel: [143624.821191] block drbd0: resync
bitmap: bits=262127 words=4096
Dec 2 08:47:46 node2 kernel: [143624.821195] block drbd0: size = 1024
MB (1048508 KB)
Dec 2 08:47:46 node2 kernel: [143624.821526] block drbd0: recounting of
set bits took additional 0 jiffies
Dec 2 08:47:46 node2 kernel: [143624.821530] block drbd0: 0 KB (0 bits)
marked out-of-sync by on disk bit-map.
Dec 2 08:47:46 node2 kernel: [143624.821537] block drbd0: disk(
Attaching -> Consistent )
Dec 2 08:47:46 node2 kernel: [143624.826011] block drbd0: conn(
StandAlone -> Unconnected )
Dec 2 08:47:46 node2 kernel: [143624.826040] block drbd0: Starting
receiver thread (from drbd0_worker [15839])
Dec 2 08:47:46 node2 kernel: [143624.826079] block drbd0: receiver
(re)started
Dec 2 08:47:46 node2 kernel: [143624.826085] block drbd0: conn(
Unconnected -> WFConnection )
Dec 2 08:47:47 node2 kernel: [143625.074611] block drbd0: helper
command: /sbin/drbdadm fence-peer minor-0
Dec 2 08:47:47 node2 kernel: [143625.133531] block drbd0: Handshake
successful: Agreed network protocol version 91
Dec 2 08:47:48 node2 crm-fence-peer.sh[15957]: invoked for cluster-ocfs
Dec 2 08:47:48 node2 crm-fence-peer.sh[15957]: INFO suitable constraint
already placed: 'drbd-fence-by-handler-cluster-ocfs-ms_drbd_cluster_ocfs'
Dec 2 08:47:48 node2 kernel: [143626.109469] block drbd0: helper
command: /sbin/drbdadm fence-peer minor-0 exit code 4 (0x400)
Dec 2 08:47:48 node2 kernel: [143626.109473] block drbd0: fence-peer
helper returned 4 (peer was fenced)
Dec 2 08:47:48 node2 kernel: [143626.109481] block drbd0: role(
Secondary -> Primary ) disk( Consistent -> UpToDate ) pdsk( DUnknown ->
Outdated )
Dec 2 08:47:48 node2 kernel: [143626.109833] block drbd0: Creating new
current UUID
Dec 2 08:47:48 node2 kernel: [143626.110142] block drbd0: conn(
WFConnection -> WFReportParams )
Dec 2 08:47:48 node2 kernel: [143626.110165] block drbd0: Starting
asender thread (from drbd0_receiver [15850])
Dec 2 08:47:48 node2 kernel: [143626.110232] block drbd0:
data-integrity-alg: <not-used>
Dec 2 08:47:48 node2 kernel: [143626.110246] block drbd0:
drbd_sync_handshake:
Dec 2 08:47:48 node2 kernel: [143626.110250] block drbd0: self
DF81B74C806482F7:D97B1C1FD6F6A890:ADA4AB4A18C972CD:C25D6F4D22879C5C
bits:0 flags:0
Dec 2 08:47:48 node2 kernel: [143626.110254] block drbd0: peer
ADA4AB4A18C972CC:0000000000000000:BBD6AD66F9958D13:FE0A927040B5EE53
bits:0 flags:0
Dec 2 08:47:48 node2 kernel: [143626.110258] block drbd0:
uuid_compare()=2 by rule 80
Dec 2 08:47:48 node2 kernel: [143626.110260] block drbd0: Becoming sync
source due to disk states.
Dec 2 08:47:48 node2 kernel: [143626.110263] block drbd0: Writing the
whole bitmap, full sync required after drbd_sync_handshake.
Dec 2 08:47:48 node2 kernel: [143626.110667] block drbd0: 1024 MB
(262127 bits) marked out-of-sync by on disk bit-map.
Dec 2 08:47:48 node2 kernel: [143626.110915] block drbd0: peer( Unknown
-> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( Outdated ->
Inconsistent )
Dec 2 08:47:48 node2 kernel: [143626.115046] block drbd0: conn(
WFBitMapS -> SyncSource )
Dec 2 08:47:48 node2 kernel: [143626.115057] block drbd0: Began resync
as SyncSource (will sync 1048508 KB [262127 bits set]).
Dec 2 08:49:18 node2 kernel: [143716.815795] block drbd0: Resync done
(total 90 sec; paused 0 sec; 11648 K/sec)
Dec 2 08:49:18 node2 kernel: [143716.815803] block drbd0: conn(
SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )
Dec 2 08:49:18 node2 kernel: [143716.998813] block drbd0: peer(
Secondary -> Primary )
THANKS!
More information about the Pacemaker
mailing list