[Pacemaker] LVM restarts after SLES upgrade

Frank Meier frank.meier at hr-group.de
Tue Apr 24 13:51:30 UTC 2012


I didn't notice anything unusual, so far I can appraise.
Mit freundlichen Grüßen

Frank Meier
UNIX-Basis

Hamm Reno Group GmbH
Industriegebiet West | D-66987 Thaleischweiler-Fröschen
T.+49(0)6334 444-8322 | F.+49(0)6334 444-8190
frank.meier at hr-group.de | www.reno.de
___________________________________________________________________

Sitz: Am Tie 7 | D-49086 Osnabrück
Handelsregister Osnabrück HRB 19587
Geschäftsführer: Hans-Jürgen de Fries,
Jens Gransee, Manfred Klumpp,
Robert Reisch

Am 24.04.2012 12:02, schrieb pacemaker-request at oss.clusterlabs.org:
> Send Pacemaker mailing list submissions to
>         pacemaker at oss.clusterlabs.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> or, via email, send a message with subject or body 'help' to
>         pacemaker-request at oss.clusterlabs.org
> 
> You can reach the person managing the list at
>         pacemaker-owner at oss.clusterlabs.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Pacemaker digest..."
> 
> 
> Today's Topics:
> 
>    1. Re: LVM restarts after SLES upgrade (emmanuel segura)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Tue, 24 Apr 2012 10:08:05 +0000
> From: emmanuel segura <emi2fast at gmail.com>
> To: The Pacemaker cluster resource manager
>         <pacemaker at oss.clusterlabs.org>
> Subject: Re: [Pacemaker] LVM restarts after SLES upgrade
> Message-ID:
>         <CAE7pJ3DUtrvJR-QJFu_Ku1Gv1XiuUYvSiWwuqJ_FRn_EX_-O6w at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
> 
> Hello Frank
> 
> If you would like to see what happening when the lvm hungs, put your
> lvm configuration in DEBUG MODE
> 
> man lvm.conf
> 
> 
> 2012/4/24, Frank Meier <frank.meier at hr-group.de>:
>> Oh, you are right, but this is an mistake by reconfigure the
>> testszenario. At the original installation it was in the right order.
>> Thanks
>>
>> Mit freundlichen Gr??en
>>
>> Frank Meier
>> UNIX-Basis
>>
>> Hamm Reno Group GmbH
>> Industriegebiet West | D-66987 Thaleischweiler-Fr?schen
>> T.+49(0)6334 444-8322 | F.+49(0)6334 444-8190
>> frank.meier at hr-group.de | www.reno.de
>> ___________________________________________________________________
>>
>> Sitz: Am Tie 7 | D-49086 Osnabr?ck
>> Handelsregister Osnabr?ck HRB 19587
>> Gesch?ftsf?hrer: Hans-J?rgen de Fries,
>> Jens Gransee, Manfred Klumpp,
>> Robert Reisch
>>
>> Am 24.04.2012 11:28, schrieb pacemaker-request at oss.clusterlabs.org:
>>> Send Pacemaker mailing list submissions to
>>>         pacemaker at oss.clusterlabs.org
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>         http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>> or, via email, send a message with subject or body 'help' to
>>>         pacemaker-request at oss.clusterlabs.org
>>>
>>> You can reach the person managing the list at
>>>         pacemaker-owner at oss.clusterlabs.org
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of Pacemaker digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>>    1. Re: LVM restarts after SLES upgrade (emmanuel segura)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Tue, 24 Apr 2012 09:34:12 +0000
>>> From: emmanuel segura <emi2fast at gmail.com>
>>> To: The Pacemaker cluster resource manager
>>>         <pacemaker at oss.clusterlabs.org>
>>> Subject: Re: [Pacemaker] LVM restarts after SLES upgrade
>>> Message-ID:
>>>
>>> <CAE7pJ3DGvkzMR1d9HNm76s8wtrAj0t1==31RnafHe-YEgTV06Q at mail.gmail.com>
>>> Content-Type: text/plain; charset=ISO-8859-1
>>>
>>> Hello Frank
>>>
>>> Maybe this it's not the probelem, but i see this constrain wrong from
>>> my point of view
>>> =============================================================
>>> order o-Testclustervm inf: c-xen-vg-fs vm-clusterTest
>>> order o-clvmglue-xenvgfs inf: c-clvm-glue c-xen-vg-fs
>>> =============================================================
>>> to be
>>> =============================================================
>>> order o-clvmglue-xenvgfs inf: c-clvm-glue c-xen-vg-fs
>>> order o-Testclustervm inf: c-xen-vg-fs vm-clusterTest
>>> =============================================================
>>> 2012/4/24, Frank Meier <frank.meier at hr-group.de>:
>>>> Hi,
>>>>
>>>> Every time the vgdisplay -v TestXenVG is hanging(ca.2min)
>>>>
>>>> I see two of this peocesses:
>>>> /bin/sh /usr/lib/ocf/resource.d//heartbeat/LVM monitor
>>>> /bin/sh /usr/lib/ocf/resource.d//heartbeat/LVM monitor
>>>> is this OK, or have we a race condition?
>>>>
>>>>
>>>>
>>>> Hi,
>>>>
>>>> I do a new try.
>>>>
>>>> I've moved our productive VMs to another server and reconfigure the
>>>> cluster. But the problem is the same: every hour(or shorter, it seemd to
>>>> be  full, half or quarter hour), the LVM is restarted.
>>>> Last I saw vgdisplay -v TestXenVG hanging, I don't if this was every
>>>> time.
>>>>
>>>> I hope anybody can help. It has to be run expressly.
>>>>
>>>> Thanks
>>>>
>>>> Frank
>>>>
>>>>
>>>> vgs
>>>>   VG        #PV #LV #SN Attr   VSize  VFree
>>>>   TestXenVG   2   2   0 wz--nc 67.99g 61.80g
>>>>
>>>> xencluster1:~ # ps aux|grep clvm
>>>> root     21529  0.0  0.0   4520   528 pts/2    S+   17:36   0:00 grep
>>>> clvm
>>>> root     21756  0.0  0.1 155348 62920 ?        SLsl 13:44   0:07
>>>> /usr/sbin/clvmd -d0
>>>>
>>>>
>>>> xencluster1:~ # time vgdisplay -v TestXenVG
>>>>     Using volume group(s) on command line
>>>>     Finding volume group "TestXenVG"
>>>>   --- Volume group ---
>>>>   VG Name               TestXenVG
>>>>   System ID
>>>>   Format                lvm2
>>>>   Metadata Areas        2
>>>>   Metadata Sequence No  6
>>>>   VG Access             read/write
>>>>   VG Status             resizable
>>>>   Clustered             yes
>>>>   Shared                no
>>>>   MAX LV                0
>>>>   Cur LV                2
>>>>   Open LV               2
>>>>   Max PV                0
>>>>   Cur PV                2
>>>>   Act PV                2
>>>>   VG Size               67.99 GiB
>>>>   PE Size               4.00 MiB
>>>>   Total PE              17406
>>>>   Alloc PE / Size       1586 / 6.20 GiB
>>>>   Free  PE / Size       15820 / 61.80 GiB
>>>>   VG UUID               hveP8R-8pXh-b9QB-OGKG-ZmPa-xXXQ-f3MYGx
>>>>
>>>>   --- Logical volume ---
>>>>   LV Name                /dev/TestXenVG/vm1
>>>>   VG Name                TestXenVG
>>>>   LV UUID                W01y7D-7WfV-bBiU-pmnm-Tecg-ecrb-qjJ7GJ
>>>>   LV Write Access        read/write
>>>>   LV Status              available
>>>>   # open                 1
>>>>   LV Size                6.00 GiB
>>>>   Current LE             1536
>>>>   Segments               1
>>>>   Allocation             inherit
>>>>   Read ahead sectors     auto
>>>>   - currently set to     1024
>>>>   Block device           253:4
>>>>
>>>>   --- Logical volume ---
>>>>   LV Name                /dev/TestXenVG/xen_config
>>>>   VG Name                TestXenVG
>>>>   LV UUID                lrEjLC-yrlv-Kddh-JIb8-A1pT-t6b1-W4dw3f
>>>>   LV Write Access        read/write
>>>>   LV Status              available
>>>>   # open                 1
>>>>   LV Size                200.00 MiB
>>>>   Current LE             50
>>>>   Segments               1
>>>>   Allocation             inherit
>>>>   Read ahead sectors     auto
>>>>   - currently set to     1024
>>>>   Block device           253:5
>>>>
>>>>   --- Physical volumes ---
>>>>   PV Name               /dev/mapper/mpathi
>>>>   PV UUID               N7NE4R-v1ji-23cS-NzIn-v6Zt-S3Rp-HlB5jS
>>>>   PV Status             allocatable
>>>>   Total PE / Free PE    8703 / 7117
>>>>
>>>>   PV Name               /dev/mapper/mpathj
>>>>   PV UUID               cR567j-PM7y-5NUN-SGsJ-JO2h-rn0e-jTk7sm
>>>>   PV Status             allocatable
>>>>   Total PE / Free PE    8703 / 8703
>>>>
>>>>
>>>> real    0m0.033s
>>>> user    0m0.004s
>>>> sys     0m0.016s
>>>>
>>>> CIB:
>>>>
>>>> node xencluster1 \
>>>>         attributes standby="off"
>>>> node xencluster2 \
>>>>         attributes standby="off"
>>>> primitive STONITH-Node1 stonith:meatware \
>>>>         meta target-role="Started" is-managed="true" \
>>>>         operations $id="STONITH-Node1-operations" \
>>>>         op monitor interval="3600" timeout="20" start-delay="15" \
>>>>         params hostlist="xencluster1"
>>>> primitive STONITH-Node2 stonith:meatware \
>>>>         meta target-role="Started" is-managed="true" \
>>>>         operations $id="STONITH-Node2-operations" \
>>>>         op monitor interval="3600" timeout="20" start-delay="15" \
>>>>         params hostlist="xencluster2"
>>>> primitive clvm ocf:lvm2:clvmd \
>>>>         operations $id="clvm-operations" \
>>>>         op monitor interval="10" timeout="20" \
>>>>         op start interval="0" timeout="90" \
>>>>         op stop interval="0" timeout="100"
>>>> primitive clvm-xenvg ocf:heartbeat:LVM \
>>>>         operations $id="clvm-xenvg-operations" \
>>>>         op monitor interval="10" timeout="90" \
>>>>         op start interval="0" timeout="60" \
>>>>         op stop interval="0" timeout="60" \
>>>>         params volgrpname="TestXenVG" \
>>>>         meta target-role="Started"
>>>> primitive cmirror ocf:lvm2:cmirrord \
>>>>         operations $id="cmirror-operations" \
>>>>         op monitor interval="10" timeout="20" \
>>>>         op start interval="0" timeout="90" \
>>>>         op stop interval="0" timeout="100" \
>>>>         params daemon_timeout="30"
>>>> primitive dlm ocf:pacemaker:controld \
>>>>         operations $id="dlm-operations" \
>>>>         op monitor interval="60" timeout="60" start-delay="0" \
>>>>         op start interval="0" timeout="90" \
>>>>         op stop interval="0" timeout="100"
>>>> primitive fs-config-xen ocf:heartbeat:Filesystem \
>>>>         operations $id="fs-config-xen-operations" \
>>>>         op monitor interval="20" timeout="40" \
>>>>         params device="/dev/mapper/TestXenVG-xen_config"
>>>> directory="/etc/xen/vm" fstype="ocfs2"
>>>> primitive o2cb ocf:ocfs2:o2cb \
>>>>         operations $id="o2cb-operations" \
>>>>         op monitor interval="60" timeout="60" \
>>>>         op start interval="0" timeout="90" \
>>>>         op stop interval="0" timeout="100"
>>>> primitive vm-clusterTest ocf:heartbeat:Xen \
>>>>         meta target-role="Started" allow-migrate="true" is-managed="true"
>>>> \
>>>>         operations $id="vm-clusterTest-operations" \
>>>>         op monitor interval="10" timeout="30" \
>>>>         op start interval="0" timeout="60" \
>>>>         op stop interval="0" timeout="60" \
>>>>         op migrate_to interval="0" timeout="360" \
>>>>         params xmfile="/etc/xen/vm/ClusterTest"
>>>> group clvm-glue dlm clvm o2cb cmirror
>>>> group xen-vg-fs clvm-xenvg fs-config-xen
>>>> clone c-clvm-glue clvm-glue \
>>>>         meta target-role="Started" interleave="true" ordered="true"
>>>> clone c-xen-vg-fs xen-vg-fs \
>>>>         meta target-role="Started" interleave="true" ordered="true"
>>>> is-managed="true"
>>>> location STONITHnode1Constraint STONITH-Node1 -inf: xencluster1
>>>> location STONITHnode2Constraint STONITH-Node2 -inf: xencluster2
>>>> colocation colo-clvmglue-xenvgsfs inf: c-xen-vg-fs c-clvm-glue
>>>> order o-Testclustervm inf: c-xen-vg-fs vm-clusterTest
>>>> order o-clvmglue-xenvgfs inf: c-clvm-glue c-xen-vg-fs
>>>> property $id="cib-bootstrap-options" \
>>>>         dc-version="1.1.6-b988976485d15cb702c9307df55512d323831a5e" \
>>>>         no-quorum-policy="ignore" \
>>>>         default-resource-stickiness="100000" \
>>>>         last-lrm-refresh="1335186478" \
>>>>         cluster-infrastructure="openais" \
>>>>         expected-quorum-votes="2" \
>>>>         default-action-timeout="120s" \
>>>>         maintenance-mode="false"
>>>> op_defaults $id="op_defaults-options" \
>>>>         record-pending="false"
>>>>
>>>>
>>>>
>>>> Log:
>>>> Apr 23 17:14:29 xencluster1 clvmd[21756]: Syncing device names
>>>> Apr 23 17:14:29 xencluster1 clvmd[21756]: Reply from node 4914420a: 0
>>>> bytes
>>>> Apr 23 17:14:29 xencluster1 clvmd[21756]: Got 1 replies, expecting: 2
>>>> Apr 23 17:14:29 xencluster1 clvmd[21756]: LVM thread waiting for work
>>>> Apr 23 17:15:59 xencluster1 lrmd: [8619]: WARN: clvm-xenvg:0:monitor
>>>> process (PID 8351) timed out (try 1).  Killing with signal SIGTERM (15).
>>>> Apr 23 17:15:59 xencluster1 lrmd: [8619]: WARN: operation monitor[339]
>>>> on clvm-xenvg:0 for client 8622: pid 8351 timed out
>>>> Apr 23 17:15:59 xencluster1 crmd: [8622]: ERROR: process_lrm_event: LRM
>>>> operation clvm-xenvg:0_monitor_10000 (339) Timed Out (timeout=90000ms)
>>>> pr 23 17:15:59 xencluster1 crmd: [8622]: info: process_graph_event:
>>>> Action clvm-xenvg:0_monitor_10000 arrived after a completed transition
>>>> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: abort_transition_graph:
>>>> process_graph_event:481 - Triggered transition abort (complete=1,
>>>> tag=lrm_rsc_op, id=clvm-xenvg:0_monitor_10000,
>>>> magic=2:-2;12:384:0:a2db503d-6be0-4ed9-a94c-0e7031c748ad, cib=0.8361.49)
>>>> : Inactive graph
>>>> Apr 23 17:15:59 xencluster1 crmd: [8622]: WARN: update_failcount:
>>>> Updating failcount for clvm-xenvg:0 on xencluster1 after failed monitor:
>>>> rc=-2 (update=value++, time=1335194159)
>>>> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: do_state_transition:
>>>> State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
>>>> cause=C_FSA_INTERNAL origin=abort_transition_graph ]
>>>> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: do_state_transition: All
>>>> 2 cluster nodes are eligible to run resources.
>>>> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: do_pe_invoke: Query
>>>> 2608: Requesting the current CIB: S_POLICY_ENGINE
>>>> Apr 23 17:15:59 xencluster1 attrd: [8620]: notice: attrd_trigger_update:
>>>> Sending flush op to all hosts for: fail-count-clvm-xenvg:0 (8)
>>>> Apr 23 17:15:59 xencluster1 attrd: [8620]: notice: attrd_perform_update:
>>>> Sent update 1231: fail-count-clvm-xenvg:0=8
>>>> Apr 23 17:15:59 xencluster1 attrd: [8620]: notice: attrd_trigger_update:
>>>> Sending flush op to all hosts for: last-failure-clvm-xenvg:0 (1335194159)
>>>> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: do_pe_invoke_callback:
>>>> Invoking the PE: query=2608, ref=pe_calc-dc-1335194159-1765, seq=2556,
>>>> quorate=1
>>>> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: abort_transition_graph:
>>>> te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair,
>>>> id=status-xencluster1-fail-count-clvm-xenvg.0,
>>>> name=fail-count-clvm-xenvg:0, value=8, magic=NA, cib=0.8361.50) :
>>>> Transient attribute: update
>>>> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: do_pe_invoke: Query
>>>> 2609: Requesting the current CIB: S_POLICY_ENGINE
>>>> Apr 23 17:15:59 xencluster1 attrd: [8620]: notice: attrd_perform_update:
>>>> Sent update 1233: last-failure-clvm-xenvg:0=1335194159
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_config: On
>>>> loss of CCM Quorum: Ignore
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
>>>> Operation STONITH-Node1_last_failure_0 found resource STONITH-Node1
>>>> active on xencluster2
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
>>>> Operation dlm:1_last_failure_0 found resource dlm:1 active on xencluster2
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
>>>> Operation clvm:1_last_failure_0 found resource clvm:1 active on
>>>> xencluster2
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
>>>> Operation cmirror:1_last_failure_0 found resource cmirror:1 active on
>>>> xencluster2
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
>>>> Operation clvm:0_last_failure_0 found resource clvm:0 active on
>>>> xencluster1
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
>>>> Operation STONITH-Node2_last_failure_0 found resource STONITH-Node2
>>>> active on xencluster1
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
>>>> Operation dlm:0_last_failure_0 found resource dlm:0 active on xencluster1
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
>>>> Operation clvm-xenvg:0_last_failure_0 found resource clvm-xenvg:0 active
>>>> on xencluster1
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: WARN: unpack_rsc_op:
>>>> Processing failed op clvm-xenvg:0_monitor_10000 on xencluster1: unknown
>>>> exec error (-2)
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
>>>> Operation cmirror:0_last_failure_0 found resource cmirror:0 active on
>>>> xencluster1
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
>>>> Operation o2cb:0_last_failure_0 found resource o2cb:0 active on
>>>> xencluster1
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice:
>>>> common_apply_stickiness: c-xen-vg-fs can fail 999993 more times on
>>>> xencluster1 before being forced off
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice:
>>>> common_apply_stickiness: c-xen-vg-fs can fail 999993 more times on
>>>> xencluster1 before being forced off
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice:
>>>> common_apply_stickiness: c-xen-vg-fs can fail 999993 more times on
>>>> xencluster2 before being forced off
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice:
>>>> common_apply_stickiness: c-xen-vg-fs can fail 999993 more times on
>>>> xencluster2 before being forced off
>>>> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: do_pe_invoke_callback:
>>>> Invoking the PE: query=2609, ref=pe_calc-dc-1335194159-1766, seq=2556,
>>>> quorate=1
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: RecurringOp:  Start
>>>> recurring monitor (10s) for clvm-xenvg:0 on xencluster1
>>>> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: abort_transition_graph:
>>>> te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair,
>>>> id=status-xencluster1-last-failure-clvm-xenvg.0,
>>>> name=last-failure-clvm-xenvg:0, value=1335194159, magic=NA,
>>>> cib=0.8361.51) : Transient attribute: update
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: LogActions: Recover
>>>> clvm-xenvg:0   (Started xencluster1)
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: LogActions: Restart
>>>> fs-config-xen:0        (Started xencluster1)
>>>> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: do_pe_invoke: Query
>>>> 2610: Requesting the current CIB: S_POLICY_ENGINE
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: LogActions: Leave
>>>>  clvm-xenvg:1   (Started xencluster2)
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: LogActions: Leave
>>>>  fs-config-xen:1        (Started xencluster2)
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: LogActions: Leave
>>>>  STONITH-Node1  (Started xencluster2)
>>>> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: LogActions: Leave
>>>>  STONITH-Node2  (Started xencluster1)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Mit freundlichen Gr??en
>>>>
>>>> Frank Meier
>>>> UNIX-Basis
>>>>
>>>> Hamm Reno Group GmbH
>>>> Industriegebiet West | D-66987 Thaleischweiler-Fr?schen
>>>> T.+49(0)6334 444-322 | F.+49(0)6334 444-190
>>>> frank.meier at hr-group.de | www.my-reno.com
>>>> ___________________________________________________________________
>>>>
>>>> Sitz: Am Tie 7 | D-49086 Osnabr?ck
>>>> Handelsregister Osnabr?ck HRB 19587
>>>> Gesch?ftsf?hrer: Hans-J?rgen de Fries,
>>>> Jens Gransee, Manfred Klumpp,
>>>> Robert Reisch, Albert Scherer
>>>> ________________________________________
>>>> Von: pacemaker-request at oss.clusterlabs.org
>>>> [pacemaker-request at oss.clusterlabs.org]
>>>> Gesendet: Donnerstag, 19. April 2012 10:07
>>>> An: pacemaker at oss.clusterlabs.org
>>>> Betreff: Pacemaker Digest, Vol 53, Issue 42
>>>>
>>>> Send Pacemaker mailing list submissions to
>>>>         pacemaker at oss.clusterlabs.org
>>>>
>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>         http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>> or, via email, send a message with subject or body 'help' to
>>>>         pacemaker-request at oss.clusterlabs.org
>>>>
>>>> You can reach the person managing the list at
>>>>         pacemaker-owner at oss.clusterlabs.org
>>>>
>>>> When replying, please edit your Subject line so it is more specific
>>>> than "Re: Contents of Pacemaker digest..."
>>>>
>>>>
>>>> Today's Topics:
>>>>
>>>>    1. Re: Pacemaker Digest, Vol 53, Issue 40 (emmanuel segura)
>>>>
>>>>
>>>> ----------------------------------------------------------------------
>>>>
>>>> Message: 1
>>>> Date: Thu, 19 Apr 2012 10:12:37 +0200
>>>> From: emmanuel segura <emi2fast at gmail.com>
>>>> To: The Pacemaker cluster resource manager
>>>>         <pacemaker at oss.clusterlabs.org>
>>>> Subject: Re: [Pacemaker] Pacemaker Digest, Vol 53, Issue 40
>>>> Message-ID:
>>>>
>>>> <CAE7pJ3B5L1m2s=OQ=cUDd9=mqy50ORhP+eiDteCRp5_6JLR6jQ at mail.gmail.com>
>>>> Content-Type: text/plain; charset="iso-8859-1"
>>>>
>>>> vgchange -cy XenVG
>>>>
>>>> your vg it's not clustered and check the clvmd process
>>>>
>>>> ps -ef | grep clvmd
>>>>
>>>> Il giorno 19 aprile 2012 09:45, Frank Meier <frank.meier at hr-group.de> ha
>>>> scritto:
>>>>
>>>>> xencluster1:~ # vgs
>>>>>  VG    #PV #LV #SN Attr   VSize   VFree
>>>>>  XenVG   8  32   0 wz--n- 938.16g 530.98g
>>>>>
>>>>>
>>>>> Mit freundlichen Gr??en
>>>>>
>>>>> Frank Meier
>>>>> UNIX-Basis
>>>>>
>>>>> Hamm Reno Group GmbH
>>>>> Industriegebiet West | D-66987 Thaleischweiler-Fr?schen
>>>>> T.+49(0)6334 444-8322 | F.+49(0)6334 444-8190
>>>>> frank.meier at hr-group.de | www.reno.de
>>>>> ___________________________________________________________________
>>>>>
>>>>> Sitz: Am Tie 7 | D-49086 Osnabr?ck
>>>>> Handelsregister Osnabr?ck HRB 19587
>>>>> Gesch?ftsf?hrer: Hans-J?rgen de Fries,
>>>>> Jens Gransee, Manfred Klumpp,
>>>>> Robert Reisch
>>>>>
>>>>> Am 19.04.2012 09:32, schrieb pacemaker-request at oss.clusterlabs.org:
>>>>>> Send Pacemaker mailing list submissions to
>>>>>>         pacemaker at oss.clusterlabs.org
>>>>>>
>>>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>>>         http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>> or, via email, send a message with subject or body 'help' to
>>>>>>         pacemaker-request at oss.clusterlabs.org
>>>>>>
>>>>>> You can reach the person managing the list at
>>>>>>         pacemaker-owner at oss.clusterlabs.org
>>>>>>
>>>>>> When replying, please edit your Subject line so it is more specific
>>>>>> than "Re: Contents of Pacemaker digest..."
>>>>>>
>>>>>>
>>>>>> Today's Topics:
>>>>>>
>>>>>>    1. Re: Pacemaker Digest, Vol 53, Issue 38 (emmanuel segura)
>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>>
>>>>>> Message: 1
>>>>>> Date: Thu, 19 Apr 2012 09:37:31 +0200
>>>>>> From: emmanuel segura <emi2fast at gmail.com>
>>>>>> To: The Pacemaker cluster resource manager
>>>>>>         <pacemaker at oss.clusterlabs.org>
>>>>>> Subject: Re: [Pacemaker] Pacemaker Digest, Vol 53, Issue 38
>>>>>> Message-ID:
>>>>>>         <CAE7pJ3CovBAsjf+vbbJGEKuqTTvFiWWpn0S=UvOTSoyQpeTa=
>>>>> w at mail.gmail.com>
>>>>>> Content-Type: text/plain; charset="iso-8859-1"
>>>>>>
>>>>>> give this command
>>>>>>
>>>>>> vgs
>>>>>>
>>>>>> Il giorno 19 aprile 2012 09:32, Frank Meier <frank.meier at hr-group.de>
>>>>>> ha
>>>>>> scritto:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> here my lvm.conf:
>>>>>>>
>>>>>>> devices {
>>>>>>>    dir = "/dev"
>>>>>>>    scan = [ "/dev" ]
>>>>>>>    filter = [ "a|/dev/mapper|", "r/.*/" ]
>>>>>>>    cache = "/etc/lvm/.cache"
>>>>>>>    write_cache_state = 1
>>>>>>>    types = [ "device-mapper", 16 ]
>>>>>>>    sysfs_scan = 1
>>>>>>>    md_component_detection = 1
>>>>>>> }
>>>>>>> log {
>>>>>>>    verbose = 0
>>>>>>>    syslog = 1
>>>>>>>    overwrite = 0
>>>>>>>    level = 0
>>>>>>>    indent = 1
>>>>>>>    command_names = 0
>>>>>>>    prefix = "  "
>>>>>>> }
>>>>>>> backup {
>>>>>>>    backup = 1
>>>>>>>    backup_dir = "/etc/lvm/backup"
>>>>>>>    archive = 1
>>>>>>>    archive_dir = "/etc/lvm/archive"
>>>>>>>    retain_min = 10
>>>>>>>    retain_days = 30
>>>>>>> }
>>>>>>> shell {
>>>>>>>    history_size = 100
>>>>>>> }
>>>>>>> global {
>>>>>>>    umask = 077
>>>>>>>    test = 0
>>>>>>>    activation = 1
>>>>>>>    proc = "/proc"
>>>>>>>    locking_type = 3
>>>>>>>    locking_dir = "/var/lock/lvm"
>>>>>>> }
>>>>>>> activation {
>>>>>>>    missing_stripe_filler = "error"
>>>>>>>    mirror_region_size = 512
>>>>>>>    reserved_stack = 256
>>>>>>>    reserved_memory = 8192
>>>>>>>    process_priority = -18
>>>>>>> }
>>>>>>> dmeventd {
>>>>>>>        mirror_library = "libdevmapper-event-lvm2mirror.so.2.02"
>>>>>>>        snapshot_library = "libdevmapper-event-lvm2snapshot.so.2.02"
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> Mit freundlichen Gr??en
>>>>>>>
>>>>>>> Frank Meier
>>>>>>> UNIX-Basis
>>>>>>>
>>>>>>> Hamm Reno Group GmbH
>>>>>>> Industriegebiet West | D-66987 Thaleischweiler-Fr?schen
>>>>>>> T.+49(0)6334 444-8322 | F.+49(0)6334 444-8190
>>>>>>> frank.meier at hr-group.de | www.reno.de
>>>>>>> ___________________________________________________________________
>>>>>>>
>>>>>>> Sitz: Am Tie 7 | D-49086 Osnabr?ck
>>>>>>> Handelsregister Osnabr?ck HRB 19587
>>>>>>> Gesch?ftsf?hrer: Hans-J?rgen de Fries,
>>>>>>> Jens Gransee, Manfred Klumpp,
>>>>>>> Robert Reisch
>>>>>>>
>>>>>>> Am 19.04.2012 09:13, schrieb pacemaker-request at oss.clusterlabs.org:
>>>>>>>> Send Pacemaker mailing list submissions to
>>>>>>>>         pacemaker at oss.clusterlabs.org
>>>>>>>>
>>>>>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>>>>>         http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>> or, via email, send a message with subject or body 'help' to
>>>>>>>>         pacemaker-request at oss.clusterlabs.org
>>>>>>>>
>>>>>>>> You can reach the person managing the list at
>>>>>>>>         pacemaker-owner at oss.clusterlabs.org
>>>>>>>>
>>>>>>>> When replying, please edit your Subject line so it is more specific
>>>>>>>> than "Re: Contents of Pacemaker digest..."
>>>>>>>>
>>>>>>>>
>>>>>>>> Today's Topics:
>>>>>>>>
>>>>>>>>    1. Re: LVM restarts after SLES upgrade (emmanuel segura)
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>
>>>>>>>> Message: 1
>>>>>>>> Date: Thu, 19 Apr 2012 09:18:37 +0200
>>>>>>>> From: emmanuel segura <emi2fast at gmail.com>
>>>>>>>> To: The Pacemaker cluster resource manager
>>>>>>>>         <pacemaker at oss.clusterlabs.org>
>>>>>>>> Subject: Re: [Pacemaker] LVM restarts after SLES upgrade
>>>>>>>> Message-ID:
>>>>>>>>         <CAE7pJ3Ab1ZzSpou0=vvrO=
>>>>>>> mb3ur2+K1XhALLJEpj-sOOUEfwTA at mail.gmail.com>
>>>>>>>> Content-Type: text/plain; charset="iso-8859-1"
>>>>>>>>
>>>>>>>> maybe you have problem with clvm
>>>>>>>>
>>>>>>>>  can you show me your vm configuration?
>>>>>>>>
>>>>>>>> Il giorno 19 aprile 2012 08:29, Frank Meier <frank.meier at hr-group.de>
>>>>> ha
>>>>>>>> scritto:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I've installed a 2-Node Xen-Cluster with SLES 11 SP1.
>>>>>>>>>
>>>>>>>>> After an upgrade to SLES11 SP2 the cluster won't work as the old
>>>>>>>>> one.
>>>>>>>>>
>>>>>>>>> After some time, all LVs reorganize and so the VM where restarted.
>>>>>>>>>
>>>>>>>>> Any clue?
>>>>>>>>>
>>>>>>>>> CIB:
>>>>>>>>>
>>>>>>>>> node xencluster1 \
>>>>>>>>>        attributes standby="off"
>>>>>>>>> node xencluster2 \
>>>>>>>>>        attributes standby="off"
>>>>>>>>> primitive STONITH--Node1 stonith:meatware \
>>>>>>>>>        meta is-managed="true" target-role="Started" \
>>>>>>>>>        operations $id="STONITH--Node1-operations" \
>>>>>>>>>        op monitor interval="15" timeout="15" start-delay="15" \
>>>>>>>>>        params hostlist="xencluster1"
>>>>>>>>> primitive STONITH-Node2 stonith:meatware \
>>>>>>>>>        meta target-role="started" is-managed="true" \
>>>>>>>>>        operations $id="STONITH-Node2-operations" \
>>>>>>>>>        op monitor interval="15" timeout="15" start-delay="15" \
>>>>>>>>>        params hostlist="xencluster2"
>>>>>>>>> primitive clvm ocf:lvm2:clvmd \
>>>>>>>>>        operations $id="clvm-operations" \
>>>>>>>>>        op monitor interval="10" timeout="20" \
>>>>>>>>>        op start interval="0" timeout="90" \
>>>>>>>>>        op stop interval="0" timeout="100" \
>>>>>>>>>        params daemon_timeout="30"
>>>>>>>>> primitive clvm-xenvg ocf:heartbeat:LVM \
>>>>>>>>>        operations $id="clvm-xenvg-operations" \
>>>>>>>>>        op monitor interval="10" timeout="30" \
>>>>>>>>>        params volgrpname="XenVG"
>>>>>>>>> primitive cmirror ocf:lvm2:cmirrord \
>>>>>>>>>        operations $id="cmirror-operations" \
>>>>>>>>>        op monitor interval="10" timeout="20" \
>>>>>>>>>        op start interval="0" timeout="90" \
>>>>>>>>>        op stop interval="0" timeout="100" \
>>>>>>>>>        params daemon_timeout="30"
>>>>>>>>> primitive dlm ocf:pacemaker:controld \
>>>>>>>>>        operations $id="dlm-operations" \
>>>>>>>>>        op monitor interval="10" timeout="20" start-delay="0" \
>>>>>>>>>        op start interval="0" timeout="90" \
>>>>>>>>>        op stop interval="0" timeout="100"
>>>>>>>>> primitive fs-config-xen ocf:heartbeat:Filesystem \
>>>>>>>>>        meta is-managed="true" target-role="Started" \
>>>>>>>>>        operations $id="fs-config-xen-operations" \
>>>>>>>>>        op monitor interval="20" timeout="40" \
>>>>>>>>>        params device="/dev/mapper/XenVG-xenconfig"
>>>>>>> directory="/etc/xen/vm"
>>>>>>>>> fstype="ocfs2"
>>>>>>>>> primitive o2cb ocf:ocfs2:o2cb \
>>>>>>>>>        operations $id="o2cb-operations" \
>>>>>>>>>        op monitor interval="10" timeout="20" \
>>>>>>>>>        op start interval="0" timeout="90" \
>>>>>>>>>        op stop interval="0" timeout="100"
>>>>>>>>> primitive vm-cim ocf:heartbeat:Xen \
>>>>>>>>>        meta target-role="Started" is-managed="true"
>>>>>>> allow-migrate="true" \
>>>>>>>>>        operations $id="vm-cim-operations" \
>>>>>>>>>        op monitor interval="10" timeout="30" \
>>>>>>>>>        op start interval="0" timeout="60" \
>>>>>>>>>        op stop interval="0" timeout="60" \
>>>>>>>>>        op migrate_to interval="0" timeout="360" \
>>>>>>>>>        params xmfile="/etc/xen/vm/cim"
>>>>>>>>> primitive vm-deprepo ocf:heartbeat:Xen \
>>>>>>>>>        meta target-role="started" is-managed="true"
>>>>>>> allow-migrate="true" \
>>>>>>>>>        operations $id="vm-deprepo-operations" \
>>>>>>>>>        op monitor interval="10" timeout="30" \
>>>>>>>>>        op start interval="0" timeout="60" \
>>>>>>>>>        op stop interval="0" timeout="60" \
>>>>>>>>>        op migrate_to interval="0" timeout="360" \
>>>>>>>>>        params xmfile="/etc/xen/vm/debrepo"
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> group clvm-glue dlm clvm o2cb cmirror \
>>>>>>>>>        meta target-role="Started"
>>>>>>>>> group xen-vg-fs clvm-xenvg fs-config-xen
>>>>>>>>> clone c-clvm-glue clvm-glue \
>>>>>>>>>        meta interleave="true" ordered="true" target-role="Started"
>>>>>>>>> clone c-xen-vg-fs xen-vg-fs \
>>>>>>>>>        meta interleave="true" ordered="true" is-managed="true"
>>>>>>>>> target-role="Started"
>>>>>>>>> location STONITHnode1Constraint STONITH--Node1 -inf: xencluster1
>>>>>>>>> colocation colo-clvmglue-xenvgsfs inf: c-xen-vg-fs c-clvm-glue
>>>>>>>>> order o-cim inf: c-xen-vg-fs vm-cim
>>>>>>>>> order o-clvmglue-xenvgfs inf: c-clvm-glue c-xen-vg-fs
>>>>>>>>> order o-deprepo inf: c-xen-vg-fs vm-deprepo
>>>>>>>>> order o-ilmt inf: c-xen-vg-fs vm-ilmt
>>>>>>>>> order o-intranet inf: c-xen-vg-fs vm-intranet
>>>>>>>>> order o-nagois inf: c-xen-vg-fs vm-nagios
>>>>>>>>> order o-oneiroi inf: c-xen-vg-fs vm-oneiroi
>>>>>>>>> order o-otbridge inf: c-xen-vg-fs vm-otbridge
>>>>>>>>> order o-soa inf: c-xen-vg-fs vm-soa
>>>>>>>>> order o-topdesk-entw inf: c-xen-vg-fs vm-topdesk-entw
>>>>>>>>> order o-traumschiff inf: c-xen-vg-fs vm-traumschiff
>>>>>>>>> order o-virenscanner inf: c-xen-vg-fs vm-virenscanner
>>>>>>>>> property $id="cib-bootstrap-options" \
>>>>>>>>>        dc-version="1.1.6-b988976485d15cb702c9307df55512d323831a5e" \
>>>>>>>>>        no-quorum-policy="ignore" \
>>>>>>>>>        default-resource-stickiness="100000" \
>>>>>>>>>        last-lrm-refresh="1334518148" \
>>>>>>>>>        cluster-infrastructure="openais" \
>>>>>>>>>        expected-quorum-votes="2" \
>>>>>>>>>        default-action-timeout="120s" \
>>>>>>>>>        maintenance-mode="true"
>>>>>>>>> op_defaults $id="op_defaults-options" \
>>>>>>>>>        record-pending="false"
>>>>>>>>>
>>>>>>>>> Node 1Logs:
>>>>>>>>>
>>>>>>>>> Apr 15 22:01:10 xencluster1 clvmd[8763]: Sending message to all
>>>>> cluster
>>>>>>>>> nodes
>>>>>>>>> Apr 15 22:01:10 xencluster1 clvmd[8763]: process_work_item: local
>>>>>>>>> Apr 15 22:01:10 xencluster1 clvmd[8763]: process_local_command:
>>>>> unknown
>>>>>>>>> (0x2d) msg=0x7f5c40010e80, msglen =32, client=0x7f5c400578d0
>>>>>>>>> Apr 15 22:01:10 xencluster1 clvmd[8763]: Syncing device names
>>>>>>>>> Apr 15 22:01:10 xencluster1 clvmd[8763]: Reply from node 4914420a: 0
>>>>>>> bytes
>>>>>>>>> Apr 15 22:01:10 xencluster1 clvmd[8763]: Got 1 replies, expecting: 2
>>>>>>>>> Apr 15 22:01:10 xencluster1 clvmd[8763]: LVM thread waiting for work
>>>>>>>>> Apr 15 22:01:10 xencluster1 clvmd[8763]: 1226064394 got message from
>>>>>>>>> nodeid 1226064394 for 0. len 32
>>>>>>>>> Apr 15 22:01:40 xencluster1 lrmd: [8093]: WARN: clvm-xenvg:0:monitor
>>>>>>>>> process (PID 31785) timed out (try 1).  Killing with signal SIGTERM
>>>>>>> (15).
>>>>>>>>> Apr 15 22:01:40 xencluster1 lrmd: [8093]: WARN: operation
>>>>>>>>> monitor[92]
>>>>> on
>>>>>>>>> clvm-xenvg:0 for client 8096: pid 31785 timed out
>>>>>>>>> Apr 15 22:01:40 xencluster1 crmd: [8096]: ERROR: process_lrm_event:
>>>>> LRM
>>>>>>>>> operation clvm-xenvg:0_monitor_10000 (92) Timed Out
>>>>>>>>> (timeout=30000ms)
>>>>>>>>> Apr 15 22:01:42 xencluster1 corosync[8071]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 21ac
>>>>>>>>> Apr 15 22:01:42 xencluster1 corosync[8071]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 21ab
>>>>>>>>> Apr 15 22:01:42 xencluster1 corosync[8071]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 21ac
>>>>>>>>> Apr 15 22:01:42 xencluster1 corosync[8071]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 21ab
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> pr 15 22:01:50 xencluster1 clvmd[8763]: Send local reply
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: Read on local socket 14,
>>>>>>>>> len
>>>>> =
>>>>>>> 32
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: check_all_clvmds_running
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: down_callback. node
>>>>> 1226064394,
>>>>>>>>> state = 3
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: down_callback. node
>>>>> 1309950474,
>>>>>>>>> state = 3
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: Got pre command
>>>>>>>>> condition...
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: Writing status 0 down pipe
>>>>>>>>> 16
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: Waiting to do post command
>>>>>>>>> -
>>>>>>>>> state = 0
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: read on PIPE 15: 4 bytes:
>>>>>>> status:
>>>>>>>>> 0
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: background routine status
>>>>>>>>> was
>>>>>>>>> 0, sock_client=0x7f5c40057c10
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: distribute command: XID =
>>>>> 15473
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: num_nodes = 2
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: add_to_lvmqueue:
>>>>>>>>> cmd=0x7f5c4000dbd0. client=0x7f5c40057c10, msg=0x7f5c40057880,
>>>>>>>>> len=32,
>>>>>>>>> csid=(nil), xid=15473
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: Sending message to all
>>>>> cluster
>>>>>>>>> nodes
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: process_work_item: local
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: process_local_command:
>>>>> unknown
>>>>>>>>> (0x2d) msg=0x7f5c40010e80, msglen =32, client=0x7f5c40057c10
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: Syncing device names
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: Reply from node 4914420a: 0
>>>>>>> bytes
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: Got 1 replies, expecting: 2
>>>>>>>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: LVM thread waiting for work
>>>>>>>>> Apr 15 22:01:50 xencluster1 corosync[8071]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 21ab
>>>>>>>>> Apr 15 22:01:50 xencluster1 corosync[8071]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 21ac
>>>>>>>>> Apr 15 22:01:50 xencluster1 corosync[8071]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 21ab
>>>>>>>>> ...
>>>>>>>>> Apr 15 22:02:19 xencluster1 corosync[8071]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 21ac
>>>>>>>>> Apr 15 22:02:20 xencluster1 corosync[8071]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 21ab
>>>>>>>>> Apr 15 22:02:20 xencluster1 lrmd: [8093]: WARN: clvm-xenvg:0:monitor
>>>>>>>>> process (PID 771) timed out (try 1).  Killing with signal SIGTERM
>>>>> (15).
>>>>>>>>> Apr 15 22:02:20 xencluster1 lrmd: [8093]: WARN: operation
>>>>>>>>> monitor[92]
>>>>> on
>>>>>>>>> clvm-xenvg:0 for client 8096: pid 771 timed out
>>>>>>>>> Apr 15 22:02:20 xencluster1 crmd: [8096]: ERROR: process_lrm_event:
>>>>> LRM
>>>>>>>>> operation clvm-xenvg:0_monitor_10000 (92) Timed Out
>>>>>>>>> (timeout=30000ms)
>>>>>>>>> Apr 15 22:02:20 xencluster1 corosync[8071]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 21ac
>>>>>>>>> Apr 15 22:02:20 xencluster1 corosync[8071]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 21ab
>>>>>>>>> ...
>>>>>>>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: Got new connection on fd 17
>>>>>>>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: Read on local socket 17,
>>>>>>>>> len
>>>>> =
>>>>>>> 28
>>>>>>>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: creating pipe, [18, 19]
>>>>>>>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: Creating pre&post thread
>>>>>>>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: Created pre&post thread,
>>>>> state
>>>>>>> = 0
>>>>>>>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: in sub thread: client =
>>>>>>>>> 0x7f5c40058080
>>>>>>>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: Sub thread ready for work.
>>>>>>>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: doing PRE command LOCK_VG
>>>>>>>>> 'V_XenVG' at 1 (client=0x7f5c40058080)
>>>>>>>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: lock_resource 'V_XenVG',
>>>>>>>>> flags=0, mode=3
>>>>>>>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: lock_resource returning 0,
>>>>>>>>> lock_id=3a0001
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Apr 15 22:02:48 xencluster1 corosync[8071]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 21ab
>>>>>>>>> Apr 15 22:02:48 xencluster1 cib: [8092]: info: cib_stats: Processed
>>>>>>>>> 4
>>>>>>>>> operations (2500.00us average, 0% utilization) in the last 10min
>>>>>>>>> Apr 15 22:02:48 xencluster1 corosync[8071]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 21ac
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Apr 15 22:03:00 xencluster1 corosync[8071]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 21ac
>>>>>>>>> Apr 15 22:03:00 xencluster1 lrmd: [8093]: WARN: clvm-xenvg:0:monitor
>>>>>>>>> process (PID 2256) timed out (try 1).  Killing with signal SIGTERM
>>>>> (15).
>>>>>>>>> Apr 15 22:03:00 xencluster1 lrmd: [8093]: WARN: operation
>>>>>>>>> monitor[92]
>>>>> on
>>>>>>>>> clvm-xenvg:0 for client 8096: pid 2256 timed out
>>>>>>>>> Apr 15 22:03:00 xencluster1 crmd: [8096]: ERROR: process_lrm_event:
>>>>> LRM
>>>>>>>>> operation clvm-xenvg:0_monitor_10000 (92) Timed Out
>>>>>>>>> (timeout=30000ms)
>>>>>>>>> Apr 15 22:03:00 xencluster1 corosync[8071]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 21ab
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Node2 Log
>>>>>>>>>
>>>>>>>>> Apr 15 22:00:51 xencluster2 clvmd[12666]: Joined child thread
>>>>>>>>> Apr 15 22:00:51 xencluster2 clvmd[12666]: ret == 0, errno = 0.
>>>>> removing
>>>>>>>>> client
>>>>>>>>> Apr 15 22:00:51 xencluster2 clvmd[12666]: add_to_lvmqueue:
>>>>>>>>> cmd=0x7fa3d0028710. client=0x7fa3d0028760, msg=(nil), len=0,
>>>>> csid=(nil),
>>>>>>>>> xid=14778
>>>>>>>>> Apr 15 22:00:51 xencluster2 clvmd[12666]: process_work_item: free fd
>>>>> -1
>>>>>>>>> Apr 15 22:00:51 xencluster2 clvmd[12666]: LVM thread waiting for
>>>>>>>>> work
>>>>>>>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: crm_timer_popped:
>>>>>>>>> PEngine Recheck Timer (I_PE_CALC) just popped (900000ms)
>>>>>>>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: do_state_transition:
>>>>>>>>> State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
>>>>>>>>> cause=C_TIMER_POPPED origin=crm_timer_popped ]
>>>>>>>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: do_state_transition:
>>>>>>>>> Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
>>>>>>>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: do_state_transition:
>>>>> All
>>>>>>>>> 2 cluster nodes are eligible to run resources.
>>>>>>>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: do_pe_invoke: Query
>>>>> 983:
>>>>>>>>> Requesting the current CIB: S_POLICY_ENGINE
>>>>>>>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info:
>>>>>>>>> do_pe_invoke_callback:
>>>>>>>>> Invoking the PE: query=983, ref=pe_calc-dc-1334520052-604, seq=2212,
>>>>>>>>> quorate=1
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_config:
>>>>>>>>> On
>>>>>>>>> loss of CCM Quorum: Ignore
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation clvm:1_last_failure_0 found resource clvm:1 active on
>>>>>>> xencluster2
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation STONITH-Node2_last_failure_0 found resource STONITH-Node2
>>>>>>>>> active on xencluster2
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation STONITH--Node1_last_failure_0 found resource
>>>>>>>>> STONITH--Node1
>>>>>>>>> active on xencluster2
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation cmirror:1_last_failure_0 found resource cmirror:1 active
>>>>>>>>> on
>>>>>>>>> xencluster2
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation dlm:1_last_failure_0 found resource dlm:1 active on
>>>>>>> xencluster2
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation o2cb:1_last_failure_0 found resource o2cb:1 active on
>>>>>>> xencluster2
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation dlm:0_last_failure_0 found resource dlm:0 active on
>>>>>>> xencluster1
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation clvm:0_last_failure_0 found resource clvm:0 active on
>>>>>>> xencluster1
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation o2cb:0_last_failure_0 found resource o2cb:0 active on
>>>>>>> xencluster1
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation clvm-xenvg:0_last_failure_0 found resource clvm-xenvg:0
>>>>> active
>>>>>>>>> on xencluster1
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: WARN: unpack_rsc_op:
>>>>>>>>> Processing failed op vm-deprepo_last_failure_0 on xencluster1:
>>>>>>>>> unknown
>>>>>>>>> error (1)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  STONITH-Node2  (Started xencluster2)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  STONITH--Node1 (Started xencluster2)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  dlm:0  (Started xencluster1)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  clvm:0 (Started xencluster1)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  o2cb:0 (Started xencluster1)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  cmirror:0      (Started xencluster1)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  dlm:1  (Started xencluster2)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  clvm:1 (Started xencluster2)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  o2cb:1 (Started xencluster2)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  cmirror:1      (Started xencluster2)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  clvm-xenvg:0   (Started xencluster1)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  fs-config-xen:0        (Started xencluster1)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  clvm-xenvg:1   (Started xencluster2)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  fs-config-xen:1        (Started xencluster2)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  vm-ilmt        (Started xencluster1)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  vm-topdesk-entw        (Started xencluster1)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  vm-otbridge    (Started xencluster2)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  vm-virenscanner        (Started xencluster1)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  vm-deprepo     (Started xencluster1)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  vm-traumschiff (Started xencluster2)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  vm-cim (Started xencluster2)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  vm-oneiroi     (Started xencluster1)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  vm-intranet    (Started xencluster1)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  vm-soa (Started xencluster1)
>>>>>>>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice:
>>>>> process_pe_message:
>>>>>>>>> Transition 80: PEngine Input stored in:
>>>>>>> /var/lib/pengine/pe-input-271.bz2
>>>>>>>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: do_state_transition:
>>>>>>>>> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
>>>>>>>>> input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
>>>>>>>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: unpack_graph:
>>>>>>>>> Unpacked
>>>>>>>>> transition 80: 0 actions in 0 synapses
>>>>>>>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: do_te_invoke:
>>>>> Processing
>>>>>>>>> graph 80 (ref=pe_calc-dc-1334520052-604) derived from
>>>>>>>>> /var/lib/pengine/pe-input-271.bz2
>>>>>>>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: run_graph:
>>>>>>>>> ====================================================
>>>>>>>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: notice: run_graph:
>>>>> Transition
>>>>>>>>> 80 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
>>>>>>>>> Source=/var/lib/pengine/pe-input-271.bz2): Complete
>>>>>>>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: te_graph_trigger:
>>>>>>>>> Transition 80 is now complete
>>>>>>>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: notify_crmd:
>>>>> Transition
>>>>>>>>> 80 status: done - <null>
>>>>>>>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: do_state_transition:
>>>>>>>>> State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
>>>>>>>>> cause=C_FSA_INTERNAL origin=notify_crmd ]
>>>>>>>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: do_state_transition:
>>>>>>>>> Starting PEngine Recheck Timer
>>>>>>>>> Apr 15 22:01:00 xencluster2 clvmd[12666]: 1309950474 got message
>>>>>>>>> from
>>>>>>>>> nodeid 1226064394 for 0. len 32
>>>>>>>>> Apr 15 22:01:00 xencluster2 clvmd[12666]: add_to_lvmqueue:
>>>>>>>>> cmd=0x7fa3d0028780. client=0x6934a0, msg=0x7fa3d910063c, len=32,
>>>>>>>>> csid=0x7fffb4e5d944, xid=0
>>>>>>>>> Apr 15 22:01:00 xencluster2 clvmd[12666]: process_work_item: remote
>>>>>>>>> Apr 15 22:01:00 xencluster2 clvmd[12666]: process_remote_command
>>>>> unknown
>>>>>>>>> (0x2d) for clientid 0x5000000 XID 15337 on node 4914420a
>>>>>>>>> Apr 15 22:01:00 xencluster2 clvmd[12666]: Syncing device names
>>>>>>>>> Apr 15 22:01:00 xencluster2 clvmd[12666]: LVM thread waiting for
>>>>>>>>> work
>>>>>>>>> Apr 15 22:01:00 xencluster2 clvmd[12666]: 1309950474 got message
>>>>>>>>> from
>>>>>>>>> nodeid 1309950474 for 1226064394. len 18
>>>>>>>>> Apr 15 22:01:00 xencluster2 clvmd[12666]: 1309950474 got message
>>>>>>>>> from
>>>>>>>>> nodeid 1226064394 for 0. len 32
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Apr 15 22:01:02 xencluster2 clvmd[12666]: Joined child thread
>>>>>>>>> Apr 15 22:01:02 xencluster2 clvmd[12666]: ret == 0, errno = 0.
>>>>> removing
>>>>>>>>> client
>>>>>>>>> Apr 15 22:01:02 xencluster2 clvmd[12666]: add_to_lvmqueue:
>>>>>>>>> cmd=0x7fa3d004d7a0. client=0x7fa3d004d7f0, msg=(nil), len=0,
>>>>> csid=(nil),
>>>>>>>>> xid=14848
>>>>>>>>> Apr 15 22:01:02 xencluster2 clvmd[12666]: process_work_item: free fd
>>>>> -1
>>>>>>>>> Apr 15 22:01:02 xencluster2 clvmd[12666]: LVM thread waiting for
>>>>>>>>> work
>>>>>>>>> Apr 15 22:01:06 xencluster2 stonith-ng: [8283]: info:
>>>>>>>>> stonith_command:
>>>>>>>>> Processed st_execute from lrmd: rc=-1
>>>>>>>>> Apr 15 22:01:06 xencluster2 stonith: parse config info
>>>>> info=xencluster1
>>>>>>>>> Apr 15 22:01:06 xencluster2 stonith: meatware device OK.
>>>>>>>>> Apr 15 22:01:06 xencluster2 stonith-ng: [8283]: info: log_operation:
>>>>>>>>> STONITH--Node1: Performing: stonith -t meatware -S
>>>>>>>>> Apr 15 22:01:06 xencluster2 stonith-ng: [8283]: info: log_operation:
>>>>>>>>> STONITH--Node1: success:  0
>>>>>>>>> Apr 15 22:01:06 xencluster2 stonith-ng: [8283]: info:
>>>>>>>>> stonith_device_execute: Nothing to do for STONITH--Node1
>>>>>>>>> Apr 15 22:01:06 xencluster2 stonith-ng: [8283]: info:
>>>>>>>>> stonith_command:
>>>>>>>>> Processed st_execute from lrmd: rc=-1
>>>>>>>>> Apr 15 22:01:06 xencluster2 stonith: parse config info
>>>>> info=xencluster2
>>>>>>>>> Apr 15 22:01:06 xencluster2 stonith: meatware device OK.
>>>>>>>>> Apr 15 22:01:06 xencluster2 stonith-ng: [8283]: info: log_operation:
>>>>>>>>> STONITH-Node2: Performing: stonith -t meatware -S
>>>>>>>>> Apr 15 22:01:06 xencluster2 stonith-ng: [8283]: info: log_operation:
>>>>>>>>> STONITH-Node2: success:  0
>>>>>>>>> Apr 15 22:01:06 xencluster2 stonith-ng: [8283]: info:
>>>>>>>>> stonith_device_execute: Nothing to do for STONITH-Node2
>>>>>>>>> Apr 15 22:01:10 xencluster2 clvmd[12666]: 1309950474 got message
>>>>>>>>> from
>>>>>>>>> nodeid 1226064394 for 0. len 32
>>>>>>>>> Apr 15 22:01:10 xencluster2 clvmd[12666]: add_to_lvmqueue:
>>>>>>>>> cmd=0x7fa3d004d810. client=0x6934a0, msg=0x7fa3d9100cfc, len=32,
>>>>>>>>> csid=0x7fffb4e5d944, xid=0
>>>>>>>>> Apr 15 22:01:10 xencluster2 clvmd[12666]: process_work_item: remote
>>>>>>>>> Apr 15 22:01:10 xencluster2 clvmd[12666]: process_remote_command
>>>>> unknown
>>>>>>>>> (0x2d) for clientid 0x5000000 XID 15407 on node 4914420a
>>>>>>>>> Apr 15 22:01:10 xencluster2 clvmd[12666]: Syncing device names
>>>>>>>>> Apr 15 22:01:10 xencluster2 clvmd[12666]: LVM thread waiting for
>>>>>>>>> work
>>>>>>>>> Apr 15 22:01:10 xencluster2 clvmd[12666]: 1309950474 got message
>>>>>>>>> from
>>>>>>>>> nodeid 1309950474 for 1226064394. len 18
>>>>>>>>> Apr 15 22:01:10 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196
>>>>>>>>> Apr 15 22:01:10 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196
>>>>>>>>> Apr 15 22:01:10 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196
>>>>>>>>> ....
>>>>>>>>> Apr 15 22:01:11 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196
>>>>>>>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: Got new connection on fd 5
>>>>>>>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: Read on local socket 5,
>>>>>>>>> len
>>>>> =
>>>>>>> 28
>>>>>>>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: creating pipe, [12, 13]
>>>>>>>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: Creating pre&post thread
>>>>>>>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: Created pre&post thread,
>>>>>>> state =
>>>>>>>>> 0
>>>>>>>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: in sub thread: client =
>>>>>>>>> 0x7fa3d004d810
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: Syncing device names
>>>>>>>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: Reply from node 4e14420a:
>>>>>>>>> 0
>>>>>>> bytes
>>>>>>>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: Got 1 replies, expecting:
>>>>>>>>> 2
>>>>>>>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: LVM thread waiting for
>>>>>>>>> work
>>>>>>>>> Apr 15 22:01:12 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196
>>>>>>>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: 1309950474 got message
>>>>>>>>> from
>>>>>>>>> nodeid 1309950474 for 0. len 32
>>>>>>>>> Apr 15 22:01:12 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196
>>>>>>>>> Apr 15 22:01:12 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196
>>>>>>>>> ....
>>>>>>>>> Apr 15 22:01:21 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> Apr 15 22:01:21 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> Apr 15 22:01:21 xencluster2 stonith-ng: [8283]: info:
>>>>>>>>> stonith_command:
>>>>>>>>> Processed st_execute from lrmd: rc=-1
>>>>>>>>> Apr 15 22:01:21 xencluster2 stonith: parse config info
>>>>> info=xencluster1
>>>>>>>>> Apr 15 22:01:21 xencluster2 stonith: meatware device OK.
>>>>>>>>> Apr 15 22:01:21 xencluster2 stonith-ng: [8283]: info: log_operation:
>>>>>>>>> STONITH--Node1: Performing: stonith -t meatware -S
>>>>>>>>> Apr 15 22:01:21 xencluster2 stonith-ng: [8283]: info: log_operation:
>>>>>>>>> STONITH--Node1: success:  0
>>>>>>>>> Apr 15 22:01:21 xencluster2 stonith-ng: [8283]: info:
>>>>>>>>> stonith_device_execute: Nothing to do for STONITH--Node1
>>>>>>>>> Apr 15 22:01:21 xencluster2 stonith-ng: [8283]: info:
>>>>>>>>> stonith_command:
>>>>>>>>> Processed st_execute from lrmd: rc=-1
>>>>>>>>> Apr 15 22:01:21 xencluster2 stonith: parse config info
>>>>> info=xencluster2
>>>>>>>>> Apr 15 22:01:21 xencluster2 stonith: meatware device OK.
>>>>>>>>> Apr 15 22:01:21 xencluster2 stonith-ng: [8283]: info: log_operation:
>>>>>>>>> STONITH-Node2: Performing: stonith -t meatware -S
>>>>>>>>> Apr 15 22:01:21 xencluster2 stonith-ng: [8283]: info: log_operation:
>>>>>>>>> STONITH-Node2: success:  0
>>>>>>>>> Apr 15 22:01:21 xencluster2 stonith-ng: [8283]: info:
>>>>>>>>> stonith_device_execute: Nothing to do for STONITH-Node2
>>>>>>>>> Apr 15 22:01:21 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> Apr 15 22:01:22 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> Apr 15 22:01:22 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> ....
>>>>>>>>> Apr 15 22:01:36 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> Apr 15 22:01:36 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> Apr 15 22:01:36 xencluster2 stonith-ng: [8283]: info:
>>>>>>>>> stonith_command:
>>>>>>>>> Processed st_execute from lrmd: rc=-1
>>>>>>>>> Apr 15 22:01:36 xencluster2 stonith: parse config info
>>>>> info=xencluster1
>>>>>>>>> Apr 15 22:01:36 xencluster2 stonith: meatware device OK.
>>>>>>>>> Apr 15 22:01:36 xencluster2 stonith-ng: [8283]: info: log_operation:
>>>>>>>>> STONITH--Node1: Performing: stonith -t meatware -S
>>>>>>>>> Apr 15 22:01:36 xencluster2 stonith-ng: [8283]: info: log_operation:
>>>>>>>>> STONITH--Node1: success:  0
>>>>>>>>> Apr 15 22:01:36 xencluster2 stonith-ng: [8283]: info:
>>>>>>>>> stonith_device_execute: Nothing to do for STONITH--Node1
>>>>>>>>> Apr 15 22:01:36 xencluster2 stonith-ng: [8283]: info:
>>>>>>>>> stonith_command:
>>>>>>>>> Processed st_execute from lrmd: rc=-1
>>>>>>>>> Apr 15 22:01:36 xencluster2 stonith: parse config info
>>>>> info=xencluster2
>>>>>>>>> Apr 15 22:01:36 xencluster2 stonith: meatware device OK.
>>>>>>>>> Apr 15 22:01:36 xencluster2 stonith-ng: [8283]: info: log_operation:
>>>>>>>>> STONITH-Node2: Performing: stonith -t meatware -S
>>>>>>>>> Apr 15 22:01:36 xencluster2 stonith-ng: [8283]: info: log_operation:
>>>>>>>>> STONITH-Node2: success:  0
>>>>>>>>> Apr 15 22:01:36 xencluster2 stonith-ng: [8283]: info:
>>>>>>>>> stonith_device_execute: Nothing to do for STONITH-Node2
>>>>>>>>> Apr 15 22:01:36 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> Apr 15 22:01:37 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> ....
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: process_graph_event:
>>>>>>>>> Detected action clvm-xenvg:0_monitor_10000 from a different
>>>>> transition:
>>>>>>>>> 62 vs. 80
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info:
>>>>> abort_transition_graph:
>>>>>>>>> process_graph_event:476 - Triggered transition abort (complete=1,
>>>>>>>>> tag=lrm_rsc_op, id=clvm-xenvg:0_monitor_10000,
>>>>>>>>> magic=2:-2;61:62:0:72497771-fdb2-4ca7-a723-2cc360861bb0,
>>>>> cib=0.5816.4) :
>>>>>>>>> Old event
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: WARN: update_failcount:
>>>>>>>>> Updating failcount for clvm-xenvg:0 on xencluster1 after failed
>>>>> monitor:
>>>>>>>>> rc=-2 (update=value++, time=1334520100)
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: do_state_transition:
>>>>>>>>> State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
>>>>>>>>> cause=C_FSA_INTERNAL origin=abort_transition_graph ]
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: do_state_transition:
>>>>> All
>>>>>>>>> 2 cluster nodes are eligible to run resources.
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: do_pe_invoke: Query
>>>>> 984:
>>>>>>>>> Requesting the current CIB: S_POLICY_ENGINE
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197 219a 219b 219c 219d
>>>>>>>>> ...
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197 219a 219b 219c 219d
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info:
>>>>>>>>> do_pe_invoke_callback:
>>>>>>>>> Invoking the PE: query=984, ref=pe_calc-dc-1334520100-605, seq=2212,
>>>>>>>>> quorate=1
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197 219a 219b 219c 219d
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197 219a 219b 219c 219d
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_config:
>>>>>>>>> On
>>>>>>>>> loss of CCM Quorum: Ignore
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation clvm:1_last_failure_0 found resource clvm:1 active on
>>>>>>> xencluster2
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation STONITH-Node2_last_failure_0 found resource STONITH-Node2
>>>>>>>>> active on xencluster2
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation STONITH--Node1_last_failure_0 found resource
>>>>>>>>> STONITH--Node1
>>>>>>>>> active on xencluster2
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation cmirror:1_last_failure_0 found resource cmirror:1 active
>>>>>>>>> on
>>>>>>>>> xencluster2
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation dlm:1_last_failure_0 found resource dlm:1 active on
>>>>>>> xencluster2
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation o2cb:1_last_failure_0 found resource o2cb:1 active on
>>>>>>> xencluster2
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation cmirror:0_last_failure_0 found resource cmirror:0 active
>>>>>>>>> on
>>>>>>>>> xencluster1
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation dlm:0_last_failure_0 found resource dlm:0 active on
>>>>>>> xencluster1
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation clvm:0_last_failure_0 found resource clvm:0 active on
>>>>>>> xencluster1
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation o2cb:0_last_failure_0 found resource o2cb:0 active on
>>>>>>> xencluster1
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>>>>>>>>> Operation clvm-xenvg:0_last_failure_0 found resource clvm-xenvg:0
>>>>> active
>>>>>>>>> on xencluster1
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: WARN: unpack_rsc_op:
>>>>>>>>> Processing failed op clvm-xenvg:0_monitor_10000 on xencluster1:
>>>>> unknown
>>>>>>>>> exec error (-2)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: WARN: unpack_rsc_op:
>>>>>>>>> Processing failed op vm-deprepo_last_failure_0 on xencluster1:
>>>>>>>>> unknown
>>>>>>>>> error (1)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: RecurringOp:
>>>>>  Start
>>>>>>>>> recurring monitor (10s) for clvm-xenvg:0 on xencluster1
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  STONITH-Node2  (Started xencluster2)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  STONITH--Node1 (Started xencluster2)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  dlm:0  (Started xencluster1)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  clvm:0 (Started xencluster1)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  o2cb:0 (Started xencluster1)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  cmirror:0      (Started xencluster1)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  dlm:1  (Started xencluster2)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  clvm:1 (Started xencluster2)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  o2cb:1 (Started xencluster2)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  cmirror:1      (Started xencluster2)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>> Recover
>>>>>>>>> clvm-xenvg:0   (Started xencluster1)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>> Restart
>>>>>>>>> fs-config-xen:0        (Started xencluster1)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  clvm-xenvg:1   (Started xencluster2)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  fs-config-xen:1        (Started xencluster2)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>> Restart
>>>>>>>>> vm-ilmt        (Started xencluster1)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>> Restart
>>>>>>>>> vm-topdesk-entw        (Started xencluster1)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  vm-otbridge    (Started xencluster2)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>> Restart
>>>>>>>>> vm-virenscanner        (Started xencluster1)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>> Restart
>>>>>>>>> vm-deprepo     (Started xencluster1)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  vm-traumschiff (Started xencluster2)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>>>>>> Leave
>>>>>>>>>  vm-cim (Started xencluster2)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>> Restart
>>>>>>>>> vm-oneiroi     (Started xencluster1)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>> Restart
>>>>>>>>> vm-intranet    (Started xencluster1)
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>>>>> Restart
>>>>>>>>> vm-soa (Started xencluster1)
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: do_state_transition:
>>>>>>>>> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
>>>>>>>>> input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: unpack_graph:
>>>>>>>>> Unpacked
>>>>>>>>> transition 81: 39 actions in 39 synapses
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: do_te_invoke:
>>>>> Processing
>>>>>>>>> graph 81 (ref=pe_calc-dc-1334520100-605) derived from
>>>>>>>>> /var/lib/pengine/pe-input-272.bz2
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: te_rsc_command:
>>>>>>>>> Initiating action 82: stop vm-ilmt_stop_0 on xencluster1
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: te_rsc_command:
>>>>>>>>> Initiating action 86: stop vm-topdesk-entw_stop_0 on xencluster1
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: te_rsc_command:
>>>>>>>>> Initiating action 90: stop vm-virenscanner_stop_0 on xencluster1
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: te_rsc_command:
>>>>>>>>> Initiating action 92: stop vm-deprepo_stop_0 on xencluster1
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: te_rsc_command:
>>>>>>>>> Initiating action 98: stop vm-oneiroi_stop_0 on xencluster1
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: te_rsc_command:
>>>>>>>>> Initiating action 100: stop vm-intranet_stop_0 on xencluster1
>>>>>>>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: te_rsc_command:
>>>>>>>>> Initiating action 102: stop vm-soa_stop_0 on xencluster1
>>>>>>>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice:
>>>>> process_pe_message:
>>>>>>>>> Transition 81: PEngine Input stored in:
>>>>>>> /var/lib/pengine/pe-input-272.bz2
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197 219a 219b 219c 219d
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197 219a 219b 219c 219d
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197 219a 219b 219c 219d
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197 219a 219b 219c 219d
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197 219a 219b 219c 219d
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197 219a 219b 219c 219d 219e 219f 21a0 21a1 21a2 21a3 21a4
>>>>>>>>> 21a5
>>>>>>>>> Apr 15 22:01:40 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197 219a 219b 219c 219d 219e 219f 21a0 21a1 21a2 21a3 21a4
>>>>>>>>> 21a5
>>>>>>>>> ...
>>>>>>>>> Apr 15 22:01:41 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197 219a 219b 219c 219d 219e 219f 21a0 21a1 21a2 21a3 21a4
>>>>>>>>> 21a5
>>>>>>>>> Apr 15 22:01:42 xencluster2 lrmd: [7675]: WARN: clvm-xenvg:1:monitor
>>>>>>>>> process (PID 29785) timed out (try 1).  Killing with signal SIGTERM
>>>>>>> (15).
>>>>>>>>> Apr 15 22:01:42 xencluster2 lrmd: [7675]: WARN: operation
>>>>>>>>> monitor[125]
>>>>>>>>> on clvm-xenvg:1 for client 7678: pid 29785 timed out
>>>>>>>>> Apr 15 22:01:42 xencluster2 crmd: [7678]: ERROR: process_lrm_event:
>>>>> LRM
>>>>>>>>> operation clvm-xenvg:1_monitor_10000 (125) Timed Out
>>>>>>>>> (timeout=30000ms)
>>>>>>>>> Apr 15 22:01:42 xencluster2 crmd: [7678]: info: process_graph_event:
>>>>>>>>> Detected action clvm-xenvg:1_monitor_10000 from a different
>>>>> transition:
>>>>>>>>> 68 vs. 81
>>>>>>>>> Apr 15 22:01:42 xencluster2 crmd: [7678]: info:
>>>>> abort_transition_graph:
>>>>>>>>> process_graph_event:476 - Triggered transition abort (complete=0,
>>>>>>>>> tag=lrm_rsc_op, id=clvm-xenvg:1_monitor_10000,
>>>>>>>>> magic=2:-2;61:68:0:72497771-fdb2-4ca7-a723-2cc360861bb0,
>>>>> cib=0.5816.5) :
>>>>>>>>> Old event
>>>>>>>>> Apr 15 22:01:42 xencluster2 crmd: [7678]: info:
>>>>>>>>> update_abort_priority:
>>>>>>>>> Abort priority upgraded from 0 to 1000000
>>>>>>>>> Apr 15 22:01:42 xencluster2 crmd: [7678]: info:
>>>>>>>>> update_abort_priority:
>>>>>>>>> Abort action done superceeded by restart
>>>>>>>>> Apr 15 22:01:42 xencluster2 crmd: [7678]: WARN: update_failcount:
>>>>>>>>> Updating failcount for clvm-xenvg:1 on xencluster2 after failed
>>>>> monitor:
>>>>>>>>> rc=-2 (update=value++, time=1334520102)
>>>>>>>>> Apr 15 22:01:42 xencluster2 attrd: [7676]: notice:
>>>>> attrd_trigger_update:
>>>>>>>>> Sending flush op to all hosts for: fail-count-clvm-xenvg:1 (1)
>>>>>>>>> Apr 15 22:01:42 xencluster2 attrd: [7676]: notice:
>>>>> attrd_perform_update:
>>>>>>>>> Sent update 448: fail-count-clvm-xenvg:1=1
>>>>>>>>> Apr 15 22:01:42 xencluster2 attrd: [7676]: notice:
>>>>> attrd_trigger_update:
>>>>>>>>> Sending flush op to all hosts for: last-failure-clvm-xenvg:1
>>>>>>> (1334520102)
>>>>>>>>> Apr 15 22:01:42 xencluster2 crmd: [7678]: info:
>>>>> abort_transition_graph:
>>>>>>>>> te_update_diff:176 - Triggered transition abort (complete=0,
>>>>> tag=nvpair,
>>>>>>>>> id=status-xencluster2-fail-count-clvm-xenvg.1,
>>>>>>>>> name=fail-count-clvm-xenvg:1, value=1, magic=NA, cib=0.5816.6) :
>>>>>>>>> Transient attribute: update
>>>>>>>>> Apr 15 22:01:42 xencluster2 attrd: [7676]: notice:
>>>>> attrd_perform_update:
>>>>>>>>> Sent update 451: last-failure-clvm-xenvg:1=1334520102
>>>>>>>>> Apr 15 22:01:42 xencluster2 crmd: [7678]: info:
>>>>> abort_transition_graph:
>>>>>>>>> te_update_diff:176 - Triggered transition abort (complete=0,
>>>>> tag=nvpair,
>>>>>>>>> id=status-xencluster2-last-failure-clvm-xenvg.1,
>>>>>>>>> name=last-failure-clvm-xenvg:1, value=1334520102, magic=NA,
>>>>>>>>> cib=0.5816.7) : Transient attribute: update
>>>>>>>>> Apr 15 22:01:42 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197 219a 219b 219c 219d 219e 219f 21a0 21a1 21a2 21a3 21a4
>>>>>>>>> 21a5
>>>>>>>>> Apr 15 22:01:42 xencluster2 corosync[7666]:   [TOTEM ] Retransmit
>>>>> List:
>>>>>>>>> 2196 2197 219a 219b 219c 219d 219e 219f 21a0 21a1 21a2 21a3 21a4
>>>>>>>>> 21a5
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Mit freundlichen Gr??en
>>>>>>>>>
>>>>>>>>> Frank Meier
>>>>>>>>> UNIX-Basis
>>>>>>>>>
>>>>>>>>> Hamm Reno Group GmbH
>>>>>>>>> Industriegebiet West | D-66987 Thaleischweiler-Fr?schen
>>>>>>>>> T.+49(0)6334 444-8322 | F.+49(0)6334 444-8190
>>>>>>>>> frank.meier at hr-group.de | www.reno.de
>>>>>>>>> ___________________________________________________________________
>>>>>>>>>
>>>>>>>>> Sitz: Am Tie 7 | D-49086 Osnabr?ck
>>>>>>>>> Handelsregister Osnabr?ck HRB 19587
>>>>>>>>> Gesch?ftsf?hrer: Hans-J?rgen de Fries,
>>>>>>>>> Jens Gransee, Manfred Klumpp,
>>>>>>>>> Robert Reisch
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>>>
>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>> Getting started:
>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>> -------------- next part --------------
>>>>>>>> An HTML attachment was scrubbed...
>>>>>>>> URL: <
>>>>>>>
>>>>> http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20120419/00a36dbe/attachment.html
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Pacemaker mailing list
>>>>>>>> Pacemaker at oss.clusterlabs.org
>>>>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>>
>>>>>>>>
>>>>>>>> End of Pacemaker Digest, Vol 53, Issue 38
>>>>>>>> *****************************************
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>
>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>> Getting started:
>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>> -------------- next part --------------
>>>>>> An HTML attachment was scrubbed...
>>>>>> URL: <
>>>>> http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20120419/63b888a3/attachment.html
>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>>
>>>>>> _______________________________________________
>>>>>> Pacemaker mailing list
>>>>>> Pacemaker at oss.clusterlabs.org
>>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>
>>>>>>
>>>>>> End of Pacemaker Digest, Vol 53, Issue 40
>>>>>> *****************************************
>>>>>
>>>>> _______________________________________________
>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>
>>>>> Project Home: http://www.clusterlabs.org
>>>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>> -------------- next part --------------
>>>> An HTML attachment was scrubbed...
>>>> URL:
>>>> <http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20120419/f21bf7c7/attachment.html>
>>>>
>>>> ------------------------------
>>>>
>>>> _______________________________________________
>>>> Pacemaker mailing list
>>>> Pacemaker at oss.clusterlabs.org
>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>
>>>>
>>>> End of Pacemaker Digest, Vol 53, Issue 42
>>>> *****************************************
>>>>
>>>> _______________________________________________
>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>
>>>> Project Home: http://www.clusterlabs.org
>>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>> Bugs: http://bugs.clusterlabs.org
>>>>
>>>
>>>
>>> --
>>> esta es mi vida e me la vivo hasta que dios quiera
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> _______________________________________________
>>> Pacemaker mailing list
>>> Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>>
>>> End of Pacemaker Digest, Vol 53, Issue 63
>>> *****************************************
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
> 
> 
> --
> esta es mi vida e me la vivo hasta que dios quiera
> 
> 
> 
> ------------------------------
> 
> _______________________________________________
> Pacemaker mailing list
> Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> 
> End of Pacemaker Digest, Vol 53, Issue 65
> *****************************************




More information about the Pacemaker mailing list