[Pacemaker] LVM restarts after SLES upgrade
emmanuel segura
emi2fast at gmail.com
Tue Apr 24 05:34:12 EDT 2012
Hello Frank
Maybe this it's not the probelem, but i see this constrain wrong from
my point of view
=============================================================
order o-Testclustervm inf: c-xen-vg-fs vm-clusterTest
order o-clvmglue-xenvgfs inf: c-clvm-glue c-xen-vg-fs
=============================================================
to be
=============================================================
order o-clvmglue-xenvgfs inf: c-clvm-glue c-xen-vg-fs
order o-Testclustervm inf: c-xen-vg-fs vm-clusterTest
=============================================================
2012/4/24, Frank Meier <frank.meier at hr-group.de>:
> Hi,
>
> Every time the vgdisplay -v TestXenVG is hanging(ca.2min)
>
> I see two of this peocesses:
> /bin/sh /usr/lib/ocf/resource.d//heartbeat/LVM monitor
> /bin/sh /usr/lib/ocf/resource.d//heartbeat/LVM monitor
> is this OK, or have we a race condition?
>
>
>
> Hi,
>
> I do a new try.
>
> I've moved our productive VMs to another server and reconfigure the
> cluster. But the problem is the same: every hour(or shorter, it seemd to
> be full, half or quarter hour), the LVM is restarted.
> Last I saw vgdisplay -v TestXenVG hanging, I don't if this was every time.
>
> I hope anybody can help. It has to be run expressly.
>
> Thanks
>
> Frank
>
>
> vgs
> VG #PV #LV #SN Attr VSize VFree
> TestXenVG 2 2 0 wz--nc 67.99g 61.80g
>
> xencluster1:~ # ps aux|grep clvm
> root 21529 0.0 0.0 4520 528 pts/2 S+ 17:36 0:00 grep clvm
> root 21756 0.0 0.1 155348 62920 ? SLsl 13:44 0:07
> /usr/sbin/clvmd -d0
>
>
> xencluster1:~ # time vgdisplay -v TestXenVG
> Using volume group(s) on command line
> Finding volume group "TestXenVG"
> --- Volume group ---
> VG Name TestXenVG
> System ID
> Format lvm2
> Metadata Areas 2
> Metadata Sequence No 6
> VG Access read/write
> VG Status resizable
> Clustered yes
> Shared no
> MAX LV 0
> Cur LV 2
> Open LV 2
> Max PV 0
> Cur PV 2
> Act PV 2
> VG Size 67.99 GiB
> PE Size 4.00 MiB
> Total PE 17406
> Alloc PE / Size 1586 / 6.20 GiB
> Free PE / Size 15820 / 61.80 GiB
> VG UUID hveP8R-8pXh-b9QB-OGKG-ZmPa-xXXQ-f3MYGx
>
> --- Logical volume ---
> LV Name /dev/TestXenVG/vm1
> VG Name TestXenVG
> LV UUID W01y7D-7WfV-bBiU-pmnm-Tecg-ecrb-qjJ7GJ
> LV Write Access read/write
> LV Status available
> # open 1
> LV Size 6.00 GiB
> Current LE 1536
> Segments 1
> Allocation inherit
> Read ahead sectors auto
> - currently set to 1024
> Block device 253:4
>
> --- Logical volume ---
> LV Name /dev/TestXenVG/xen_config
> VG Name TestXenVG
> LV UUID lrEjLC-yrlv-Kddh-JIb8-A1pT-t6b1-W4dw3f
> LV Write Access read/write
> LV Status available
> # open 1
> LV Size 200.00 MiB
> Current LE 50
> Segments 1
> Allocation inherit
> Read ahead sectors auto
> - currently set to 1024
> Block device 253:5
>
> --- Physical volumes ---
> PV Name /dev/mapper/mpathi
> PV UUID N7NE4R-v1ji-23cS-NzIn-v6Zt-S3Rp-HlB5jS
> PV Status allocatable
> Total PE / Free PE 8703 / 7117
>
> PV Name /dev/mapper/mpathj
> PV UUID cR567j-PM7y-5NUN-SGsJ-JO2h-rn0e-jTk7sm
> PV Status allocatable
> Total PE / Free PE 8703 / 8703
>
>
> real 0m0.033s
> user 0m0.004s
> sys 0m0.016s
>
> CIB:
>
> node xencluster1 \
> attributes standby="off"
> node xencluster2 \
> attributes standby="off"
> primitive STONITH-Node1 stonith:meatware \
> meta target-role="Started" is-managed="true" \
> operations $id="STONITH-Node1-operations" \
> op monitor interval="3600" timeout="20" start-delay="15" \
> params hostlist="xencluster1"
> primitive STONITH-Node2 stonith:meatware \
> meta target-role="Started" is-managed="true" \
> operations $id="STONITH-Node2-operations" \
> op monitor interval="3600" timeout="20" start-delay="15" \
> params hostlist="xencluster2"
> primitive clvm ocf:lvm2:clvmd \
> operations $id="clvm-operations" \
> op monitor interval="10" timeout="20" \
> op start interval="0" timeout="90" \
> op stop interval="0" timeout="100"
> primitive clvm-xenvg ocf:heartbeat:LVM \
> operations $id="clvm-xenvg-operations" \
> op monitor interval="10" timeout="90" \
> op start interval="0" timeout="60" \
> op stop interval="0" timeout="60" \
> params volgrpname="TestXenVG" \
> meta target-role="Started"
> primitive cmirror ocf:lvm2:cmirrord \
> operations $id="cmirror-operations" \
> op monitor interval="10" timeout="20" \
> op start interval="0" timeout="90" \
> op stop interval="0" timeout="100" \
> params daemon_timeout="30"
> primitive dlm ocf:pacemaker:controld \
> operations $id="dlm-operations" \
> op monitor interval="60" timeout="60" start-delay="0" \
> op start interval="0" timeout="90" \
> op stop interval="0" timeout="100"
> primitive fs-config-xen ocf:heartbeat:Filesystem \
> operations $id="fs-config-xen-operations" \
> op monitor interval="20" timeout="40" \
> params device="/dev/mapper/TestXenVG-xen_config"
> directory="/etc/xen/vm" fstype="ocfs2"
> primitive o2cb ocf:ocfs2:o2cb \
> operations $id="o2cb-operations" \
> op monitor interval="60" timeout="60" \
> op start interval="0" timeout="90" \
> op stop interval="0" timeout="100"
> primitive vm-clusterTest ocf:heartbeat:Xen \
> meta target-role="Started" allow-migrate="true" is-managed="true" \
> operations $id="vm-clusterTest-operations" \
> op monitor interval="10" timeout="30" \
> op start interval="0" timeout="60" \
> op stop interval="0" timeout="60" \
> op migrate_to interval="0" timeout="360" \
> params xmfile="/etc/xen/vm/ClusterTest"
> group clvm-glue dlm clvm o2cb cmirror
> group xen-vg-fs clvm-xenvg fs-config-xen
> clone c-clvm-glue clvm-glue \
> meta target-role="Started" interleave="true" ordered="true"
> clone c-xen-vg-fs xen-vg-fs \
> meta target-role="Started" interleave="true" ordered="true"
> is-managed="true"
> location STONITHnode1Constraint STONITH-Node1 -inf: xencluster1
> location STONITHnode2Constraint STONITH-Node2 -inf: xencluster2
> colocation colo-clvmglue-xenvgsfs inf: c-xen-vg-fs c-clvm-glue
> order o-Testclustervm inf: c-xen-vg-fs vm-clusterTest
> order o-clvmglue-xenvgfs inf: c-clvm-glue c-xen-vg-fs
> property $id="cib-bootstrap-options" \
> dc-version="1.1.6-b988976485d15cb702c9307df55512d323831a5e" \
> no-quorum-policy="ignore" \
> default-resource-stickiness="100000" \
> last-lrm-refresh="1335186478" \
> cluster-infrastructure="openais" \
> expected-quorum-votes="2" \
> default-action-timeout="120s" \
> maintenance-mode="false"
> op_defaults $id="op_defaults-options" \
> record-pending="false"
>
>
>
> Log:
> Apr 23 17:14:29 xencluster1 clvmd[21756]: Syncing device names
> Apr 23 17:14:29 xencluster1 clvmd[21756]: Reply from node 4914420a: 0 bytes
> Apr 23 17:14:29 xencluster1 clvmd[21756]: Got 1 replies, expecting: 2
> Apr 23 17:14:29 xencluster1 clvmd[21756]: LVM thread waiting for work
> Apr 23 17:15:59 xencluster1 lrmd: [8619]: WARN: clvm-xenvg:0:monitor
> process (PID 8351) timed out (try 1). Killing with signal SIGTERM (15).
> Apr 23 17:15:59 xencluster1 lrmd: [8619]: WARN: operation monitor[339]
> on clvm-xenvg:0 for client 8622: pid 8351 timed out
> Apr 23 17:15:59 xencluster1 crmd: [8622]: ERROR: process_lrm_event: LRM
> operation clvm-xenvg:0_monitor_10000 (339) Timed Out (timeout=90000ms)
> pr 23 17:15:59 xencluster1 crmd: [8622]: info: process_graph_event:
> Action clvm-xenvg:0_monitor_10000 arrived after a completed transition
> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: abort_transition_graph:
> process_graph_event:481 - Triggered transition abort (complete=1,
> tag=lrm_rsc_op, id=clvm-xenvg:0_monitor_10000,
> magic=2:-2;12:384:0:a2db503d-6be0-4ed9-a94c-0e7031c748ad, cib=0.8361.49)
> : Inactive graph
> Apr 23 17:15:59 xencluster1 crmd: [8622]: WARN: update_failcount:
> Updating failcount for clvm-xenvg:0 on xencluster1 after failed monitor:
> rc=-2 (update=value++, time=1335194159)
> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: do_state_transition:
> State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_FSA_INTERNAL origin=abort_transition_graph ]
> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: do_state_transition: All
> 2 cluster nodes are eligible to run resources.
> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: do_pe_invoke: Query
> 2608: Requesting the current CIB: S_POLICY_ENGINE
> Apr 23 17:15:59 xencluster1 attrd: [8620]: notice: attrd_trigger_update:
> Sending flush op to all hosts for: fail-count-clvm-xenvg:0 (8)
> Apr 23 17:15:59 xencluster1 attrd: [8620]: notice: attrd_perform_update:
> Sent update 1231: fail-count-clvm-xenvg:0=8
> Apr 23 17:15:59 xencluster1 attrd: [8620]: notice: attrd_trigger_update:
> Sending flush op to all hosts for: last-failure-clvm-xenvg:0 (1335194159)
> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: do_pe_invoke_callback:
> Invoking the PE: query=2608, ref=pe_calc-dc-1335194159-1765, seq=2556,
> quorate=1
> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: abort_transition_graph:
> te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair,
> id=status-xencluster1-fail-count-clvm-xenvg.0,
> name=fail-count-clvm-xenvg:0, value=8, magic=NA, cib=0.8361.50) :
> Transient attribute: update
> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: do_pe_invoke: Query
> 2609: Requesting the current CIB: S_POLICY_ENGINE
> Apr 23 17:15:59 xencluster1 attrd: [8620]: notice: attrd_perform_update:
> Sent update 1233: last-failure-clvm-xenvg:0=1335194159
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_config: On
> loss of CCM Quorum: Ignore
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
> Operation STONITH-Node1_last_failure_0 found resource STONITH-Node1
> active on xencluster2
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
> Operation dlm:1_last_failure_0 found resource dlm:1 active on xencluster2
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
> Operation clvm:1_last_failure_0 found resource clvm:1 active on xencluster2
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
> Operation cmirror:1_last_failure_0 found resource cmirror:1 active on
> xencluster2
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
> Operation clvm:0_last_failure_0 found resource clvm:0 active on xencluster1
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
> Operation STONITH-Node2_last_failure_0 found resource STONITH-Node2
> active on xencluster1
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
> Operation dlm:0_last_failure_0 found resource dlm:0 active on xencluster1
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
> Operation clvm-xenvg:0_last_failure_0 found resource clvm-xenvg:0 active
> on xencluster1
> Apr 23 17:15:59 xencluster1 pengine: [8621]: WARN: unpack_rsc_op:
> Processing failed op clvm-xenvg:0_monitor_10000 on xencluster1: unknown
> exec error (-2)
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
> Operation cmirror:0_last_failure_0 found resource cmirror:0 active on
> xencluster1
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: unpack_rsc_op:
> Operation o2cb:0_last_failure_0 found resource o2cb:0 active on xencluster1
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice:
> common_apply_stickiness: c-xen-vg-fs can fail 999993 more times on
> xencluster1 before being forced off
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice:
> common_apply_stickiness: c-xen-vg-fs can fail 999993 more times on
> xencluster1 before being forced off
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice:
> common_apply_stickiness: c-xen-vg-fs can fail 999993 more times on
> xencluster2 before being forced off
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice:
> common_apply_stickiness: c-xen-vg-fs can fail 999993 more times on
> xencluster2 before being forced off
> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: do_pe_invoke_callback:
> Invoking the PE: query=2609, ref=pe_calc-dc-1335194159-1766, seq=2556,
> quorate=1
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: RecurringOp: Start
> recurring monitor (10s) for clvm-xenvg:0 on xencluster1
> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: abort_transition_graph:
> te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair,
> id=status-xencluster1-last-failure-clvm-xenvg.0,
> name=last-failure-clvm-xenvg:0, value=1335194159, magic=NA,
> cib=0.8361.51) : Transient attribute: update
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: LogActions: Recover
> clvm-xenvg:0 (Started xencluster1)
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: LogActions: Restart
> fs-config-xen:0 (Started xencluster1)
> Apr 23 17:15:59 xencluster1 crmd: [8622]: info: do_pe_invoke: Query
> 2610: Requesting the current CIB: S_POLICY_ENGINE
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: LogActions: Leave
> clvm-xenvg:1 (Started xencluster2)
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: LogActions: Leave
> fs-config-xen:1 (Started xencluster2)
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: LogActions: Leave
> STONITH-Node1 (Started xencluster2)
> Apr 23 17:15:59 xencluster1 pengine: [8621]: notice: LogActions: Leave
> STONITH-Node2 (Started xencluster1)
>
>
>
>
>
>
> Mit freundlichen Grüßen
>
> Frank Meier
> UNIX-Basis
>
> Hamm Reno Group GmbH
> Industriegebiet West | D-66987 Thaleischweiler-Fröschen
> T.+49(0)6334 444-322 | F.+49(0)6334 444-190
> frank.meier at hr-group.de | www.my-reno.com
> ___________________________________________________________________
>
> Sitz: Am Tie 7 | D-49086 Osnabrück
> Handelsregister Osnabrück HRB 19587
> Geschäftsführer: Hans-Jürgen de Fries,
> Jens Gransee, Manfred Klumpp,
> Robert Reisch, Albert Scherer
> ________________________________________
> Von: pacemaker-request at oss.clusterlabs.org
> [pacemaker-request at oss.clusterlabs.org]
> Gesendet: Donnerstag, 19. April 2012 10:07
> An: pacemaker at oss.clusterlabs.org
> Betreff: Pacemaker Digest, Vol 53, Issue 42
>
> Send Pacemaker mailing list submissions to
> pacemaker at oss.clusterlabs.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> or, via email, send a message with subject or body 'help' to
> pacemaker-request at oss.clusterlabs.org
>
> You can reach the person managing the list at
> pacemaker-owner at oss.clusterlabs.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Pacemaker digest..."
>
>
> Today's Topics:
>
> 1. Re: Pacemaker Digest, Vol 53, Issue 40 (emmanuel segura)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 19 Apr 2012 10:12:37 +0200
> From: emmanuel segura <emi2fast at gmail.com>
> To: The Pacemaker cluster resource manager
> <pacemaker at oss.clusterlabs.org>
> Subject: Re: [Pacemaker] Pacemaker Digest, Vol 53, Issue 40
> Message-ID:
> <CAE7pJ3B5L1m2s=OQ=cUDd9=mqy50ORhP+eiDteCRp5_6JLR6jQ at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> vgchange -cy XenVG
>
> your vg it's not clustered and check the clvmd process
>
> ps -ef | grep clvmd
>
> Il giorno 19 aprile 2012 09:45, Frank Meier <frank.meier at hr-group.de> ha
> scritto:
>
>> xencluster1:~ # vgs
>> VG #PV #LV #SN Attr VSize VFree
>> XenVG 8 32 0 wz--n- 938.16g 530.98g
>>
>>
>> Mit freundlichen Gr??en
>>
>> Frank Meier
>> UNIX-Basis
>>
>> Hamm Reno Group GmbH
>> Industriegebiet West | D-66987 Thaleischweiler-Fr?schen
>> T.+49(0)6334 444-8322 | F.+49(0)6334 444-8190
>> frank.meier at hr-group.de | www.reno.de
>> ___________________________________________________________________
>>
>> Sitz: Am Tie 7 | D-49086 Osnabr?ck
>> Handelsregister Osnabr?ck HRB 19587
>> Gesch?ftsf?hrer: Hans-J?rgen de Fries,
>> Jens Gransee, Manfred Klumpp,
>> Robert Reisch
>>
>> Am 19.04.2012 09:32, schrieb pacemaker-request at oss.clusterlabs.org:
>> > Send Pacemaker mailing list submissions to
>> > pacemaker at oss.clusterlabs.org
>> >
>> > To subscribe or unsubscribe via the World Wide Web, visit
>> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> > or, via email, send a message with subject or body 'help' to
>> > pacemaker-request at oss.clusterlabs.org
>> >
>> > You can reach the person managing the list at
>> > pacemaker-owner at oss.clusterlabs.org
>> >
>> > When replying, please edit your Subject line so it is more specific
>> > than "Re: Contents of Pacemaker digest..."
>> >
>> >
>> > Today's Topics:
>> >
>> > 1. Re: Pacemaker Digest, Vol 53, Issue 38 (emmanuel segura)
>> >
>> >
>> > ----------------------------------------------------------------------
>> >
>> > Message: 1
>> > Date: Thu, 19 Apr 2012 09:37:31 +0200
>> > From: emmanuel segura <emi2fast at gmail.com>
>> > To: The Pacemaker cluster resource manager
>> > <pacemaker at oss.clusterlabs.org>
>> > Subject: Re: [Pacemaker] Pacemaker Digest, Vol 53, Issue 38
>> > Message-ID:
>> > <CAE7pJ3CovBAsjf+vbbJGEKuqTTvFiWWpn0S=UvOTSoyQpeTa=
>> w at mail.gmail.com>
>> > Content-Type: text/plain; charset="iso-8859-1"
>> >
>> > give this command
>> >
>> > vgs
>> >
>> > Il giorno 19 aprile 2012 09:32, Frank Meier <frank.meier at hr-group.de> ha
>> > scritto:
>> >
>> >> Hi,
>> >>
>> >> here my lvm.conf:
>> >>
>> >> devices {
>> >> dir = "/dev"
>> >> scan = [ "/dev" ]
>> >> filter = [ "a|/dev/mapper|", "r/.*/" ]
>> >> cache = "/etc/lvm/.cache"
>> >> write_cache_state = 1
>> >> types = [ "device-mapper", 16 ]
>> >> sysfs_scan = 1
>> >> md_component_detection = 1
>> >> }
>> >> log {
>> >> verbose = 0
>> >> syslog = 1
>> >> overwrite = 0
>> >> level = 0
>> >> indent = 1
>> >> command_names = 0
>> >> prefix = " "
>> >> }
>> >> backup {
>> >> backup = 1
>> >> backup_dir = "/etc/lvm/backup"
>> >> archive = 1
>> >> archive_dir = "/etc/lvm/archive"
>> >> retain_min = 10
>> >> retain_days = 30
>> >> }
>> >> shell {
>> >> history_size = 100
>> >> }
>> >> global {
>> >> umask = 077
>> >> test = 0
>> >> activation = 1
>> >> proc = "/proc"
>> >> locking_type = 3
>> >> locking_dir = "/var/lock/lvm"
>> >> }
>> >> activation {
>> >> missing_stripe_filler = "error"
>> >> mirror_region_size = 512
>> >> reserved_stack = 256
>> >> reserved_memory = 8192
>> >> process_priority = -18
>> >> }
>> >> dmeventd {
>> >> mirror_library = "libdevmapper-event-lvm2mirror.so.2.02"
>> >> snapshot_library = "libdevmapper-event-lvm2snapshot.so.2.02"
>> >> }
>> >>
>> >>
>> >> Mit freundlichen Gr??en
>> >>
>> >> Frank Meier
>> >> UNIX-Basis
>> >>
>> >> Hamm Reno Group GmbH
>> >> Industriegebiet West | D-66987 Thaleischweiler-Fr?schen
>> >> T.+49(0)6334 444-8322 | F.+49(0)6334 444-8190
>> >> frank.meier at hr-group.de | www.reno.de
>> >> ___________________________________________________________________
>> >>
>> >> Sitz: Am Tie 7 | D-49086 Osnabr?ck
>> >> Handelsregister Osnabr?ck HRB 19587
>> >> Gesch?ftsf?hrer: Hans-J?rgen de Fries,
>> >> Jens Gransee, Manfred Klumpp,
>> >> Robert Reisch
>> >>
>> >> Am 19.04.2012 09:13, schrieb pacemaker-request at oss.clusterlabs.org:
>> >>> Send Pacemaker mailing list submissions to
>> >>> pacemaker at oss.clusterlabs.org
>> >>>
>> >>> To subscribe or unsubscribe via the World Wide Web, visit
>> >>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> >>> or, via email, send a message with subject or body 'help' to
>> >>> pacemaker-request at oss.clusterlabs.org
>> >>>
>> >>> You can reach the person managing the list at
>> >>> pacemaker-owner at oss.clusterlabs.org
>> >>>
>> >>> When replying, please edit your Subject line so it is more specific
>> >>> than "Re: Contents of Pacemaker digest..."
>> >>>
>> >>>
>> >>> Today's Topics:
>> >>>
>> >>> 1. Re: LVM restarts after SLES upgrade (emmanuel segura)
>> >>>
>> >>>
>> >>> ----------------------------------------------------------------------
>> >>>
>> >>> Message: 1
>> >>> Date: Thu, 19 Apr 2012 09:18:37 +0200
>> >>> From: emmanuel segura <emi2fast at gmail.com>
>> >>> To: The Pacemaker cluster resource manager
>> >>> <pacemaker at oss.clusterlabs.org>
>> >>> Subject: Re: [Pacemaker] LVM restarts after SLES upgrade
>> >>> Message-ID:
>> >>> <CAE7pJ3Ab1ZzSpou0=vvrO=
>> >> mb3ur2+K1XhALLJEpj-sOOUEfwTA at mail.gmail.com>
>> >>> Content-Type: text/plain; charset="iso-8859-1"
>> >>>
>> >>> maybe you have problem with clvm
>> >>>
>> >>> can you show me your vm configuration?
>> >>>
>> >>> Il giorno 19 aprile 2012 08:29, Frank Meier <frank.meier at hr-group.de>
>> ha
>> >>> scritto:
>> >>>
>> >>>> Hi,
>> >>>>
>> >>>> I've installed a 2-Node Xen-Cluster with SLES 11 SP1.
>> >>>>
>> >>>> After an upgrade to SLES11 SP2 the cluster won't work as the old one.
>> >>>>
>> >>>> After some time, all LVs reorganize and so the VM where restarted.
>> >>>>
>> >>>> Any clue?
>> >>>>
>> >>>> CIB:
>> >>>>
>> >>>> node xencluster1 \
>> >>>> attributes standby="off"
>> >>>> node xencluster2 \
>> >>>> attributes standby="off"
>> >>>> primitive STONITH--Node1 stonith:meatware \
>> >>>> meta is-managed="true" target-role="Started" \
>> >>>> operations $id="STONITH--Node1-operations" \
>> >>>> op monitor interval="15" timeout="15" start-delay="15" \
>> >>>> params hostlist="xencluster1"
>> >>>> primitive STONITH-Node2 stonith:meatware \
>> >>>> meta target-role="started" is-managed="true" \
>> >>>> operations $id="STONITH-Node2-operations" \
>> >>>> op monitor interval="15" timeout="15" start-delay="15" \
>> >>>> params hostlist="xencluster2"
>> >>>> primitive clvm ocf:lvm2:clvmd \
>> >>>> operations $id="clvm-operations" \
>> >>>> op monitor interval="10" timeout="20" \
>> >>>> op start interval="0" timeout="90" \
>> >>>> op stop interval="0" timeout="100" \
>> >>>> params daemon_timeout="30"
>> >>>> primitive clvm-xenvg ocf:heartbeat:LVM \
>> >>>> operations $id="clvm-xenvg-operations" \
>> >>>> op monitor interval="10" timeout="30" \
>> >>>> params volgrpname="XenVG"
>> >>>> primitive cmirror ocf:lvm2:cmirrord \
>> >>>> operations $id="cmirror-operations" \
>> >>>> op monitor interval="10" timeout="20" \
>> >>>> op start interval="0" timeout="90" \
>> >>>> op stop interval="0" timeout="100" \
>> >>>> params daemon_timeout="30"
>> >>>> primitive dlm ocf:pacemaker:controld \
>> >>>> operations $id="dlm-operations" \
>> >>>> op monitor interval="10" timeout="20" start-delay="0" \
>> >>>> op start interval="0" timeout="90" \
>> >>>> op stop interval="0" timeout="100"
>> >>>> primitive fs-config-xen ocf:heartbeat:Filesystem \
>> >>>> meta is-managed="true" target-role="Started" \
>> >>>> operations $id="fs-config-xen-operations" \
>> >>>> op monitor interval="20" timeout="40" \
>> >>>> params device="/dev/mapper/XenVG-xenconfig"
>> >> directory="/etc/xen/vm"
>> >>>> fstype="ocfs2"
>> >>>> primitive o2cb ocf:ocfs2:o2cb \
>> >>>> operations $id="o2cb-operations" \
>> >>>> op monitor interval="10" timeout="20" \
>> >>>> op start interval="0" timeout="90" \
>> >>>> op stop interval="0" timeout="100"
>> >>>> primitive vm-cim ocf:heartbeat:Xen \
>> >>>> meta target-role="Started" is-managed="true"
>> >> allow-migrate="true" \
>> >>>> operations $id="vm-cim-operations" \
>> >>>> op monitor interval="10" timeout="30" \
>> >>>> op start interval="0" timeout="60" \
>> >>>> op stop interval="0" timeout="60" \
>> >>>> op migrate_to interval="0" timeout="360" \
>> >>>> params xmfile="/etc/xen/vm/cim"
>> >>>> primitive vm-deprepo ocf:heartbeat:Xen \
>> >>>> meta target-role="started" is-managed="true"
>> >> allow-migrate="true" \
>> >>>> operations $id="vm-deprepo-operations" \
>> >>>> op monitor interval="10" timeout="30" \
>> >>>> op start interval="0" timeout="60" \
>> >>>> op stop interval="0" timeout="60" \
>> >>>> op migrate_to interval="0" timeout="360" \
>> >>>> params xmfile="/etc/xen/vm/debrepo"
>> >>>>
>> >>>>
>> >>>> group clvm-glue dlm clvm o2cb cmirror \
>> >>>> meta target-role="Started"
>> >>>> group xen-vg-fs clvm-xenvg fs-config-xen
>> >>>> clone c-clvm-glue clvm-glue \
>> >>>> meta interleave="true" ordered="true" target-role="Started"
>> >>>> clone c-xen-vg-fs xen-vg-fs \
>> >>>> meta interleave="true" ordered="true" is-managed="true"
>> >>>> target-role="Started"
>> >>>> location STONITHnode1Constraint STONITH--Node1 -inf: xencluster1
>> >>>> colocation colo-clvmglue-xenvgsfs inf: c-xen-vg-fs c-clvm-glue
>> >>>> order o-cim inf: c-xen-vg-fs vm-cim
>> >>>> order o-clvmglue-xenvgfs inf: c-clvm-glue c-xen-vg-fs
>> >>>> order o-deprepo inf: c-xen-vg-fs vm-deprepo
>> >>>> order o-ilmt inf: c-xen-vg-fs vm-ilmt
>> >>>> order o-intranet inf: c-xen-vg-fs vm-intranet
>> >>>> order o-nagois inf: c-xen-vg-fs vm-nagios
>> >>>> order o-oneiroi inf: c-xen-vg-fs vm-oneiroi
>> >>>> order o-otbridge inf: c-xen-vg-fs vm-otbridge
>> >>>> order o-soa inf: c-xen-vg-fs vm-soa
>> >>>> order o-topdesk-entw inf: c-xen-vg-fs vm-topdesk-entw
>> >>>> order o-traumschiff inf: c-xen-vg-fs vm-traumschiff
>> >>>> order o-virenscanner inf: c-xen-vg-fs vm-virenscanner
>> >>>> property $id="cib-bootstrap-options" \
>> >>>> dc-version="1.1.6-b988976485d15cb702c9307df55512d323831a5e" \
>> >>>> no-quorum-policy="ignore" \
>> >>>> default-resource-stickiness="100000" \
>> >>>> last-lrm-refresh="1334518148" \
>> >>>> cluster-infrastructure="openais" \
>> >>>> expected-quorum-votes="2" \
>> >>>> default-action-timeout="120s" \
>> >>>> maintenance-mode="true"
>> >>>> op_defaults $id="op_defaults-options" \
>> >>>> record-pending="false"
>> >>>>
>> >>>> Node 1Logs:
>> >>>>
>> >>>> Apr 15 22:01:10 xencluster1 clvmd[8763]: Sending message to all
>> cluster
>> >>>> nodes
>> >>>> Apr 15 22:01:10 xencluster1 clvmd[8763]: process_work_item: local
>> >>>> Apr 15 22:01:10 xencluster1 clvmd[8763]: process_local_command:
>> unknown
>> >>>> (0x2d) msg=0x7f5c40010e80, msglen =32, client=0x7f5c400578d0
>> >>>> Apr 15 22:01:10 xencluster1 clvmd[8763]: Syncing device names
>> >>>> Apr 15 22:01:10 xencluster1 clvmd[8763]: Reply from node 4914420a: 0
>> >> bytes
>> >>>> Apr 15 22:01:10 xencluster1 clvmd[8763]: Got 1 replies, expecting: 2
>> >>>> Apr 15 22:01:10 xencluster1 clvmd[8763]: LVM thread waiting for work
>> >>>> Apr 15 22:01:10 xencluster1 clvmd[8763]: 1226064394 got message from
>> >>>> nodeid 1226064394 for 0. len 32
>> >>>> Apr 15 22:01:40 xencluster1 lrmd: [8093]: WARN: clvm-xenvg:0:monitor
>> >>>> process (PID 31785) timed out (try 1). Killing with signal SIGTERM
>> >> (15).
>> >>>> Apr 15 22:01:40 xencluster1 lrmd: [8093]: WARN: operation monitor[92]
>> on
>> >>>> clvm-xenvg:0 for client 8096: pid 31785 timed out
>> >>>> Apr 15 22:01:40 xencluster1 crmd: [8096]: ERROR: process_lrm_event:
>> LRM
>> >>>> operation clvm-xenvg:0_monitor_10000 (92) Timed Out (timeout=30000ms)
>> >>>> Apr 15 22:01:42 xencluster1 corosync[8071]: [TOTEM ] Retransmit
>> List:
>> >>>> 21ac
>> >>>> Apr 15 22:01:42 xencluster1 corosync[8071]: [TOTEM ] Retransmit
>> List:
>> >>>> 21ab
>> >>>> Apr 15 22:01:42 xencluster1 corosync[8071]: [TOTEM ] Retransmit
>> List:
>> >>>> 21ac
>> >>>> Apr 15 22:01:42 xencluster1 corosync[8071]: [TOTEM ] Retransmit
>> List:
>> >>>> 21ab
>> >>>>
>> >>>>
>> >>>> pr 15 22:01:50 xencluster1 clvmd[8763]: Send local reply
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: Read on local socket 14, len
>> =
>> >> 32
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: check_all_clvmds_running
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: down_callback. node
>> 1226064394,
>> >>>> state = 3
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: down_callback. node
>> 1309950474,
>> >>>> state = 3
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: Got pre command condition...
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: Writing status 0 down pipe
>> >>>> 16
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: Waiting to do post command -
>> >>>> state = 0
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: read on PIPE 15: 4 bytes:
>> >> status:
>> >>>> 0
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: background routine status
>> >>>> was
>> >>>> 0, sock_client=0x7f5c40057c10
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: distribute command: XID =
>> 15473
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: num_nodes = 2
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: add_to_lvmqueue:
>> >>>> cmd=0x7f5c4000dbd0. client=0x7f5c40057c10, msg=0x7f5c40057880,
>> >>>> len=32,
>> >>>> csid=(nil), xid=15473
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: Sending message to all
>> cluster
>> >>>> nodes
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: process_work_item: local
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: process_local_command:
>> unknown
>> >>>> (0x2d) msg=0x7f5c40010e80, msglen =32, client=0x7f5c40057c10
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: Syncing device names
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: Reply from node 4914420a: 0
>> >> bytes
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: Got 1 replies, expecting: 2
>> >>>> Apr 15 22:01:50 xencluster1 clvmd[8763]: LVM thread waiting for work
>> >>>> Apr 15 22:01:50 xencluster1 corosync[8071]: [TOTEM ] Retransmit
>> List:
>> >>>> 21ab
>> >>>> Apr 15 22:01:50 xencluster1 corosync[8071]: [TOTEM ] Retransmit
>> List:
>> >>>> 21ac
>> >>>> Apr 15 22:01:50 xencluster1 corosync[8071]: [TOTEM ] Retransmit
>> List:
>> >>>> 21ab
>> >>>> ...
>> >>>> Apr 15 22:02:19 xencluster1 corosync[8071]: [TOTEM ] Retransmit
>> List:
>> >>>> 21ac
>> >>>> Apr 15 22:02:20 xencluster1 corosync[8071]: [TOTEM ] Retransmit
>> List:
>> >>>> 21ab
>> >>>> Apr 15 22:02:20 xencluster1 lrmd: [8093]: WARN: clvm-xenvg:0:monitor
>> >>>> process (PID 771) timed out (try 1). Killing with signal SIGTERM
>> (15).
>> >>>> Apr 15 22:02:20 xencluster1 lrmd: [8093]: WARN: operation monitor[92]
>> on
>> >>>> clvm-xenvg:0 for client 8096: pid 771 timed out
>> >>>> Apr 15 22:02:20 xencluster1 crmd: [8096]: ERROR: process_lrm_event:
>> LRM
>> >>>> operation clvm-xenvg:0_monitor_10000 (92) Timed Out (timeout=30000ms)
>> >>>> Apr 15 22:02:20 xencluster1 corosync[8071]: [TOTEM ] Retransmit
>> List:
>> >>>> 21ac
>> >>>> Apr 15 22:02:20 xencluster1 corosync[8071]: [TOTEM ] Retransmit
>> List:
>> >>>> 21ab
>> >>>> ...
>> >>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: Got new connection on fd 17
>> >>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: Read on local socket 17, len
>> =
>> >> 28
>> >>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: creating pipe, [18, 19]
>> >>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: Creating pre&post thread
>> >>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: Created pre&post thread,
>> state
>> >> = 0
>> >>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: in sub thread: client =
>> >>>> 0x7f5c40058080
>> >>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: Sub thread ready for work.
>> >>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: doing PRE command LOCK_VG
>> >>>> 'V_XenVG' at 1 (client=0x7f5c40058080)
>> >>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: lock_resource 'V_XenVG',
>> >>>> flags=0, mode=3
>> >>>> Apr 15 22:02:30 xencluster1 clvmd[8763]: lock_resource returning 0,
>> >>>> lock_id=3a0001
>> >>>>
>> >>>>
>> >>>>
>> >>>> Apr 15 22:02:48 xencluster1 corosync[8071]: [TOTEM ] Retransmit
>> List:
>> >>>> 21ab
>> >>>> Apr 15 22:02:48 xencluster1 cib: [8092]: info: cib_stats: Processed 4
>> >>>> operations (2500.00us average, 0% utilization) in the last 10min
>> >>>> Apr 15 22:02:48 xencluster1 corosync[8071]: [TOTEM ] Retransmit
>> List:
>> >>>> 21ac
>> >>>>
>> >>>>
>> >>>>
>> >>>> Apr 15 22:03:00 xencluster1 corosync[8071]: [TOTEM ] Retransmit
>> List:
>> >>>> 21ac
>> >>>> Apr 15 22:03:00 xencluster1 lrmd: [8093]: WARN: clvm-xenvg:0:monitor
>> >>>> process (PID 2256) timed out (try 1). Killing with signal SIGTERM
>> (15).
>> >>>> Apr 15 22:03:00 xencluster1 lrmd: [8093]: WARN: operation monitor[92]
>> on
>> >>>> clvm-xenvg:0 for client 8096: pid 2256 timed out
>> >>>> Apr 15 22:03:00 xencluster1 crmd: [8096]: ERROR: process_lrm_event:
>> LRM
>> >>>> operation clvm-xenvg:0_monitor_10000 (92) Timed Out (timeout=30000ms)
>> >>>> Apr 15 22:03:00 xencluster1 corosync[8071]: [TOTEM ] Retransmit
>> List:
>> >>>> 21ab
>> >>>>
>> >>>>
>> >>>> Node2 Log
>> >>>>
>> >>>> Apr 15 22:00:51 xencluster2 clvmd[12666]: Joined child thread
>> >>>> Apr 15 22:00:51 xencluster2 clvmd[12666]: ret == 0, errno = 0.
>> removing
>> >>>> client
>> >>>> Apr 15 22:00:51 xencluster2 clvmd[12666]: add_to_lvmqueue:
>> >>>> cmd=0x7fa3d0028710. client=0x7fa3d0028760, msg=(nil), len=0,
>> csid=(nil),
>> >>>> xid=14778
>> >>>> Apr 15 22:00:51 xencluster2 clvmd[12666]: process_work_item: free fd
>> -1
>> >>>> Apr 15 22:00:51 xencluster2 clvmd[12666]: LVM thread waiting for work
>> >>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: crm_timer_popped:
>> >>>> PEngine Recheck Timer (I_PE_CALC) just popped (900000ms)
>> >>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: do_state_transition:
>> >>>> State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
>> >>>> cause=C_TIMER_POPPED origin=crm_timer_popped ]
>> >>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: do_state_transition:
>> >>>> Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
>> >>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: do_state_transition:
>> All
>> >>>> 2 cluster nodes are eligible to run resources.
>> >>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: do_pe_invoke: Query
>> 983:
>> >>>> Requesting the current CIB: S_POLICY_ENGINE
>> >>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info:
>> >>>> do_pe_invoke_callback:
>> >>>> Invoking the PE: query=983, ref=pe_calc-dc-1334520052-604, seq=2212,
>> >>>> quorate=1
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_config:
>> >>>> On
>> >>>> loss of CCM Quorum: Ignore
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation clvm:1_last_failure_0 found resource clvm:1 active on
>> >> xencluster2
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation STONITH-Node2_last_failure_0 found resource STONITH-Node2
>> >>>> active on xencluster2
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation STONITH--Node1_last_failure_0 found resource STONITH--Node1
>> >>>> active on xencluster2
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation cmirror:1_last_failure_0 found resource cmirror:1 active on
>> >>>> xencluster2
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation dlm:1_last_failure_0 found resource dlm:1 active on
>> >> xencluster2
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation o2cb:1_last_failure_0 found resource o2cb:1 active on
>> >> xencluster2
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation dlm:0_last_failure_0 found resource dlm:0 active on
>> >> xencluster1
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation clvm:0_last_failure_0 found resource clvm:0 active on
>> >> xencluster1
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation o2cb:0_last_failure_0 found resource o2cb:0 active on
>> >> xencluster1
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation clvm-xenvg:0_last_failure_0 found resource clvm-xenvg:0
>> active
>> >>>> on xencluster1
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: WARN: unpack_rsc_op:
>> >>>> Processing failed op vm-deprepo_last_failure_0 on xencluster1:
>> >>>> unknown
>> >>>> error (1)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> STONITH-Node2 (Started xencluster2)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> STONITH--Node1 (Started xencluster2)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> dlm:0 (Started xencluster1)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> clvm:0 (Started xencluster1)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> o2cb:0 (Started xencluster1)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> cmirror:0 (Started xencluster1)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> dlm:1 (Started xencluster2)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> clvm:1 (Started xencluster2)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> o2cb:1 (Started xencluster2)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> cmirror:1 (Started xencluster2)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> clvm-xenvg:0 (Started xencluster1)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> fs-config-xen:0 (Started xencluster1)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> clvm-xenvg:1 (Started xencluster2)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> fs-config-xen:1 (Started xencluster2)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> vm-ilmt (Started xencluster1)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> vm-topdesk-entw (Started xencluster1)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> vm-otbridge (Started xencluster2)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> vm-virenscanner (Started xencluster1)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> vm-deprepo (Started xencluster1)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> vm-traumschiff (Started xencluster2)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> vm-cim (Started xencluster2)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> vm-oneiroi (Started xencluster1)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> vm-intranet (Started xencluster1)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> vm-soa (Started xencluster1)
>> >>>> Apr 15 22:00:52 xencluster2 pengine: [7677]: notice:
>> process_pe_message:
>> >>>> Transition 80: PEngine Input stored in:
>> >> /var/lib/pengine/pe-input-271.bz2
>> >>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: do_state_transition:
>> >>>> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
>> >>>> input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
>> >>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: unpack_graph:
>> >>>> Unpacked
>> >>>> transition 80: 0 actions in 0 synapses
>> >>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: do_te_invoke:
>> Processing
>> >>>> graph 80 (ref=pe_calc-dc-1334520052-604) derived from
>> >>>> /var/lib/pengine/pe-input-271.bz2
>> >>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: run_graph:
>> >>>> ====================================================
>> >>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: notice: run_graph:
>> Transition
>> >>>> 80 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
>> >>>> Source=/var/lib/pengine/pe-input-271.bz2): Complete
>> >>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: te_graph_trigger:
>> >>>> Transition 80 is now complete
>> >>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: notify_crmd:
>> Transition
>> >>>> 80 status: done - <null>
>> >>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: do_state_transition:
>> >>>> State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
>> >>>> cause=C_FSA_INTERNAL origin=notify_crmd ]
>> >>>> Apr 15 22:00:52 xencluster2 crmd: [7678]: info: do_state_transition:
>> >>>> Starting PEngine Recheck Timer
>> >>>> Apr 15 22:01:00 xencluster2 clvmd[12666]: 1309950474 got message from
>> >>>> nodeid 1226064394 for 0. len 32
>> >>>> Apr 15 22:01:00 xencluster2 clvmd[12666]: add_to_lvmqueue:
>> >>>> cmd=0x7fa3d0028780. client=0x6934a0, msg=0x7fa3d910063c, len=32,
>> >>>> csid=0x7fffb4e5d944, xid=0
>> >>>> Apr 15 22:01:00 xencluster2 clvmd[12666]: process_work_item: remote
>> >>>> Apr 15 22:01:00 xencluster2 clvmd[12666]: process_remote_command
>> unknown
>> >>>> (0x2d) for clientid 0x5000000 XID 15337 on node 4914420a
>> >>>> Apr 15 22:01:00 xencluster2 clvmd[12666]: Syncing device names
>> >>>> Apr 15 22:01:00 xencluster2 clvmd[12666]: LVM thread waiting for work
>> >>>> Apr 15 22:01:00 xencluster2 clvmd[12666]: 1309950474 got message from
>> >>>> nodeid 1309950474 for 1226064394. len 18
>> >>>> Apr 15 22:01:00 xencluster2 clvmd[12666]: 1309950474 got message from
>> >>>> nodeid 1226064394 for 0. len 32
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> Apr 15 22:01:02 xencluster2 clvmd[12666]: Joined child thread
>> >>>> Apr 15 22:01:02 xencluster2 clvmd[12666]: ret == 0, errno = 0.
>> removing
>> >>>> client
>> >>>> Apr 15 22:01:02 xencluster2 clvmd[12666]: add_to_lvmqueue:
>> >>>> cmd=0x7fa3d004d7a0. client=0x7fa3d004d7f0, msg=(nil), len=0,
>> csid=(nil),
>> >>>> xid=14848
>> >>>> Apr 15 22:01:02 xencluster2 clvmd[12666]: process_work_item: free fd
>> -1
>> >>>> Apr 15 22:01:02 xencluster2 clvmd[12666]: LVM thread waiting for work
>> >>>> Apr 15 22:01:06 xencluster2 stonith-ng: [8283]: info:
>> >>>> stonith_command:
>> >>>> Processed st_execute from lrmd: rc=-1
>> >>>> Apr 15 22:01:06 xencluster2 stonith: parse config info
>> info=xencluster1
>> >>>> Apr 15 22:01:06 xencluster2 stonith: meatware device OK.
>> >>>> Apr 15 22:01:06 xencluster2 stonith-ng: [8283]: info: log_operation:
>> >>>> STONITH--Node1: Performing: stonith -t meatware -S
>> >>>> Apr 15 22:01:06 xencluster2 stonith-ng: [8283]: info: log_operation:
>> >>>> STONITH--Node1: success: 0
>> >>>> Apr 15 22:01:06 xencluster2 stonith-ng: [8283]: info:
>> >>>> stonith_device_execute: Nothing to do for STONITH--Node1
>> >>>> Apr 15 22:01:06 xencluster2 stonith-ng: [8283]: info:
>> >>>> stonith_command:
>> >>>> Processed st_execute from lrmd: rc=-1
>> >>>> Apr 15 22:01:06 xencluster2 stonith: parse config info
>> info=xencluster2
>> >>>> Apr 15 22:01:06 xencluster2 stonith: meatware device OK.
>> >>>> Apr 15 22:01:06 xencluster2 stonith-ng: [8283]: info: log_operation:
>> >>>> STONITH-Node2: Performing: stonith -t meatware -S
>> >>>> Apr 15 22:01:06 xencluster2 stonith-ng: [8283]: info: log_operation:
>> >>>> STONITH-Node2: success: 0
>> >>>> Apr 15 22:01:06 xencluster2 stonith-ng: [8283]: info:
>> >>>> stonith_device_execute: Nothing to do for STONITH-Node2
>> >>>> Apr 15 22:01:10 xencluster2 clvmd[12666]: 1309950474 got message from
>> >>>> nodeid 1226064394 for 0. len 32
>> >>>> Apr 15 22:01:10 xencluster2 clvmd[12666]: add_to_lvmqueue:
>> >>>> cmd=0x7fa3d004d810. client=0x6934a0, msg=0x7fa3d9100cfc, len=32,
>> >>>> csid=0x7fffb4e5d944, xid=0
>> >>>> Apr 15 22:01:10 xencluster2 clvmd[12666]: process_work_item: remote
>> >>>> Apr 15 22:01:10 xencluster2 clvmd[12666]: process_remote_command
>> unknown
>> >>>> (0x2d) for clientid 0x5000000 XID 15407 on node 4914420a
>> >>>> Apr 15 22:01:10 xencluster2 clvmd[12666]: Syncing device names
>> >>>> Apr 15 22:01:10 xencluster2 clvmd[12666]: LVM thread waiting for work
>> >>>> Apr 15 22:01:10 xencluster2 clvmd[12666]: 1309950474 got message from
>> >>>> nodeid 1309950474 for 1226064394. len 18
>> >>>> Apr 15 22:01:10 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196
>> >>>> Apr 15 22:01:10 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196
>> >>>> Apr 15 22:01:10 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196
>> >>>> ....
>> >>>> Apr 15 22:01:11 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196
>> >>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: Got new connection on fd 5
>> >>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: Read on local socket 5, len
>> =
>> >> 28
>> >>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: creating pipe, [12, 13]
>> >>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: Creating pre&post thread
>> >>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: Created pre&post thread,
>> >> state =
>> >>>> 0
>> >>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: in sub thread: client =
>> >>>> 0x7fa3d004d810
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: Syncing device names
>> >>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: Reply from node 4e14420a: 0
>> >> bytes
>> >>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: Got 1 replies, expecting: 2
>> >>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: LVM thread waiting for work
>> >>>> Apr 15 22:01:12 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196
>> >>>> Apr 15 22:01:12 xencluster2 clvmd[12666]: 1309950474 got message from
>> >>>> nodeid 1309950474 for 0. len 32
>> >>>> Apr 15 22:01:12 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196
>> >>>> Apr 15 22:01:12 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196
>> >>>> ....
>> >>>> Apr 15 22:01:21 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> Apr 15 22:01:21 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> Apr 15 22:01:21 xencluster2 stonith-ng: [8283]: info:
>> >>>> stonith_command:
>> >>>> Processed st_execute from lrmd: rc=-1
>> >>>> Apr 15 22:01:21 xencluster2 stonith: parse config info
>> info=xencluster1
>> >>>> Apr 15 22:01:21 xencluster2 stonith: meatware device OK.
>> >>>> Apr 15 22:01:21 xencluster2 stonith-ng: [8283]: info: log_operation:
>> >>>> STONITH--Node1: Performing: stonith -t meatware -S
>> >>>> Apr 15 22:01:21 xencluster2 stonith-ng: [8283]: info: log_operation:
>> >>>> STONITH--Node1: success: 0
>> >>>> Apr 15 22:01:21 xencluster2 stonith-ng: [8283]: info:
>> >>>> stonith_device_execute: Nothing to do for STONITH--Node1
>> >>>> Apr 15 22:01:21 xencluster2 stonith-ng: [8283]: info:
>> >>>> stonith_command:
>> >>>> Processed st_execute from lrmd: rc=-1
>> >>>> Apr 15 22:01:21 xencluster2 stonith: parse config info
>> info=xencluster2
>> >>>> Apr 15 22:01:21 xencluster2 stonith: meatware device OK.
>> >>>> Apr 15 22:01:21 xencluster2 stonith-ng: [8283]: info: log_operation:
>> >>>> STONITH-Node2: Performing: stonith -t meatware -S
>> >>>> Apr 15 22:01:21 xencluster2 stonith-ng: [8283]: info: log_operation:
>> >>>> STONITH-Node2: success: 0
>> >>>> Apr 15 22:01:21 xencluster2 stonith-ng: [8283]: info:
>> >>>> stonith_device_execute: Nothing to do for STONITH-Node2
>> >>>> Apr 15 22:01:21 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> Apr 15 22:01:22 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> Apr 15 22:01:22 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> ....
>> >>>> Apr 15 22:01:36 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> Apr 15 22:01:36 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> Apr 15 22:01:36 xencluster2 stonith-ng: [8283]: info:
>> >>>> stonith_command:
>> >>>> Processed st_execute from lrmd: rc=-1
>> >>>> Apr 15 22:01:36 xencluster2 stonith: parse config info
>> info=xencluster1
>> >>>> Apr 15 22:01:36 xencluster2 stonith: meatware device OK.
>> >>>> Apr 15 22:01:36 xencluster2 stonith-ng: [8283]: info: log_operation:
>> >>>> STONITH--Node1: Performing: stonith -t meatware -S
>> >>>> Apr 15 22:01:36 xencluster2 stonith-ng: [8283]: info: log_operation:
>> >>>> STONITH--Node1: success: 0
>> >>>> Apr 15 22:01:36 xencluster2 stonith-ng: [8283]: info:
>> >>>> stonith_device_execute: Nothing to do for STONITH--Node1
>> >>>> Apr 15 22:01:36 xencluster2 stonith-ng: [8283]: info:
>> >>>> stonith_command:
>> >>>> Processed st_execute from lrmd: rc=-1
>> >>>> Apr 15 22:01:36 xencluster2 stonith: parse config info
>> info=xencluster2
>> >>>> Apr 15 22:01:36 xencluster2 stonith: meatware device OK.
>> >>>> Apr 15 22:01:36 xencluster2 stonith-ng: [8283]: info: log_operation:
>> >>>> STONITH-Node2: Performing: stonith -t meatware -S
>> >>>> Apr 15 22:01:36 xencluster2 stonith-ng: [8283]: info: log_operation:
>> >>>> STONITH-Node2: success: 0
>> >>>> Apr 15 22:01:36 xencluster2 stonith-ng: [8283]: info:
>> >>>> stonith_device_execute: Nothing to do for STONITH-Node2
>> >>>> Apr 15 22:01:36 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> Apr 15 22:01:37 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> ....
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: process_graph_event:
>> >>>> Detected action clvm-xenvg:0_monitor_10000 from a different
>> transition:
>> >>>> 62 vs. 80
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info:
>> abort_transition_graph:
>> >>>> process_graph_event:476 - Triggered transition abort (complete=1,
>> >>>> tag=lrm_rsc_op, id=clvm-xenvg:0_monitor_10000,
>> >>>> magic=2:-2;61:62:0:72497771-fdb2-4ca7-a723-2cc360861bb0,
>> cib=0.5816.4) :
>> >>>> Old event
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: WARN: update_failcount:
>> >>>> Updating failcount for clvm-xenvg:0 on xencluster1 after failed
>> monitor:
>> >>>> rc=-2 (update=value++, time=1334520100)
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: do_state_transition:
>> >>>> State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
>> >>>> cause=C_FSA_INTERNAL origin=abort_transition_graph ]
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: do_state_transition:
>> All
>> >>>> 2 cluster nodes are eligible to run resources.
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: do_pe_invoke: Query
>> 984:
>> >>>> Requesting the current CIB: S_POLICY_ENGINE
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197 219a 219b 219c 219d
>> >>>> ...
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197 219a 219b 219c 219d
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info:
>> >>>> do_pe_invoke_callback:
>> >>>> Invoking the PE: query=984, ref=pe_calc-dc-1334520100-605, seq=2212,
>> >>>> quorate=1
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197 219a 219b 219c 219d
>> >>>> ...
>> >>>>
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197 219a 219b 219c 219d
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_config:
>> >>>> On
>> >>>> loss of CCM Quorum: Ignore
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation clvm:1_last_failure_0 found resource clvm:1 active on
>> >> xencluster2
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation STONITH-Node2_last_failure_0 found resource STONITH-Node2
>> >>>> active on xencluster2
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation STONITH--Node1_last_failure_0 found resource STONITH--Node1
>> >>>> active on xencluster2
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation cmirror:1_last_failure_0 found resource cmirror:1 active on
>> >>>> xencluster2
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation dlm:1_last_failure_0 found resource dlm:1 active on
>> >> xencluster2
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation o2cb:1_last_failure_0 found resource o2cb:1 active on
>> >> xencluster2
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation cmirror:0_last_failure_0 found resource cmirror:0 active on
>> >>>> xencluster1
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation dlm:0_last_failure_0 found resource dlm:0 active on
>> >> xencluster1
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation clvm:0_last_failure_0 found resource clvm:0 active on
>> >> xencluster1
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation o2cb:0_last_failure_0 found resource o2cb:0 active on
>> >> xencluster1
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: unpack_rsc_op:
>> >>>> Operation clvm-xenvg:0_last_failure_0 found resource clvm-xenvg:0
>> active
>> >>>> on xencluster1
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: WARN: unpack_rsc_op:
>> >>>> Processing failed op clvm-xenvg:0_monitor_10000 on xencluster1:
>> unknown
>> >>>> exec error (-2)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: WARN: unpack_rsc_op:
>> >>>> Processing failed op vm-deprepo_last_failure_0 on xencluster1:
>> >>>> unknown
>> >>>> error (1)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: RecurringOp:
>> Start
>> >>>> recurring monitor (10s) for clvm-xenvg:0 on xencluster1
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> STONITH-Node2 (Started xencluster2)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> STONITH--Node1 (Started xencluster2)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> dlm:0 (Started xencluster1)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> clvm:0 (Started xencluster1)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> o2cb:0 (Started xencluster1)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> cmirror:0 (Started xencluster1)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> dlm:1 (Started xencluster2)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> clvm:1 (Started xencluster2)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> o2cb:1 (Started xencluster2)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> cmirror:1 (Started xencluster2)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> Recover
>> >>>> clvm-xenvg:0 (Started xencluster1)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> Restart
>> >>>> fs-config-xen:0 (Started xencluster1)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> clvm-xenvg:1 (Started xencluster2)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> fs-config-xen:1 (Started xencluster2)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> Restart
>> >>>> vm-ilmt (Started xencluster1)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> Restart
>> >>>> vm-topdesk-entw (Started xencluster1)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> vm-otbridge (Started xencluster2)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> Restart
>> >>>> vm-virenscanner (Started xencluster1)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> Restart
>> >>>> vm-deprepo (Started xencluster1)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> vm-traumschiff (Started xencluster2)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> >>>> Leave
>> >>>> vm-cim (Started xencluster2)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> Restart
>> >>>> vm-oneiroi (Started xencluster1)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> Restart
>> >>>> vm-intranet (Started xencluster1)
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice: LogActions:
>> Restart
>> >>>> vm-soa (Started xencluster1)
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: do_state_transition:
>> >>>> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
>> >>>> input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: unpack_graph:
>> >>>> Unpacked
>> >>>> transition 81: 39 actions in 39 synapses
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: do_te_invoke:
>> Processing
>> >>>> graph 81 (ref=pe_calc-dc-1334520100-605) derived from
>> >>>> /var/lib/pengine/pe-input-272.bz2
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: te_rsc_command:
>> >>>> Initiating action 82: stop vm-ilmt_stop_0 on xencluster1
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: te_rsc_command:
>> >>>> Initiating action 86: stop vm-topdesk-entw_stop_0 on xencluster1
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: te_rsc_command:
>> >>>> Initiating action 90: stop vm-virenscanner_stop_0 on xencluster1
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: te_rsc_command:
>> >>>> Initiating action 92: stop vm-deprepo_stop_0 on xencluster1
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: te_rsc_command:
>> >>>> Initiating action 98: stop vm-oneiroi_stop_0 on xencluster1
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: te_rsc_command:
>> >>>> Initiating action 100: stop vm-intranet_stop_0 on xencluster1
>> >>>> Apr 15 22:01:40 xencluster2 crmd: [7678]: info: te_rsc_command:
>> >>>> Initiating action 102: stop vm-soa_stop_0 on xencluster1
>> >>>> Apr 15 22:01:40 xencluster2 pengine: [7677]: notice:
>> process_pe_message:
>> >>>> Transition 81: PEngine Input stored in:
>> >> /var/lib/pengine/pe-input-272.bz2
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197 219a 219b 219c 219d
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197 219a 219b 219c 219d
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197 219a 219b 219c 219d
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197 219a 219b 219c 219d
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197 219a 219b 219c 219d
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197 219a 219b 219c 219d 219e 219f 21a0 21a1 21a2 21a3 21a4 21a5
>> >>>> Apr 15 22:01:40 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197 219a 219b 219c 219d 219e 219f 21a0 21a1 21a2 21a3 21a4 21a5
>> >>>> ...
>> >>>> Apr 15 22:01:41 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197 219a 219b 219c 219d 219e 219f 21a0 21a1 21a2 21a3 21a4 21a5
>> >>>> Apr 15 22:01:42 xencluster2 lrmd: [7675]: WARN: clvm-xenvg:1:monitor
>> >>>> process (PID 29785) timed out (try 1). Killing with signal SIGTERM
>> >> (15).
>> >>>> Apr 15 22:01:42 xencluster2 lrmd: [7675]: WARN: operation
>> >>>> monitor[125]
>> >>>> on clvm-xenvg:1 for client 7678: pid 29785 timed out
>> >>>> Apr 15 22:01:42 xencluster2 crmd: [7678]: ERROR: process_lrm_event:
>> LRM
>> >>>> operation clvm-xenvg:1_monitor_10000 (125) Timed Out
>> >>>> (timeout=30000ms)
>> >>>> Apr 15 22:01:42 xencluster2 crmd: [7678]: info: process_graph_event:
>> >>>> Detected action clvm-xenvg:1_monitor_10000 from a different
>> transition:
>> >>>> 68 vs. 81
>> >>>> Apr 15 22:01:42 xencluster2 crmd: [7678]: info:
>> abort_transition_graph:
>> >>>> process_graph_event:476 - Triggered transition abort (complete=0,
>> >>>> tag=lrm_rsc_op, id=clvm-xenvg:1_monitor_10000,
>> >>>> magic=2:-2;61:68:0:72497771-fdb2-4ca7-a723-2cc360861bb0,
>> cib=0.5816.5) :
>> >>>> Old event
>> >>>> Apr 15 22:01:42 xencluster2 crmd: [7678]: info:
>> >>>> update_abort_priority:
>> >>>> Abort priority upgraded from 0 to 1000000
>> >>>> Apr 15 22:01:42 xencluster2 crmd: [7678]: info:
>> >>>> update_abort_priority:
>> >>>> Abort action done superceeded by restart
>> >>>> Apr 15 22:01:42 xencluster2 crmd: [7678]: WARN: update_failcount:
>> >>>> Updating failcount for clvm-xenvg:1 on xencluster2 after failed
>> monitor:
>> >>>> rc=-2 (update=value++, time=1334520102)
>> >>>> Apr 15 22:01:42 xencluster2 attrd: [7676]: notice:
>> attrd_trigger_update:
>> >>>> Sending flush op to all hosts for: fail-count-clvm-xenvg:1 (1)
>> >>>> Apr 15 22:01:42 xencluster2 attrd: [7676]: notice:
>> attrd_perform_update:
>> >>>> Sent update 448: fail-count-clvm-xenvg:1=1
>> >>>> Apr 15 22:01:42 xencluster2 attrd: [7676]: notice:
>> attrd_trigger_update:
>> >>>> Sending flush op to all hosts for: last-failure-clvm-xenvg:1
>> >> (1334520102)
>> >>>> Apr 15 22:01:42 xencluster2 crmd: [7678]: info:
>> abort_transition_graph:
>> >>>> te_update_diff:176 - Triggered transition abort (complete=0,
>> tag=nvpair,
>> >>>> id=status-xencluster2-fail-count-clvm-xenvg.1,
>> >>>> name=fail-count-clvm-xenvg:1, value=1, magic=NA, cib=0.5816.6) :
>> >>>> Transient attribute: update
>> >>>> Apr 15 22:01:42 xencluster2 attrd: [7676]: notice:
>> attrd_perform_update:
>> >>>> Sent update 451: last-failure-clvm-xenvg:1=1334520102
>> >>>> Apr 15 22:01:42 xencluster2 crmd: [7678]: info:
>> abort_transition_graph:
>> >>>> te_update_diff:176 - Triggered transition abort (complete=0,
>> tag=nvpair,
>> >>>> id=status-xencluster2-last-failure-clvm-xenvg.1,
>> >>>> name=last-failure-clvm-xenvg:1, value=1334520102, magic=NA,
>> >>>> cib=0.5816.7) : Transient attribute: update
>> >>>> Apr 15 22:01:42 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197 219a 219b 219c 219d 219e 219f 21a0 21a1 21a2 21a3 21a4 21a5
>> >>>> Apr 15 22:01:42 xencluster2 corosync[7666]: [TOTEM ] Retransmit
>> List:
>> >>>> 2196 2197 219a 219b 219c 219d 219e 219f 21a0 21a1 21a2 21a3 21a4 21a5
>> >>>>
>> >>>> --
>> >>>> Mit freundlichen Gr??en
>> >>>>
>> >>>> Frank Meier
>> >>>> UNIX-Basis
>> >>>>
>> >>>> Hamm Reno Group GmbH
>> >>>> Industriegebiet West | D-66987 Thaleischweiler-Fr?schen
>> >>>> T.+49(0)6334 444-8322 | F.+49(0)6334 444-8190
>> >>>> frank.meier at hr-group.de | www.reno.de
>> >>>> ___________________________________________________________________
>> >>>>
>> >>>> Sitz: Am Tie 7 | D-49086 Osnabr?ck
>> >>>> Handelsregister Osnabr?ck HRB 19587
>> >>>> Gesch?ftsf?hrer: Hans-J?rgen de Fries,
>> >>>> Jens Gransee, Manfred Klumpp,
>> >>>> Robert Reisch
>> >>>>
>> >>>> _______________________________________________
>> >>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> >>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> >>>>
>> >>>> Project Home: http://www.clusterlabs.org
>> >>>> Getting started:
>> >> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> >>>> Bugs: http://bugs.clusterlabs.org
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> esta es mi vida e me la vivo hasta que dios quiera
>> >>> -------------- next part --------------
>> >>> An HTML attachment was scrubbed...
>> >>> URL: <
>> >>
>> http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20120419/00a36dbe/attachment.html
>> >>>
>> >>>
>> >>> ------------------------------
>> >>>
>> >>> _______________________________________________
>> >>> Pacemaker mailing list
>> >>> Pacemaker at oss.clusterlabs.org
>> >>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> >>>
>> >>>
>> >>> End of Pacemaker Digest, Vol 53, Issue 38
>> >>> *****************************************
>> >>
>> >> _______________________________________________
>> >> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> >> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> >>
>> >> Project Home: http://www.clusterlabs.org
>> >> Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> >> Bugs: http://bugs.clusterlabs.org
>> >>
>> >
>> >
>> >
>> > --
>> > esta es mi vida e me la vivo hasta que dios quiera
>> > -------------- next part --------------
>> > An HTML attachment was scrubbed...
>> > URL: <
>> http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20120419/63b888a3/attachment.html
>> >
>> >
>> > ------------------------------
>> >
>> > _______________________________________________
>> > Pacemaker mailing list
>> > Pacemaker at oss.clusterlabs.org
>> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> >
>> >
>> > End of Pacemaker Digest, Vol 53, Issue 40
>> > *****************************************
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>
>
>
> --
> esta es mi vida e me la vivo hasta que dios quiera
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20120419/f21bf7c7/attachment.html>
>
> ------------------------------
>
> _______________________________________________
> Pacemaker mailing list
> Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
>
> End of Pacemaker Digest, Vol 53, Issue 42
> *****************************************
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
--
esta es mi vida e me la vivo hasta que dios quiera
More information about the Pacemaker
mailing list