[ClusterLabs] serious problem with iSCSILogicalUnit

Ken Gaillot kgaillot at redhat.com
Wed Dec 18 16:12:05 EST 2019


On Wed, 2019-12-18 at 09:58 +0100, Stefan K wrote:
> Sorry for this.
> I try to simplify my configuration and delete all unnecessary things
> and start only with one LUN, one allowed Initiators, and then step by
> step I add things..
> 
> now I ended up with the following setup:
> node 1: ha-test1 \
>         attributes \
>         attributes standby=off maintenance=off
> node 2: ha-test2 \
>         attributes \
>         attributes standby=off
> primitive ha-ip IPaddr2 \
>         params ip=172.16.101.166 cidr_netmask=24 nic=ens192 \
>         op start interval=0s timeout=20s \
>         op stop interval=0s timeout=20s \
>         op monitor interval=10s timeout=20s \
>         meta target-role=Started
> primitive iscsi-lun00 iSCSILogicalUnit \
>         params implementation=lio-t target_iqn="iqn.2003-
> 01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3" lun=0
> lio_iblock=0 path="/dev/loop1" \
>         meta
> primitive iscsi-lun01 iSCSILogicalUnit \
>         params implementation=lio-t target_iqn="iqn.2003-
> 01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3" lun=1
> lio_iblock=1 path="/dev/loop2"
> primitive iscsi-lun02 iSCSILogicalUnit \
>         params implementation=lio-t target_iqn="iqn.2003-
> 01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3" lun=2
> lio_iblock=2 path="/dev/loop3" \
>         meta
> primitive iscsi-server iSCSITarget \
>         params implementation=lio-t iqn="iqn.2003-01.org.linux-
> iscsi.vm-storage.x8664:sn.cf6fa66tgyh3" portals="172.16.101.166:3260"
> allowed_initiators="iqn.1998-01.com.vmware:brainslug99-75u88a95
> iqn.1998-01.com.vmware:brainslug69-5564u4325 iqn.1993-
> 08.org.debian:01:fee35be01c4d iqn.1998-01.com.vmware:brainslug14-
> 34a81jd123 iqn.1998-01.com.vmware:brainslug66-778jau77" \
>         meta
> colocation pcs_rsc_colocation_set_ha-ip_vm_storage_iscsi-server inf:
> ha-ip iscsi-server iscsi-lun00 iscsi-lun01 iscsi-lun02
> order pcs_rsc_order_set_iscsi-server_vm_storage_ha-ip iscsi-server
> iscsi-lun00 iscsi-lun01 iscsi-lun02 ha-ip
> property cib-bootstrap-options: \
>         have-watchdog=false \
>         dc-version=1.1.16-94ff4df \
>         cluster-infrastructure=corosync \
>         cluster-name=ha-vmstorage \
>         no-quorum-policy=stop \
>         stonith-enabled=false \
>         last-lrm-refresh=1576502457
> rsc_defaults rsc_defaults-options: \
>         resource-stickiness=100
> 
> before that I've the following order:
> order pcs_rsc_order_set_ha-ip_iscsi-server_vm_storage ha-ip:stop
> iscsi-lun00:stop iscsi-lun01:stop iscsi-lun02:stop iscsi-server:stop
> symmetrical=false
> order pcs_rsc_order_set_iscsi-server_vm_storage_ha-ip iscsi-
> server:start iscsi-lun00:start iscsi-lun01:start iscsi-lun02:start
> ha-ip:start symmetrical=false
> 
> and I thought that this and my current working constraint are the
> same, but with the 2line-configuration it doesn't work and with the
> one-liner it works fine, so can somebody explain me this? 

It sounds like a bug. If you can, please file a report at
bugs.clusterlabs.org and attach a pe-input from when it did the
unexpected actions. The pe-inputs are noted in "saving inputs" log
messages on the DC node, and the lines before that will have the
actions that were taken at that time.

> 
> On Monday, December 16, 2019 6:38:44 PM CET Andrei Borzenkov wrote:
> > 16.12.2019 18:26, Stefan K пишет:
> > > I thnik I got it..
> > > 
> > > It looks like that (A)
> > > order pcs_rsc_order_set_iscsi-server_haip iscsi-server:start
> > > iscsi-lun00:start iscsi-lun01:start iscsi-lun02:start ha-ip:start 
> > > symmetrical=false
> > 
> > It is different from configuration you show originally.
> > 
> > > order pcs_rsc_order_set_haip_iscsi-server ha-ip:stop iscsi-
> > > lun02:stop iscsi-lun01:stop iscsi-lun00:stop iscsi-server:stop
> > > symmetrical=false
> > > 
> > > and (B)
> > > order pcs_rsc_order_set_iscsi-server_vm_storage_ha-ip iscsi-
> > > server iscsi-lun00 iscsi-lun01 iscsi-lun02 ha-ip
> > > 
> > > doesn't have the same meaning?!
> > > Because with A it doesn't work, but with B it works like
> > > expected, can somebody explain me this behavior?
> > > 
> > 
> > Your original configuration was not symmetrical which may explain
> > it.
> > You never said anything about changing configuration so it is
> > unclear
> > what you tested - original statement or statement you show now.
> > 
> > > best regards
> > > Stefan
> > > 
> > > 
> > > On Thursday, December 12, 2019 4:19:19 PM CET Stefan K wrote:
> > > > So it looks like that it restart the iSCSITarget but not the
> > > > iSCSILogicalUnit, that make sense - more or less because I
> > > > change something in the iSCSITarget, but it is necessary to
> > > > have a working iSCSI.. here a the log output when I change/add
> > > > the iqn..
> > > > 
> > > > Dec 12 16:08:21 [7051] ha-test1        cib:     info:
> > > > cib_process_request:      Forwarding cib_apply_diff operation
> > > > for section 'all' to all (origin=local/cibadmin/2)
> > > > Dec 12 16:08:21 [7051] ha-test1        cib:     info:
> > > > cib_perform_op:   Diff: --- 0.58.3 2
> > > > Dec 12 16:08:21 [7051] ha-test1        cib:     info:
> > > > cib_perform_op:   Diff: +++ 0.59.0
> > > > c5423fcdc276ad43361aeb4c8081f7f4
> > > > Dec 12 16:08:21 [7051] ha-test1        cib:     info:
> > > > cib_perform_op:   +  /cib:  @epoch=59, @num_updates=0
> > > > Dec 12 16:08:21 [7051] ha-test1        cib:     info:
> > > > cib_perform_op:   +  /cib/configuration/resources/primitive[@id
> > > > ='iscsi-server']/instance_attributes[@id='iscsi-server-
> > > > instance_attributes']/nvpair[@id='iscsi-server-
> > > > instance_attributes-allowed_initiators']:  @value=iqn.1998-
> > > > 01.com.vmware:brainslug9-75488e35 iqn.1998-
> > > > 01.com.vmware:brainslug10-5564u4325 iqn.1993-
> > > > 08.org.debian:01:fee35be01c4d iqn.1998-
> > > > 01.com.vmware:brainslug10-34ad648763 iqn.1998-
> > > > 01.com.vmware:brainslug66-75488e12 iqn.1998-01.com.vmware:brai
> > > > Dec 12 16:08:21 [7051] ha-test1        cib:     info:
> > > > cib_process_request:      Completed cib_apply_diff operation
> > > > for section 'all': OK (rc=0, origin=ha-test1/cibadmin/2,
> > > > version=0.59.0)
> > > > Dec 12 16:08:21 [7051] ha-test1        cib:     info:
> > > > cib_file_backup:  Archived previous version as
> > > > /var/lib/pacemaker/cib/cib-69.raw
> > > > Dec 12 16:08:21 [7056] ha-test1       crmd:     info:
> > > > do_lrm_rsc_op:    Performing key=12:116:0:9fd4e826-f0ba-4864-
> > > > 8861-2c585d644d1c op=iscsi-server_stop_0
> > > > Dec 12 16:08:21 [7053] ha-test1       lrmd:     info:
> > > > log_execute:      executing - rsc:iscsi-server action:stop
> > > > call_id:62
> > > > Dec 12 16:08:21 [7051] ha-test1        cib:     info:
> > > > cib_file_write_with_digest:       Wrote version 0.59.0 of the
> > > > CIB to disk (digest: dcbab759c4d0e7f38234434bfbe7ca8e)
> > > > Dec 12 16:08:21 [7051] ha-test1        cib:     info:
> > > > cib_file_write_with_digest:       Reading cluster configuration
> > > > file /var/lib/pacemaker/cib/cib.6YgOXO (digest:
> > > > /var/lib/pacemaker/cib/cib.gOKlWj)
> > > > iSCSITarget(iscsi-server)[4524]:        2019/12/12_16:08:22
> > > > INFO: Deleted Target iqn.2003-01.org.linux-iscsi.vm-
> > > > storage.x8664:sn.cf6fa66tgyh3.
> > > > Dec 12 16:08:22 [7053] ha-test1       lrmd:     info:
> > > > log_finished:     finished - rsc:iscsi-server action:stop
> > > > call_id:62 pid:4524 exit-code:0 exec-time:293ms queue-time:0ms
> > > > Dec 12 16:08:22 [7056] ha-test1       crmd:   notice:
> > > > process_lrm_event:        Result of stop operation for iscsi-
> > > > server on ha-test1: 0 (ok) | call=62 key=iscsi-server_stop_0
> > > > confirmed=true cib-update=58
> > > > Dec 12 16:08:22 [7051] ha-test1        cib:     info:
> > > > cib_process_request:      Forwarding cib_modify operation for
> > > > section status to all (origin=local/crmd/58)
> > > > Dec 12 16:08:22 [7051] ha-test1        cib:     info:
> > > > cib_perform_op:   Diff: --- 0.59.0 2
> > > > Dec 12 16:08:22 [7051] ha-test1        cib:     info:
> > > > cib_perform_op:   Diff: +++ 0.59.1 (null)
> > > > Dec 12 16:08:22 [7051] ha-test1        cib:     info:
> > > > cib_perform_op:   +  /cib:  @num_updates=1
> > > > Dec 12 16:08:22 [7051] ha-test1        cib:     info:
> > > > cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1
> > > > ']/lrm_resources/lrm_resource[@id='iscsi-
> > > > server']/lrm_rsc_op[@id='iscsi-
> > > > server_last_0']:  @operation_key=iscsi-server_stop_0,
> > > > @operation=stop, @transition-key=12:116:0:9fd4e826-f0ba-4864-
> > > > 8861-2c585d644d1c, @transition-magic=0:0;12:116:0:9fd4e826-
> > > > f0ba-4864-8861-2c585d644d1c, @call-id=62, @last-run=1576163301, 
> > > > @last-rc-change=1576163301, @exec-time=293
> > > > Dec 12 16:08:22 [7051] ha-test1        cib:     info:
> > > > cib_process_request:      Completed cib_modify operation for
> > > > section status: OK (rc=0, origin=ha-test1/crmd/58,
> > > > version=0.59.1)
> > > > Dec 12 16:08:22 [7056] ha-test1       crmd:     info:
> > > > do_lrm_rsc_op:    Performing key=3:116:0:9fd4e826-f0ba-4864-
> > > > 8861-2c585d644d1c op=iscsi-server_start_0
> > > > Dec 12 16:08:22 [7053] ha-test1       lrmd:     info:
> > > > log_execute:      executing - rsc:iscsi-server action:start
> > > > call_id:63
> > > > iSCSITarget(iscsi-server)[4564]:        2019/12/12_16:08:22
> > > > INFO: Parameter auto_add_default_portal is now 'false'.
> > > > iSCSITarget(iscsi-server)[4564]:        2019/12/12_16:08:22
> > > > INFO: Created target iqn.2003-01.org.linux-iscsi.vm-
> > > > storage.x8664:sn.cf6fa66tgyh3. Created TPG 1.
> > > > iSCSITarget(iscsi-server)[4564]:        2019/12/12_16:08:22
> > > > INFO: Using default IP port 3260 Created network portal
> > > > 172.16.101.166:3260.
> > > > iSCSITarget(iscsi-server)[4564]:        2019/12/12_16:08:22
> > > > INFO: Created Node ACL for iqn.1998-01.com.vmware:brainslug9-
> > > > 75488e35
> > > > iSCSITarget(iscsi-server)[4564]:        2019/12/12_16:08:23
> > > > INFO: Created Node ACL for iqn.1998-01.com.vmware:brainslug10-
> > > > 5564u4325
> > > > iSCSITarget(iscsi-server)[4564]:        2019/12/12_16:08:23
> > > > INFO: Created Node ACL for iqn.1993-
> > > > 08.org.debian:01:fee35be01c4d
> > > > iSCSITarget(iscsi-server)[4564]:        2019/12/12_16:08:23
> > > > INFO: Created Node ACL for iqn.1998-01.com.vmware:brainslug10-
> > > > 34ad648763
> > > > iSCSITarget(iscsi-server)[4564]:        2019/12/12_16:08:23
> > > > INFO: Created Node ACL for iqn.1998-01.com.vmware:brainslug66-
> > > > 75488e12
> > > > iSCSITarget(iscsi-server)[4564]:        2019/12/12_16:08:23
> > > > INFO: Created Node ACL for iqn.1998-01.com.vmware:brainslug99-
> > > > 5564u4123
> > > > iSCSITarget(iscsi-server)[4564]:        2019/12/12_16:08:24
> > > > INFO: Parameter authentication is now '0'.
> > > > Dec 12 16:08:24 [7053] ha-test1       lrmd:     info:
> > > > log_finished:     finished - rsc:iscsi-server action:start
> > > > call_id:63 pid:4564 exit-code:0 exec-time:1781ms queue-time:0ms
> > > > Dec 12 16:08:24 [7056] ha-test1       crmd:     info:
> > > > action_synced_wait:       Managed iSCSITarget_meta-data_0
> > > > process 4695 exited with rc=0
> > > > Dec 12 16:08:24 [7051] ha-test1        cib:     info:
> > > > cib_process_request:      Forwarding cib_modify operation for
> > > > section status to all (origin=local/crmd/59)
> > > > Dec 12 16:08:24 [7056] ha-test1       crmd:   notice:
> > > > process_lrm_event:        Result of start operation for iscsi-
> > > > server on ha-test1: 0 (ok) | call=63 key=iscsi-server_start_0
> > > > confirmed=true cib-update=59
> > > > Dec 12 16:08:24 [7051] ha-test1        cib:     info:
> > > > cib_perform_op:   Diff: --- 0.59.1 2
> > > > Dec 12 16:08:24 [7051] ha-test1        cib:     info:
> > > > cib_perform_op:   Diff: +++ 0.59.2 (null)
> > > > Dec 12 16:08:24 [7051] ha-test1        cib:     info:
> > > > cib_perform_op:   +  /cib:  @num_updates=2
> > > > Dec 12 16:08:24 [7051] ha-test1        cib:     info:
> > > > cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1
> > > > ']/lrm_resources/lrm_resource[@id='iscsi-
> > > > server']/lrm_rsc_op[@id='iscsi-
> > > > server_last_0']:  @operation_key=iscsi-server_start_0,
> > > > @operation=start, @transition-key=3:116:0:9fd4e826-f0ba-4864-
> > > > 8861-2c585d644d1c, @transition-magic=0:0;3:116:0:9fd4e826-f0ba-
> > > > 4864-8861-2c585d644d1c, @call-id=63, @last-run=1576163302,
> > > > @last-rc-change=1576163302, @exec-time=1781, @op-
> > > > digest=549b6bf42c2d944da4df2c1d2d675b
> > > > Dec 12 16:08:24 [7051] ha-test1        cib:     info:
> > > > cib_process_request:      Completed cib_modify operation for
> > > > section status: OK (rc=0, origin=ha-test1/crmd/59,
> > > > version=0.59.2)
> > > > Dec 12 16:08:29 [7051] ha-test1        cib:     info:
> > > > cib_process_ping: Reporting our current digest to ha-test2:
> > > > 8bf64451f91add76d89a608b6f51a214 for 0.59.2 (0x55998d145df0 0)
> > > > 
> > > > 
> > > > 
> > > > 
> > > > On Wednesday, December 11, 2019 3:58:51 PM CET Stefan K wrote:
> > > > > Hello,
> > > > > 
> > > > > I've a working HA-Setup with iSCSI an ZFS, but last week I
> > > > > add an iSCSI allowed initiator, and than it happens - my hole
> > > > > VMware infrastructure fails because iSCSI does not working
> > > > > anymore.. today I've time to get a closer look into this..
> > > > > 
> > > > > create 2 VMs an put the same (more or less) config into it.
> > > > > What I do:
> > > > > - I create a iSCSI-target with allowed initiators
> > > > > - I create iSCSI Logical units
> > > > > 
> > > > > but I got this:
> > > > > 
> > > > > targetcli
> > > > > targetcli shell version 2.1.fb43
> > > > > Copyright 2011-2013 by Datera, Inc and others.
> > > > > For help on commands, type 'help'.
> > > > > 
> > > > > /> ls
> > > > > o- /
> > > > > .............................................................
> > > > > ............................................................
> > > > > [...]
> > > > >   o- backstores
> > > > > .............................................................
> > > > > ................................................. [...]
> > > > >   | o- block
> > > > > .............................................................
> > > > > ..................................... [Storage Objects: 3]
> > > > >   | | o- iscsi-lun00
> > > > > .............................................................
> > > > > ..... [/dev/loop1 (1.0GiB) write-thru deactivated]
> > > > >   | | o- iscsi-lun01
> > > > > .............................................................
> > > > > ..... [/dev/loop2 (1.0GiB) write-thru deactivated]
> > > > >   | | o- iscsi-lun02
> > > > > .............................................................
> > > > > .... [/dev/loop3 (0 bytes) write-thru deactivated]
> > > > >   | o- fileio
> > > > > .............................................................
> > > > > .................................... [Storage Objects: 0]
> > > > >   | o- pscsi
> > > > > .............................................................
> > > > > ..................................... [Storage Objects: 0]
> > > > >   | o- ramdisk
> > > > > .............................................................
> > > > > ................................... [Storage Objects: 0]
> > > > >   o- iscsi
> > > > > .............................................................
> > > > > ............................................... [Targets: 1]
> > > > >   | o- iqn.2003-01.org.linux-iscsi.vm-
> > > > > storage.x8664:sn.cf6fa66tgyh3
> > > > > ...................................................... [TPGs:
> > > > > 1]
> > > > >   |   o- tpg1
> > > > > .............................................................
> > > > > .................................. [no-gen-acls, no-auth]
> > > > >   |     o- acls
> > > > > .............................................................
> > > > > ............................................. [ACLs: 4]
> > > > >   |     | o- iqn.1993-08.org.debian:01:fee35be01c4d
> > > > > .............................................................
> > > > > .. [Mapped LUNs: 0]
> > > > >   |     | o- iqn.1998-01.com.vmware:brainslug10-34ad648763
> > > > > ........................................................
> > > > > [Mapped LUNs: 0]
> > > > >   |     | o- iqn.1998-01.com.vmware:brainslug10-5564u4325
> > > > > .........................................................
> > > > > [Mapped LUNs: 0]
> > > > >   |     | o- iqn.1998-01.com.vmware:brainslug9-75488e35
> > > > > ...........................................................
> > > > > [Mapped LUNs: 0]
> > > > >   |     o- luns
> > > > > .............................................................
> > > > > ............................................. [LUNs: 0]
> > > > >   |     o- portals
> > > > > .............................................................
> > > > > ....................................... [Portals: 1]
> > > > >   |       o- 172.16.101.166:3260
> > > > > .............................................................
> > > > > ................................. [OK]
> > > > >   o- loopback
> > > > > .............................................................
> > > > > ............................................ [Targets: 0]
> > > > >   o- vhost
> > > > > .............................................................
> > > > > ............................................... [Targets: 0]
> > > > > 
> > > > > 
> > > > > here you can see there are missing luns, when I move the
> > > > > ressource to the other node, it will shown the luns, if I
> > > > > then add/remove/change an "allowed_initiators" it will happen
> > > > > again - all luns are gone. And that is a very serious problem
> > > > > for us.
> > > > > 
> > > > > So my questions is, do i misconfigure something or is that a
> > > > > bug? My pacemaker config looks like the following:
> > > > > 
> > > > > crm conf sh
> > > > > node 1: ha-test1 \
> > > > >         attributes \
> > > > >         attributes standby=off maintenance=off
> > > > > node 2: ha-test2 \
> > > > >         attributes \
> > > > >         attributes standby=off
> > > > > primitive ha-ip IPaddr2 \
> > > > >         params ip=172.16.101.166 cidr_netmask=24 nic=ens192 \
> > > > >         op start interval=0s timeout=20s \
> > > > >         op stop interval=0s timeout=20s \
> > > > >         op monitor interval=10s timeout=20s \
> > > > >         meta target-role=Started
> > > > > primitive iscsi-lun00 iSCSILogicalUnit \
> > > > >         params implementation=lio-t target_iqn="iqn.2003-
> > > > > 01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3" lun=0
> > > > > lio_iblock=0 path="/dev/loop1" \
> > > > >         op start interval=0 trace_ra=1 \
> > > > >         op stop interval=0 trace_ra=1 \
> > > > >         meta target-role=Started
> > > > > primitive iscsi-lun01 iSCSILogicalUnit \
> > > > >         params implementation=lio-t target_iqn="iqn.2003-
> > > > > 01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3" lun=1
> > > > > lio_iblock=1 path="/dev/loop2" \
> > > > >         meta
> > > > > primitive iscsi-lun02 iSCSILogicalUnit \
> > > > >         params implementation=lio-t target_iqn="iqn.2003-
> > > > > 01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3" lun=2
> > > > > lio_iblock=2 path="/dev/loop3" \
> > > > >         meta
> > > > > primitive iscsi-server iSCSITarget \
> > > > >         params implementation=lio-t iqn="iqn.2003-
> > > > > 01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3"
> > > > > portals="172.16.101.166:3260" allowed_initiators="iqn.1998-
> > > > > 01.com.vmware:brainslug9-75488e35 iqn.1998-
> > > > > 01.com.vmware:brainslug10-5564u4325 iqn.1993-
> > > > > 08.org.debian:01:fee35be01c4d iqn.1998-
> > > > > 01.com.vmware:brainslug10-34ad648763" \
> > > > >         meta
> > > > > colocation pcs_rsc_colocation_set_ha-ip_vm_storage_iscsi-
> > > > > server inf: ha-ip iscsi-server iscsi-lun00 iscsi-lun01 iscsi-
> > > > > lun02
> > > > > order pcs_rsc_order_set_ha-ip_iscsi-server_vm_storage ha-
> > > > > ip:stop iscsi-lun00:stop iscsi-lun01:stop iscsi-lun02:stop
> > > > > iscsi-server:stop symmetrical=false
> > > > > order pcs_rsc_order_set_iscsi-server_vm_storage_ha-ip iscsi-
> > > > > lun00:start iscsi-server:start iscsi-lun01:start iscsi-
> > > > > lun02:start ha-ip:start symmetrical=false
> > > > > property cib-bootstrap-options: \
> > > > >         have-watchdog=false \
> > > > >         dc-version=1.1.16-94ff4df \
> > > > >         cluster-infrastructure=corosync \
> > > > >         cluster-name=ha-vmstorage \
> > > > >         no-quorum-policy=stop \
> > > > >         stonith-enabled=false \
> > > > >         last-lrm-refresh=1576056627
> > > > > rsc_defaults rsc_defaults-options: \
> > > > >         resource-stickiness=100
> > > > > 
> > > > > 
> > > > > The system is running on Debian Stretch.
> > > > > 
> > > > > Thank you very much for you help!
> > > > > 
> > > > > best regards
> > > > > Stefan
> > > > > 
> > > > > 
> > > > > _______________________________________________
> > > > > Manage your subscription:
> > > > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > > > 
> > > > > ClusterLabs home: https://www.clusterlabs.org/
> > > > > 
> > > > 
> > > > 
> > > > 
> > > > _______________________________________________
> > > > Manage your subscription:
> > > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > > 
> > > > ClusterLabs home: https://www.clusterlabs.org/
> > > > 
> > > 
> > > 
> > > 
> > > _______________________________________________
> > > Manage your subscription:
> > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > 
> > > ClusterLabs home: https://www.clusterlabs.org/
> > > 
> > 
> > _______________________________________________
> > Manage your subscription:
> > https://lists.clusterlabs.org/mailman/listinfo/users
> > 
> > ClusterLabs home: https://www.clusterlabs.org/
> 
> 
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> ClusterLabs home: https://www.clusterlabs.org/
-- 
Ken Gaillot <kgaillot at redhat.com>



More information about the Users mailing list