<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hey Octavian,<div class=""><br class=""></div><div class="">I did that too, and it worked. Afterwards, I did some more checking to see why the control flow ended up there, since the script works with tgtd and LIO (non -T / fe version).</div><div class="">It seems it has a slight implementation issue where the for loop fails if there are no portals listed or the one(s) listed is equal to the default value. Setting only one portal doesn’t help either for some reason (but I don’t see why, the loop would run exactly once). When I configure 2 or more target portals manually it works just fine. </div><div class=""><br class=""></div><div class="">This is something I can work around since I use SaltStack to configure the CIB, so I just pull the IP addresses off of the interfaces I want to use, and stick them in a space delimited list as the portals.</div><div class=""><br class=""></div><div class="">I created an issue about it at <a href="https://github.com/ClusterLabs/resource-agents/issues/1026" class="">https://github.com/ClusterLabs/resource-agents/issues/1026</a></div><div class=""><br class=""></div><div class="">At the same time, I’m still not sure about my configuration. It seems I have way more locations, colocation and ordering constraints than most configs I come across.</div><div class=""><br class=""><div class="">
Kind regards,<br class="">John Keates<br class=""><br class=""></div>
<div><br class=""><blockquote type="cite" class=""><div class="">On 26 Aug 2017, at 14:41, Octavian Ciobanu <<a href="mailto:coctavian1979@gmail.com" class="">coctavian1979@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class=""><div class="">Hey John.</div><div class=""><br class=""></div>I also encountered the same error message "ERROR: This Target already exists in configFS" a while back and when I issued targetcli and listed it configuration contents I could see the target in iscsi folder. And that was due to a force reboot of the node.</div><div class=""><br class=""></div><div class="">To solve it I've made an workaround by adding the following line "ocf_run targetcli /iscsi delete ${OCF_RESKEY_iqn}" in /usr/lib/ocf/resource.d/heartbeat/iSCSITarget at line 330 just before "ocf_run targetcli /iscsi create ${OCF_RESKEY_iqn} || exit $OCF_ERR_GENERIC". That command will delete the target to be created if already exists.</div><div class=""><br class=""></div><div class="">I hope this workaround will help you with your issue until a valid solution is available.</div><div class=""><br class=""></div><div class="">Best regards</div><div class="">Octavian Ciobanu<br class=""></div></div><div class="gmail_extra" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class=""><div class="gmail_quote">On Tue, Aug 22, 2017 at 12:19 AM, John Keates<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:john@keates.nl" target="_blank" class="">john@keates.nl</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">Hi,<br class=""><br class="">I have a strange issue where LIO-T based ISCSI targets and LUNs most of the time simply don’t work. They either don’t start, or bounce around until no more nodes are tried.<br class="">The less-than-usefull information on the logs is like:<br class=""><br class="">Aug 21 22:49:06 [10531] storage-1-prod pengine: warning: check_migration_threshold: Forcing iscsi0-target away from storage-1-prod after 1000000 failures (max=1000000)<br class=""><br class="">Aug 21 22:54:47 storage-1-prod crmd[2757]: notice: Result of start operation for ip-iscsi0-vlan40 on storage-1-prod: 0 (ok)<br class="">Aug 21 22:54:47 storage-1-prod iSCSITarget(iscsi0-target)[<wbr class="">5427]: WARNING: Configuration parameter "tid" is not supported by the iSCSI implementation and will be ignored.<br class="">Aug 21 22:54:48 storage-1-prod iSCSITarget(iscsi0-target)[<wbr class="">5427]: INFO: Parameter auto_add_default_portal is now 'false'.<br class="">Aug 21 22:54:48 storage-1-prod iSCSITarget(iscsi0-target)[<wbr class="">5427]: INFO: Created target <a href="http://iqn.2017-08.acccess.net" class="">iqn.2017-08.acccess.net</a>:prod-<wbr class="">1-ha. Created TPG 1.<br class="">Aug 21 22:54:48 storage-1-prod iSCSITarget(iscsi0-target)[<wbr class="">5427]: ERROR: This Target already exists in configFS<br class="">Aug 21 22:54:48 storage-1-prod crmd[2757]: notice: Result of start operation for iscsi0-target on storage-1-prod: 1 (unknown error)<br class="">Aug 21 22:54:49 storage-1-prod iSCSITarget(iscsi0-target)[<wbr class="">5536]: INFO: Deleted Target <a href="http://iqn.2017-08.access.net" class="">iqn.2017-08.access.net</a>:prod-1-<wbr class="">ha.<br class="">Aug 21 22:54:49 storage-1-prod crmd[2757]: notice: Result of stop operation for iscsi0-target on storage-1-prod: 0 (ok)<br class=""><br class="">Now, the unknown error seems to actually be a targetcli type of error: "This Target already exists in configFS”. Checking with targetcli shows zero configured items on either node.<br class="">Manually starting the LUNs and target gives:<br class=""><br class=""><br class="">john@storage-1-prod:~$ sudo pcs resource debug-start iscsi0-target<br class="">Error performing operation: Operation not permitted<br class="">Operation start for iscsi0-target (ocf:heartbeat:iSCSITarget) returned 1<br class=""> > stderr: WARNING: Configuration parameter "tid" is not supported by the iSCSI implementation and will be ignored.<br class=""> > stderr: INFO: Parameter auto_add_default_portal is now 'false'.<br class=""> > stderr: INFO: Created target <a href="http://iqn.2017-08.access.net" class="">iqn.2017-08.access.net</a>:prod-1-<wbr class="">ha. Created TPG 1.<br class=""> > stderr: ERROR: This Target already exists in configFS<br class=""><br class="">but now targetcli shows at least the target. Checking with crm status still shows the target as stopped.<br class="">Manually starting the LUNs gives:<br class=""><br class=""><br class="">john@storage-1-prod:~$ sudo pcs resource debug-start iscsi0-lun0<br class="">Operation start for iscsi0-lun0 (ocf:heartbeat:<wbr class="">iSCSILogicalUnit) returned 0<br class=""> > stderr: INFO: Created block storage object iscsi0-lun0 using /dev/zvol/iscsipool0/iscsi/<wbr class="">net.access.prod-1-ha-root.<br class=""> > stderr: INFO: Created LUN 0.<br class=""> > stderr: DEBUG: iscsi0-lun0 start : 0<br class="">john@storage-1-prod:~$ sudo pcs resource debug-start iscsi0-lun1<br class="">Operation start for iscsi0-lun1 (ocf:heartbeat:<wbr class="">iSCSILogicalUnit) returned 0<br class=""> > stderr: INFO: Created block storage object iscsi0-lun1 using /dev/zvol/iscsipool0/iscsi/<wbr class="">net.access.prod-1-ha-swap.<br class=""> > stderr: /usr/lib/ocf/resource.d/<wbr class="">heartbeat/iSCSILogicalUnit: line 378: /sys/kernel/config/target/<wbr class="">core/iblock_0/iscsi0-lun1/wwn/<wbr class="">vpd_unit_serial: No such file or directory<br class=""> > stderr: INFO: Created LUN 1.<br class=""> > stderr: DEBUG: iscsi0-lun1 start : 0<br class=""><br class="">So the second LUN seems to have some bad parameters created by the iSCSILogicalUnit script. Checking with targetcli however shows both LUNs and the target up and running.<br class="">Checking again with crm status (and pcs status) shows all three resources still stopped. Since LUNs are colocated with the target and the target still has fail counts, I clear them with:<br class=""><br class="">sudo pcs resource cleanup iscsi0-target<br class=""><br class="">Now the LUNs and target are all active in crm status / pcs status. But it’s quite a manual process to get this to work! I’m thinking either my configuration is bad or there is some bug somewhere in targetcli / LIO or the iSCSI heartbeat script.<br class="">On top of all the manual work, it still breaks on any action. A move, failover, reboot etc. instantly breaks it. Everything else (the underlying ZFS Pool, the DRBD device, the IPv4 IP’s etc) moves just fine, it’s only the ISCSI that’s being problematic.<br class=""><br class="">Concrete questions:<br class=""><br class="">- Is my config bad?<br class="">- Is there a known issue with ISCSI? (I have only found old references about ordering)<br class=""><br class="">I have added the output of crm config show as cib.txt and the output of a fresh boot of both nodes is:<br class=""><br class="">Current DC: storage-2-prod (version 1.1.16-94ff4df) - partition with quorum<br class="">Last updated: Mon Aug 21 22:55:05 2017<br class="">Last change: Mon Aug 21 22:36:23 2017 by root via cibadmin on storage-1-prod<br class=""><br class="">2 nodes configured<br class="">21 resources configured<br class=""><br class="">Online: [ storage-1-prod storage-2-prod ]<br class=""><br class="">Full list of resources:<br class=""><br class=""> ip-iscsi0-vlan10 (ocf::heartbeat:IPaddr2): Started storage-1-prod<br class=""> ip-iscsi0-vlan20 (ocf::heartbeat:IPaddr2): Started storage-1-prod<br class=""> ip-iscsi0-vlan30 (ocf::heartbeat:IPaddr2): Started storage-1-prod<br class=""> ip-iscsi0-vlan40 (ocf::heartbeat:IPaddr2): Started storage-1-prod<br class=""> Master/Slave Set: drbd_master_slave0 [drbd_disk0]<br class=""> Masters: [ storage-1-prod ]<br class=""> Slaves: [ storage-2-prod ]<br class=""> Master/Slave Set: drbd_master_slave1 [drbd_disk1]<br class=""> Masters: [ storage-2-prod ]<br class=""> Slaves: [ storage-1-prod ]<br class=""> ip-iscsi1-vlan10 (ocf::heartbeat:IPaddr2): Started storage-2-prod<br class=""> ip-iscsi1-vlan20 (ocf::heartbeat:IPaddr2): Started storage-2-prod<br class=""> ip-iscsi1-vlan30 (ocf::heartbeat:IPaddr2): Started storage-2-prod<br class=""> ip-iscsi1-vlan40 (ocf::heartbeat:IPaddr2): Started storage-2-prod<br class=""> st-storage-1-prod (stonith:meatware): Started storage-2-prod<br class=""> st-storage-2-prod (stonith:meatware): Started storage-1-prod<br class=""> zfs-iscsipool0 (ocf::heartbeat:ZFS): Started storage-1-prod<br class=""> zfs-iscsipool1 (ocf::heartbeat:ZFS): Started storage-2-prod<br class=""> iscsi0-lun0 (ocf::heartbeat:<wbr class="">iSCSILogicalUnit): Stopped<br class=""> iscsi0-lun1 (ocf::heartbeat:<wbr class="">iSCSILogicalUnit): Stopped<br class=""> iscsi0-target (ocf::heartbeat:iSCSITarget): Stopped<br class=""> Clone Set: dlm-clone [dlm]<br class=""> Started: [ storage-1-prod storage-2-prod ]<br class=""><br class="">Failed Actions:<br class="">* iscsi0-target_start_0 on storage-2-prod 'unknown error' (1): call=99, status=complete, exitreason='none',<br class=""> <span class="Apple-converted-space"> </span>last-rc-change='Mon Aug 21 22:54:49 2017', queued=0ms, exec=954ms<br class="">* iscsi0-target_start_0 on storage-1-prod 'unknown error' (1): call=98, status=complete, exitreason='none',<br class=""> <span class="Apple-converted-space"> </span>last-rc-change='Mon Aug 21 22:54:47 2017', queued=0ms, exec=1062ms<br class=""><br class="">Regards,<br class="">John<br class=""><br class="">______________________________<wbr class="">_________________<br class="">Users mailing list:<span class="Apple-converted-space"> </span><a href="mailto:Users@clusterlabs.org" class="">Users@clusterlabs.org</a><br class=""><a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank" class="">http://lists.clusterlabs.org/<wbr class="">mailman/listinfo/users</a><br class=""><br class="">Project Home:<span class="Apple-converted-space"> </span><a href="http://www.clusterlabs.org/" rel="noreferrer" target="_blank" class="">http://www.clusterlabs.org</a><br class="">Getting started:<span class="Apple-converted-space"> </span><a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank" class="">http://www.clusterlabs.org/<wbr class="">doc/Cluster_from_Scratch.pdf</a><br class="">Bugs:<span class="Apple-converted-space"> </span><a href="http://bugs.clusterlabs.org/" rel="noreferrer" target="_blank" class="">http://bugs.clusterlabs.org</a><br class=""><br class=""></blockquote></div><br class=""></div><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Users mailing list:<span class="Apple-converted-space"> </span></span><a href="mailto:Users@clusterlabs.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Users@clusterlabs.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="http://lists.clusterlabs.org/mailman/listinfo/users" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://lists.clusterlabs.org/mailman/listinfo/users</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Project Home:<span class="Apple-converted-space"> </span></span><a href="http://www.clusterlabs.org/" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://www.clusterlabs.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Getting started:<span class="Apple-converted-space"> </span></span><a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Bugs:<span class="Apple-converted-space"> </span></span><a href="http://bugs.clusterlabs.org/" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://bugs.clusterlabs.org</a></div></blockquote></div><br class=""></div></body></html>