<div dir="ltr"><div><div>Hey John.</div><div><br></div>I also encountered the same error message "ERROR: This Target already exists in configFS" a while back and when I issued targetcli and listed it configuration contents I could see the target in iscsi folder. And that was due to a force reboot of the node.</div><div><br></div><div>To solve it I've made an workaround by adding the following line "ocf_run targetcli /iscsi delete ${OCF_RESKEY_iqn}" in /usr/lib/ocf/resource.d/heartbeat/iSCSITarget at line 330 just before "ocf_run targetcli /iscsi create ${OCF_RESKEY_iqn} || exit $OCF_ERR_GENERIC". That command will delete the target to be created if already exists.</div><div><br></div><div>I hope this workaround will help you with your issue until a valid solution is available.</div><div><br></div><div>Best regards</div><div>Octavian Ciobanu<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 22, 2017 at 12:19 AM, John Keates <span dir="ltr"><<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I have a strange issue where LIO-T based ISCSI targets and LUNs most of the time simply don’t work. They either don’t start, or bounce around until no more nodes are tried.<br>
The less-than-usefull information on the logs is like:<br>
<br>
Aug 21 22:49:06 [10531] storage-1-prod pengine: warning: check_migration_threshold: Forcing iscsi0-target away from storage-1-prod after 1000000 failures (max=1000000)<br>
<br>
Aug 21 22:54:47 storage-1-prod crmd[2757]: notice: Result of start operation for ip-iscsi0-vlan40 on storage-1-prod: 0 (ok)<br>
Aug 21 22:54:47 storage-1-prod iSCSITarget(iscsi0-target)[<wbr>5427]: WARNING: Configuration parameter "tid" is not supported by the iSCSI implementation and will be ignored.<br>
Aug 21 22:54:48 storage-1-prod iSCSITarget(iscsi0-target)[<wbr>5427]: INFO: Parameter auto_add_default_portal is now 'false'.<br>
Aug 21 22:54:48 storage-1-prod iSCSITarget(iscsi0-target)[<wbr>5427]: INFO: Created target iqn.2017-08.acccess.net:prod-<wbr>1-ha. Created TPG 1.<br>
Aug 21 22:54:48 storage-1-prod iSCSITarget(iscsi0-target)[<wbr>5427]: ERROR: This Target already exists in configFS<br>
Aug 21 22:54:48 storage-1-prod crmd[2757]: notice: Result of start operation for iscsi0-target on storage-1-prod: 1 (unknown error)<br>
Aug 21 22:54:49 storage-1-prod iSCSITarget(iscsi0-target)[<wbr>5536]: INFO: Deleted Target iqn.2017-08.access.net:prod-1-<wbr>ha.<br>
Aug 21 22:54:49 storage-1-prod crmd[2757]: notice: Result of stop operation for iscsi0-target on storage-1-prod: 0 (ok)<br>
<br>
Now, the unknown error seems to actually be a targetcli type of error: "This Target already exists in configFS”. Checking with targetcli shows zero configured items on either node.<br>
Manually starting the LUNs and target gives:<br>
<br>
<br>
john@storage-1-prod:~$ sudo pcs resource debug-start iscsi0-target<br>
Error performing operation: Operation not permitted<br>
Operation start for iscsi0-target (ocf:heartbeat:iSCSITarget) returned 1<br>
> stderr: WARNING: Configuration parameter "tid" is not supported by the iSCSI implementation and will be ignored.<br>
> stderr: INFO: Parameter auto_add_default_portal is now 'false'.<br>
> stderr: INFO: Created target iqn.2017-08.access.net:prod-1-<wbr>ha. Created TPG 1.<br>
> stderr: ERROR: This Target already exists in configFS<br>
<br>
but now targetcli shows at least the target. Checking with crm status still shows the target as stopped.<br>
Manually starting the LUNs gives:<br>
<br>
<br>
john@storage-1-prod:~$ sudo pcs resource debug-start iscsi0-lun0<br>
Operation start for iscsi0-lun0 (ocf:heartbeat:<wbr>iSCSILogicalUnit) returned 0<br>
> stderr: INFO: Created block storage object iscsi0-lun0 using /dev/zvol/iscsipool0/iscsi/<wbr>net.access.prod-1-ha-root.<br>
> stderr: INFO: Created LUN 0.<br>
> stderr: DEBUG: iscsi0-lun0 start : 0<br>
john@storage-1-prod:~$ sudo pcs resource debug-start iscsi0-lun1<br>
Operation start for iscsi0-lun1 (ocf:heartbeat:<wbr>iSCSILogicalUnit) returned 0<br>
> stderr: INFO: Created block storage object iscsi0-lun1 using /dev/zvol/iscsipool0/iscsi/<wbr>net.access.prod-1-ha-swap.<br>
> stderr: /usr/lib/ocf/resource.d/<wbr>heartbeat/iSCSILogicalUnit: line 378: /sys/kernel/config/target/<wbr>core/iblock_0/iscsi0-lun1/wwn/<wbr>vpd_unit_serial: No such file or directory<br>
> stderr: INFO: Created LUN 1.<br>
> stderr: DEBUG: iscsi0-lun1 start : 0<br>
<br>
So the second LUN seems to have some bad parameters created by the iSCSILogicalUnit script. Checking with targetcli however shows both LUNs and the target up and running.<br>
Checking again with crm status (and pcs status) shows all three resources still stopped. Since LUNs are colocated with the target and the target still has fail counts, I clear them with:<br>
<br>
sudo pcs resource cleanup iscsi0-target<br>
<br>
Now the LUNs and target are all active in crm status / pcs status. But it’s quite a manual process to get this to work! I’m thinking either my configuration is bad or there is some bug somewhere in targetcli / LIO or the iSCSI heartbeat script.<br>
On top of all the manual work, it still breaks on any action. A move, failover, reboot etc. instantly breaks it. Everything else (the underlying ZFS Pool, the DRBD device, the IPv4 IP’s etc) moves just fine, it’s only the ISCSI that’s being problematic.<br>
<br>
Concrete questions:<br>
<br>
- Is my config bad?<br>
- Is there a known issue with ISCSI? (I have only found old references about ordering)<br>
<br>
I have added the output of crm config show as cib.txt and the output of a fresh boot of both nodes is:<br>
<br>
Current DC: storage-2-prod (version 1.1.16-94ff4df) - partition with quorum<br>
Last updated: Mon Aug 21 22:55:05 2017<br>
Last change: Mon Aug 21 22:36:23 2017 by root via cibadmin on storage-1-prod<br>
<br>
2 nodes configured<br>
21 resources configured<br>
<br>
Online: [ storage-1-prod storage-2-prod ]<br>
<br>
Full list of resources:<br>
<br>
ip-iscsi0-vlan10 (ocf::heartbeat:IPaddr2): Started storage-1-prod<br>
ip-iscsi0-vlan20 (ocf::heartbeat:IPaddr2): Started storage-1-prod<br>
ip-iscsi0-vlan30 (ocf::heartbeat:IPaddr2): Started storage-1-prod<br>
ip-iscsi0-vlan40 (ocf::heartbeat:IPaddr2): Started storage-1-prod<br>
Master/Slave Set: drbd_master_slave0 [drbd_disk0]<br>
Masters: [ storage-1-prod ]<br>
Slaves: [ storage-2-prod ]<br>
Master/Slave Set: drbd_master_slave1 [drbd_disk1]<br>
Masters: [ storage-2-prod ]<br>
Slaves: [ storage-1-prod ]<br>
ip-iscsi1-vlan10 (ocf::heartbeat:IPaddr2): Started storage-2-prod<br>
ip-iscsi1-vlan20 (ocf::heartbeat:IPaddr2): Started storage-2-prod<br>
ip-iscsi1-vlan30 (ocf::heartbeat:IPaddr2): Started storage-2-prod<br>
ip-iscsi1-vlan40 (ocf::heartbeat:IPaddr2): Started storage-2-prod<br>
st-storage-1-prod (stonith:meatware): Started storage-2-prod<br>
st-storage-2-prod (stonith:meatware): Started storage-1-prod<br>
zfs-iscsipool0 (ocf::heartbeat:ZFS): Started storage-1-prod<br>
zfs-iscsipool1 (ocf::heartbeat:ZFS): Started storage-2-prod<br>
iscsi0-lun0 (ocf::heartbeat:<wbr>iSCSILogicalUnit): Stopped<br>
iscsi0-lun1 (ocf::heartbeat:<wbr>iSCSILogicalUnit): Stopped<br>
iscsi0-target (ocf::heartbeat:iSCSITarget): Stopped<br>
Clone Set: dlm-clone [dlm]<br>
Started: [ storage-1-prod storage-2-prod ]<br>
<br>
Failed Actions:<br>
* iscsi0-target_start_0 on storage-2-prod 'unknown error' (1): call=99, status=complete, exitreason='none',<br>
last-rc-change='Mon Aug 21 22:54:49 2017', queued=0ms, exec=954ms<br>
* iscsi0-target_start_0 on storage-1-prod 'unknown error' (1): call=98, status=complete, exitreason='none',<br>
last-rc-change='Mon Aug 21 22:54:47 2017', queued=0ms, exec=1062ms<br>
<br>
Regards,<br>
John<br>
<br>______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br></blockquote></div><br></div>