[ClusterLabs] Major problem with iSCSITarget resource on top of DRBD M/S resource.

Alex Crow alex at nanogherkin.com
Sun Sep 27 14:40:01 UTC 2015 

Hi List,

I'm trying to set up a failover iSCSI storage system for oVirt using a 
self-hosted engine. I've set up DRBD in Master-Slave for two iSCSI 
targets, one for the self-hosted engine and one for the VMs. I has this 
all working perfectly, then after trying to move the engine's LUN to the 
opposite host, all hell broke loose. The VMS LUN is still fine, starts 
and migrates as it should. However the engine LUN always seems to try to 
launch the target on the host that is *NOT* the master of the DRBD 
resource. My constraints look fine, and should be self-explanatory about 
which is which:

[root at granby ~]# pcs constraint --full
Location Constraints:
Ordering Constraints:
   promote drbd-vms-iscsi then start iscsi-vms-ip (kind:Mandatory) 
(id:vm_iscsi_ip_after_drbd)
   start iscsi-vms-target then start iscsi-vms-lun (kind:Mandatory) 
(id:vms_lun_after_target)
   promote drbd-vms-iscsi then start iscsi-vms-target (kind:Mandatory) 
(id:vms_target_after_drbd)
   promote drbd-engine-iscsi then start iscsi-engine-ip (kind:Mandatory) 
(id:ip_after_drbd)
   start iscsi-engine-target then start iscsi-engine-lun 
(kind:Mandatory) (id:lun_after_target)
   promote drbd-engine-iscsi then start iscsi-engine-target 
(kind:Mandatory) (id:target_after_drbd)
Colocation Constraints:
   iscsi-vms-ip with drbd-vms-iscsi (score:INFINITY) (rsc-role:Started) 
(with-rsc-role:Master) (id:vms_ip-with-drbd)
   iscsi-vms-lun with drbd-vms-iscsi (score:INFINITY) (rsc-role:Started) 
(with-rsc-role:Master) (id:vms_lun-with-drbd)
   iscsi-vms-target with drbd-vms-iscsi (score:INFINITY) 
(rsc-role:Started) (with-rsc-role:Master) (id:vms_target-with-drbd)
   iscsi-engine-ip with drbd-engine-iscsi (score:INFINITY) 
(rsc-role:Started) (with-rsc-role:Master) (id:ip-with-drbd)
   iscsi-engine-lun with drbd-engine-iscsi (score:INFINITY) 
(rsc-role:Started) (with-rsc-role:Master) (id:lun-with-drbd)
   iscsi-engine-target with drbd-engine-iscsi (score:INFINITY) 
(rsc-role:Started) (with-rsc-role:Master) (id:target-with-drbd)

But see this from pcs status, the iSCSI target has FAILED on glenrock, 
but the DRBD master is on granby!:

[root at granby ~]# pcs status
Cluster name: storage
Last updated: Sun Sep 27 15:30:08 2015
Last change: Sun Sep 27 15:20:58 2015
Stack: cman
Current DC: glenrock - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured
10 Resources configured


Online: [ glenrock granby ]

Full list of resources:

  Master/Slave Set: drbd-vms-iscsi [drbd-vms]
      Masters: [ glenrock ]
      Slaves: [ granby ]
  iscsi-vms-target    (ocf::heartbeat:iSCSITarget): Started glenrock
  iscsi-vms-lun    (ocf::heartbeat:iSCSILogicalUnit): Started glenrock
  iscsi-vms-ip    (ocf::heartbeat:IPaddr2):    Started glenrock
  Master/Slave Set: drbd-engine-iscsi [drbd-engine]
      Masters: [ granby ]
      Slaves: [ glenrock ]
  iscsi-engine-target    (ocf::heartbeat:iSCSITarget): FAILED glenrock 
(unmanaged)
  iscsi-engine-ip    (ocf::heartbeat:IPaddr2):    Stopped
  iscsi-engine-lun    (ocf::heartbeat:iSCSILogicalUnit): Stopped

Failed actions:
     iscsi-engine-target_stop_0 on glenrock 'unknown error' (1): 
call=177, status=Timed Out, last-rc-change='Sun Sep 27 15:20:59 2015', 
queued=0ms, exec=10003ms
     iscsi-engine-target_stop_0 on glenrock 'unknown error' (1): 
call=177, status=Timed Out, last-rc-change='Sun Sep 27 15:20:59 2015', 
queued=0ms, exec=10003ms

I have tried various combinations of pcs resource clear and cleanup, but 
that all result in the same outcome - apart from on some occasions when 
one or other of the two hosts suddenly reboots!

Here is a log right after a "pcs resource cleanup" - first on the master 
for the DRBD m/s resource:
[root at granby ~]# pcs resource cleanup; tail -f /var/log/messages
All resources/stonith devices successfully cleaned up
Sep 27 15:33:42 granby crmd[3358]:   notice: process_lrm_event: 
granby-drbd-engine_monitor_0:117 [ \n ]
Sep 27 15:33:42 granby attrd[3356]:   notice: attrd_trigger_update: 
Sending flush op to all hosts for: probe_complete (true)
Sep 27 15:33:42 granby attrd[3356]:   notice: attrd_perform_update: Sent 
update 54: probe_complete=true
Sep 27 15:33:42 granby crmd[3358]:   notice: process_lrm_event: 
Operation drbd-engine_monitor_10000: master (node=granby, call=131, 
rc=8, cib-update=83, confirmed=false)
Sep 27 15:33:42 granby crmd[3358]:   notice: process_lrm_event: 
granby-drbd-engine_monitor_10000:131 [ \n ]
Sep 27 15:33:42 granby crmd[3358]:   notice: process_lrm_event: 
Operation drbd-vms_monitor_20000: ok (node=granby, call=130, rc=0, 
cib-update=84, confirmed=false)
Sep 27 15:34:46 granby crmd[3358]:   notice: do_lrm_invoke: Forcing the 
status of all resources to be redetected
Sep 27 15:34:46 granby attrd[3356]:   notice: attrd_trigger_update: 
Sending flush op to all hosts for: probe_complete (<null>)
Sep 27 15:34:46 granby crmd[3358]:   notice: process_lrm_event: 
granby-drbd-engine_monitor_10000:131 [ \n ]
Sep 27 15:34:46 granby attrd[3356]:   notice: attrd_perform_update: Sent 
delete 61: node=granby, attr=probe_complete, id=<n/a>, set=(null), 
section=status
Sep 27 15:34:46 granby attrd[3356]:   notice: attrd_perform_update: Sent 
delete 63: node=granby, attr=probe_complete, id=<n/a>, set=(null), 
section=status
Sep 27 15:34:46 granby crmd[3358]:   notice: process_lrm_event: 
Operation iscsi-vms-target_monitor_0: not running (node=granby, 
call=150, rc=7, cib-update=94, confirmed=true)
Sep 27 15:34:46 granby crmd[3358]:   notice: process_lrm_event: 
Operation iscsi-vms-lun_monitor_0: not running (node=granby, call=154, 
rc=7, cib-update=95, confirmed=true)
Sep 27 15:34:46 granby crmd[3358]:   notice: process_lrm_event: 
Operation iscsi-engine-target_monitor_0: not running (node=granby, 
call=167, rc=7, cib-update=96, confirmed=true)
Sep 27 15:34:46 granby crmd[3358]:   notice: process_lrm_event: 
Operation iscsi-engine-lun_monitor_0: not running (node=granby, 
call=175, rc=7, cib-update=97, confirmed=true)
Sep 27 15:34:46 granby crmd[3358]:   notice: process_lrm_event: 
Operation iscsi-vms-ip_monitor_0: not running (node=granby, call=158, 
rc=7, cib-update=98, confirmed=true)
Sep 27 15:34:46 granby crmd[3358]:   notice: process_lrm_event: 
Operation iscsi-engine-ip_monitor_0: not running (node=granby, call=171, 
rc=7, cib-update=99, confirmed=true)
Sep 27 15:34:46 granby crmd[3358]:   notice: process_lrm_event: 
Operation drbd-vms_monitor_0: ok (node=granby, call=146, rc=0, 
cib-update=100, confirmed=true)
Sep 27 15:34:46 granby crmd[3358]:   notice: process_lrm_event: 
Operation drbd-engine_monitor_0: master (node=granby, call=163, rc=8, 
cib-update=101, confirmed=true)
Sep 27 15:34:46 granby crmd[3358]:   notice: process_lrm_event: 
granby-drbd-engine_monitor_0:163 [ \n ]
Sep 27 15:34:46 granby attrd[3356]:   notice: attrd_trigger_update: 
Sending flush op to all hosts for: probe_complete (true)
Sep 27 15:34:46 granby attrd[3356]:   notice: attrd_perform_update: Sent 
update 67: probe_complete=true
Sep 27 15:34:47 granby crmd[3358]:   notice: process_lrm_event: 
Operation drbd-vms_monitor_20000: ok (node=granby, call=176, rc=0, 
cib-update=102, confirmed=false)
Sep 27 15:34:47 granby crmd[3358]:   notice: process_lrm_event: 
Operation drbd-engine_monitor_10000: master (node=granby, call=177, 
rc=8, cib-update=103, confirmed=false)
Sep 27 15:34:47 granby crmd[3358]:   notice: process_lrm_event: 
granby-drbd-engine_monitor_10000:177 [ \n ]

Now on the slave:


[root at glenrock ~]# pcs resource cleanup; tail -f /var/log/messages
All resources/stonith devices successfully cleaned up
Sep 27 15:34:57 glenrock pengine[3365]:  warning: unpack_rsc_op_failure: 
Processing failed op stop for iscsi-engine-target on glenrock: unknown 
error (1)
Sep 27 15:34:57 glenrock pengine[3365]:  warning: 
common_apply_stickiness: Forcing iscsi-engine-target away from glenrock 
after 1000000 failures (max=1000000)
Sep 27 15:34:57 glenrock pengine[3365]:   notice: process_pe_message: 
Calculated Transition 50: /var/lib/pacemaker/pengine/pe-input-533.bz2
Sep 27 15:34:57 glenrock crmd[3366]:   notice: run_graph: Transition 50 
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, 
Source=/var/lib/pacemaker/pengine/pe-input-533.bz2): Complete
Sep 27 15:34:57 glenrock crmd[3366]:   notice: do_state_transition: 
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS 
cause=C_FSA_INTERNAL origin=notify_crmd ]
Sep 27 15:35:49 glenrock crmd[3366]:   notice: do_lrm_invoke: Forcing 
the status of all resources to be redetected
Sep 27 15:35:49 glenrock attrd[3364]:   notice: attrd_trigger_update: 
Sending flush op to all hosts for: probe_complete (<null>)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: process_lrm_event: 
glenrock-drbd-vms_monitor_10000:278 [ \n ]
Sep 27 15:35:49 glenrock crmd[3366]:  warning: qb_ipcs_event_sendv: 
new_event_notification (3366-853-16): Broken pipe (32)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: do_state_transition: 
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC 
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Sep 27 15:35:49 glenrock attrd[3364]:   notice: attrd_perform_update: 
Sent delete 195: node=glenrock, attr=probe_complete, id=<n/a>, 
set=(null), section=status
Sep 27 15:35:49 glenrock attrd[3364]:   notice: attrd_perform_update: 
Sent delete 197: node=glenrock, attr=probe_complete, id=<n/a>, 
set=(null), section=status
Sep 27 15:35:49 glenrock pengine[3365]:   notice: unpack_config: On loss 
of CCM Quorum: Ignore
Sep 27 15:35:49 glenrock pengine[3365]:  warning: 
common_apply_stickiness: Forcing iscsi-engine-target away from glenrock 
after 1000000 failures (max=1000000)
Sep 27 15:35:49 glenrock pengine[3365]:   notice: LogActions: Start   
drbd-vms:0#011(glenrock)
Sep 27 15:35:49 glenrock pengine[3365]:   notice: LogActions: Start   
drbd-vms:1#011(granby)
Sep 27 15:35:49 glenrock pengine[3365]:   notice: LogActions: Start   
drbd-engine:0#011(glenrock)
Sep 27 15:35:49 glenrock pengine[3365]:   notice: LogActions: Start   
drbd-engine:1#011(granby)
Sep 27 15:35:49 glenrock pengine[3365]:   notice: process_pe_message: 
Calculated Transition 51: /var/lib/pacemaker/pengine/pe-input-534.bz2
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 4: monitor drbd-vms:0_monitor_0 on glenrock (local)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 13: monitor drbd-vms:1_monitor_0 on granby
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 14: monitor iscsi-vms-target_monitor_0 on granby
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 5: monitor iscsi-vms-target_monitor_0 on glenrock (local)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 15: monitor iscsi-vms-lun_monitor_0 on granby
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 6: monitor iscsi-vms-lun_monitor_0 on glenrock (local)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 16: monitor iscsi-vms-ip_monitor_0 on granby
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 7: monitor iscsi-vms-ip_monitor_0 on glenrock (local)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 8: monitor drbd-engine:0_monitor_0 on glenrock (local)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 17: monitor drbd-engine:1_monitor_0 on granby
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 18: monitor iscsi-engine-target_monitor_0 on granby
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 9: monitor iscsi-engine-target_monitor_0 on glenrock 
(local)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 19: monitor iscsi-engine-ip_monitor_0 on granby
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 10: monitor iscsi-engine-ip_monitor_0 on glenrock (local)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 20: monitor iscsi-engine-lun_monitor_0 on granby
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 11: monitor iscsi-engine-lun_monitor_0 on glenrock (local)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: process_lrm_event: 
Operation iscsi-vms-target_monitor_0: ok (node=glenrock, call=305, rc=0, 
cib-update=377, confirmed=true)
Sep 27 15:35:49 glenrock crmd[3366]:  warning: status_from_rc: Action 5 
(iscsi-vms-target_monitor_0) on glenrock failed (target: 7 vs. rc: 0): Error
Sep 27 15:35:49 glenrock crmd[3366]:   notice: abort_transition_graph: 
Transition aborted by iscsi-vms-target_monitor_0 'create' on (null): 
Event failed (magic=0:0;5:51:7:4ac61a75-532d-45c4-b07c-eee753699adc, 
cib=0.374.121, source=match_graph_event:344, 0)
Sep 27 15:35:49 glenrock crmd[3366]:  warning: status_from_rc: Action 5 
(iscsi-vms-target_monitor_0) on glenrock failed (target: 7 vs. rc: 0): Error
Sep 27 15:35:49 glenrock crmd[3366]:   notice: process_lrm_event: 
Operation iscsi-vms-lun_monitor_0: ok (node=glenrock, call=309, rc=0, 
cib-update=378, confirmed=true)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: process_lrm_event: 
Operation iscsi-engine-target_monitor_0: ok (node=glenrock, call=322, 
rc=0, cib-update=379, confirmed=true)
Sep 27 15:35:49 glenrock crmd[3366]:  warning: status_from_rc: Action 6 
(iscsi-vms-lun_monitor_0) on glenrock failed (target: 7 vs. rc: 0): Error
Sep 27 15:35:49 glenrock crmd[3366]:  warning: status_from_rc: Action 6 
(iscsi-vms-lun_monitor_0) on glenrock failed (target: 7 vs. rc: 0): Error
Sep 27 15:35:49 glenrock crmd[3366]:  warning: status_from_rc: Action 9 
(iscsi-engine-target_monitor_0) on glenrock failed (target: 7 vs. rc: 
0): Error
Sep 27 15:35:49 glenrock crmd[3366]:  warning: status_from_rc: Action 9 
(iscsi-engine-target_monitor_0) on glenrock failed (target: 7 vs. rc: 
0): Error
Sep 27 15:35:49 glenrock crmd[3366]:   notice: process_lrm_event: 
Operation iscsi-engine-lun_monitor_0: not running (node=glenrock, 
call=330, rc=7, cib-update=380, confirmed=true)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: process_lrm_event: 
Operation iscsi-engine-ip_monitor_0: not running (node=glenrock, 
call=326, rc=7, cib-update=381, confirmed=true)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: process_lrm_event: 
Operation iscsi-vms-ip_monitor_0: ok (node=glenrock, call=313, rc=0, 
cib-update=382, confirmed=true)
Sep 27 15:35:49 glenrock crmd[3366]:  warning: status_from_rc: Action 13 
(drbd-vms:1_monitor_0) on granby failed (target: 7 vs. rc: 0): Error
Sep 27 15:35:49 glenrock crmd[3366]:  warning: status_from_rc: Action 13 
(drbd-vms:1_monitor_0) on granby failed (target: 7 vs. rc: 0): Error
Sep 27 15:35:49 glenrock crmd[3366]:  warning: status_from_rc: Action 17 
(drbd-engine:1_monitor_0) on granby failed (target: 7 vs. rc: 8): Error
Sep 27 15:35:49 glenrock crmd[3366]:  warning: status_from_rc: Action 17 
(drbd-engine:1_monitor_0) on granby failed (target: 7 vs. rc: 8): Error
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 12: probe_complete probe_complete-granby on granby - 
no waiting
Sep 27 15:35:49 glenrock crmd[3366]:  warning: status_from_rc: Action 7 
(iscsi-vms-ip_monitor_0) on glenrock failed (target: 7 vs. rc: 0): Error
Sep 27 15:35:49 glenrock crmd[3366]:  warning: status_from_rc: Action 7 
(iscsi-vms-ip_monitor_0) on glenrock failed (target: 7 vs. rc: 0): Error
Sep 27 15:35:49 glenrock crmd[3366]:   notice: abort_transition_graph: 
Transition aborted by status-granby-probe_complete, probe_complete=true: 
Transient attribute change (create cib=0.374.135, 
source=te_update_diff:391, 
path=/cib/status/node_state[@id='granby']/transient_attributes[@id='granby']/instance_attributes[@id='status-granby'], 
0)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: process_lrm_event: 
Operation drbd-engine_monitor_0: ok (node=glenrock, call=318, rc=0, 
cib-update=383, confirmed=true)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: process_lrm_event: 
Operation drbd-vms_monitor_0: master (node=glenrock, call=301, rc=8, 
cib-update=384, confirmed=true)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: process_lrm_event: 
glenrock-drbd-vms_monitor_0:301 [ \n ]
Sep 27 15:35:49 glenrock crmd[3366]:  warning: status_from_rc: Action 8 
(drbd-engine:0_monitor_0) on glenrock failed (target: 7 vs. rc: 0): Error
Sep 27 15:35:49 glenrock crmd[3366]:  warning: status_from_rc: Action 8 
(drbd-engine:0_monitor_0) on glenrock failed (target: 7 vs. rc: 0): Error
Sep 27 15:35:49 glenrock crmd[3366]:  warning: status_from_rc: Action 4 
(drbd-vms:0_monitor_0) on glenrock failed (target: 7 vs. rc: 8): Error
Sep 27 15:35:49 glenrock crmd[3366]:  warning: status_from_rc: Action 4 
(drbd-vms:0_monitor_0) on glenrock failed (target: 7 vs. rc: 8): Error
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 3: probe_complete probe_complete-glenrock on glenrock 
(local) - no waiting
Sep 27 15:35:49 glenrock attrd[3364]:   notice: attrd_trigger_update: 
Sending flush op to all hosts for: probe_complete (true)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: run_graph: Transition 51 
(Complete=24, Pending=0, Fired=0, Skipped=9, Incomplete=10, 
Source=/var/lib/pacemaker/pengine/pe-input-534.bz2): Stopped
Sep 27 15:35:49 glenrock attrd[3364]:   notice: attrd_perform_update: 
Sent update 201: probe_complete=true
Sep 27 15:35:49 glenrock pengine[3365]:   notice: unpack_config: On loss 
of CCM Quorum: Ignore
Sep 27 15:35:49 glenrock pengine[3365]:  warning: 
common_apply_stickiness: Forcing iscsi-engine-target away from glenrock 
after 1000000 failures (max=1000000)
Sep 27 15:35:49 glenrock pengine[3365]:   notice: LogActions: Move    
iscsi-engine-target#011(Started glenrock -> granby)
Sep 27 15:35:49 glenrock pengine[3365]:   notice: process_pe_message: 
Calculated Transition 52: /var/lib/pacemaker/pengine/pe-input-535.bz2
Sep 27 15:35:49 glenrock pengine[3365]:   notice: unpack_config: On loss 
of CCM Quorum: Ignore
Sep 27 15:35:49 glenrock pengine[3365]:  warning: 
common_apply_stickiness: Forcing iscsi-engine-target away from glenrock 
after 1000000 failures (max=1000000)
Sep 27 15:35:49 glenrock pengine[3365]:   notice: LogActions: Move    
iscsi-engine-target#011(Started glenrock -> granby)
Sep 27 15:35:49 glenrock pengine[3365]:   notice: process_pe_message: 
Calculated Transition 53: /var/lib/pacemaker/pengine/pe-input-536.bz2
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 7: monitor drbd-vms_monitor_20000 on granby
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 12: monitor drbd-vms_monitor_10000 on glenrock (local)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 39: monitor iscsi-vms-target_monitor_10000 on glenrock 
(local)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 42: monitor iscsi-vms-lun_monitor_10000 on glenrock 
(local)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 45: monitor iscsi-vms-ip_monitor_10000 on glenrock (local)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 50: monitor drbd-engine_monitor_10000 on granby
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 53: monitor drbd-engine_monitor_20000 on glenrock (local)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: te_rsc_command: 
Initiating action 78: stop iscsi-engine-target_stop_0 on glenrock (local)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: process_lrm_event: 
Operation iscsi-vms-target_monitor_10000: ok (node=glenrock, call=332, 
rc=0, cib-update=387, confirmed=false)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: process_lrm_event: 
Operation iscsi-vms-lun_monitor_10000: ok (node=glenrock, call=333, 
rc=0, cib-update=388, confirmed=false)
Sep 27 15:35:49 glenrock iSCSITarget(iscsi-engine-target)[1079]: 
WARNING: Failed to remove target iqn.2015-09.integralife.net, retrying.
Sep 27 15:35:49 glenrock crmd[3366]:   notice: process_lrm_event: 
Operation iscsi-vms-ip_monitor_10000: ok (node=glenrock, call=334, rc=0, 
cib-update=389, confirmed=false)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: process_lrm_event: 
Operation drbd-vms_monitor_10000: master (node=glenrock, call=331, rc=8, 
cib-update=390, confirmed=false)
Sep 27 15:35:49 glenrock crmd[3366]:   notice: process_lrm_event: 
glenrock-drbd-vms_monitor_10000:331 [ \n ]
Sep 27 15:35:49 glenrock crmd[3366]:   notice: process_lrm_event: 
Operation drbd-engine_monitor_20000: ok (node=glenrock, call=335, rc=0, 
cib-update=391, confirmed=false)
Sep 27 15:35:50 glenrock iSCSITarget(iscsi-engine-target)[1079]: 
WARNING: Failed to remove target iqn.2015-09.integralife.net, retrying.
Sep 27 15:35:51 glenrock iSCSITarget(iscsi-engine-target)[1079]: 
WARNING: Failed to remove target iqn.2015-09.integralife.net, retrying.
Sep 27 15:35:52 glenrock iSCSITarget(iscsi-engine-target)[1079]: 
WARNING: Failed to remove target iqn.2015-09.integralife.net, retrying.
Sep 27 15:35:53 glenrock iSCSITarget(iscsi-engine-target)[1079]: 
WARNING: Failed to remove target iqn.2015-09.integralife.net, retrying.
Sep 27 15:35:54 glenrock iSCSITarget(iscsi-engine-target)[1079]: 
WARNING: Failed to remove target iqn.2015-09.integralife.net, retrying.
Sep 27 15:35:55 glenrock iSCSITarget(iscsi-engine-target)[1079]: 
WARNING: Failed to remove target iqn.2015-09.integralife.net, retrying.
Sep 27 15:35:56 glenrock iSCSITarget(iscsi-engine-target)[1079]: 
WARNING: Failed to remove target iqn.2015-09.integralife.net, retrying.
Sep 27 15:35:57 glenrock iSCSITarget(iscsi-engine-target)[1079]: 
WARNING: Failed to remove target iqn.2015-09.integralife.net, retrying.
Sep 27 15:35:58 glenrock iSCSITarget(iscsi-engine-target)[1079]: 
WARNING: Failed to remove target iqn.2015-09.integralife.net, retrying.
Sep 27 15:35:59 glenrock lrmd[3363]:  warning: child_timeout_callback: 
iscsi-engine-target_stop_0 process (PID 1079) timed out
Sep 27 15:35:59 glenrock lrmd[3363]:  warning: operation_finished: 
iscsi-engine-target_stop_0:1079 - timed out after 10000ms
Sep 27 15:35:59 glenrock lrmd[3363]:   notice: operation_finished: 
iscsi-engine-target_stop_0:1079:stderr [ tgtadm: can't find the target ]
Sep 27 15:35:59 glenrock lrmd[3363]:   notice: operation_finished: 
iscsi-engine-target_stop_0:1079:stderr [ tgtadm: can't find the target ]
Sep 27 15:35:59 glenrock lrmd[3363]:   notice: operation_finished: 
iscsi-engine-target_stop_0:1079:stderr [ tgtadm: can't find the target ]
Sep 27 15:35:59 glenrock lrmd[3363]:   notice: operation_finished: 
iscsi-engine-target_stop_0:1079:stderr [ tgtadm: can't find the target ]
Sep 27 15:35:59 glenrock lrmd[3363]:   notice: operation_finished: 
iscsi-engine-target_stop_0:1079:stderr [ tgtadm: can't find the target ]
Sep 27 15:35:59 glenrock lrmd[3363]:   notice: operation_finished: 
iscsi-engine-target_stop_0:1079:stderr [ tgtadm: can't find the target ]
Sep 27 15:35:59 glenrock lrmd[3363]:   notice: operation_finished: 
iscsi-engine-target_stop_0:1079:stderr [ tgtadm: can't find the target ]
Sep 27 15:35:59 glenrock lrmd[3363]:   notice: operation_finished: 
iscsi-engine-target_stop_0:1079:stderr [ tgtadm: can't find the target ]
Sep 27 15:35:59 glenrock lrmd[3363]:   notice: operation_finished: 
iscsi-engine-target_stop_0:1079:stderr [ tgtadm: can't find the target ]
Sep 27 15:35:59 glenrock lrmd[3363]:   notice: operation_finished: 
iscsi-engine-target_stop_0:1079:stderr [ tgtadm: can't find the target ]
Sep 27 15:35:59 glenrock crmd[3366]:    error: process_lrm_event: 
Operation iscsi-engine-target_stop_0: Timed Out (node=glenrock, 
call=336, timeout=10000ms)
Sep 27 15:35:59 glenrock crmd[3366]:   notice: process_lrm_event: 
glenrock-iscsi-engine-target_stop_0:336 [ tgtadm: can't find the 
target\ntgtadm: can't find the target\ntgtadm: can't find the 
target\ntgtadm: can't find the target\ntgtadm: can't find the 
target\ntgtadm: can't find the target\ntgtadm: can't find the 
target\ntgtadm: can't find the target\ntgtadm: can't find the 
target\ntgtadm: can't find the target\n ]
Sep 27 15:35:59 glenrock crmd[3366]:  warning: status_from_rc: Action 78 
(iscsi-engine-target_stop_0) on glenrock failed (target: 0 vs. rc: 1): Error
Sep 27 15:35:59 glenrock crmd[3366]:  warning: update_failcount: 
Updating failcount for iscsi-engine-target on glenrock after failed 
stop: rc=1 (update=INFINITY, time=1443364559)
Sep 27 15:35:59 glenrock crmd[3366]:   notice: abort_transition_graph: 
Transition aborted by iscsi-engine-target_stop_0 'modify' on (null): 
Event failed (magic=2:1;78:53:0:4ac61a75-532d-45c4-b07c-eee753699adc, 
cib=0.374.146, source=match_graph_event:344, 0)
Sep 27 15:35:59 glenrock attrd[3364]:   notice: attrd_trigger_update: 
Sending flush op to all hosts for: last-failure-iscsi-engine-target 
(1443364559)
Sep 27 15:35:59 glenrock crmd[3366]:  warning: update_failcount: 
Updating failcount for iscsi-engine-target on glenrock after failed 
stop: rc=1 (update=INFINITY, time=1443364559)
Sep 27 15:35:59 glenrock crmd[3366]:  warning: status_from_rc: Action 78 
(iscsi-engine-target_stop_0) on glenrock failed (target: 0 vs. rc: 1): Error
Sep 27 15:35:59 glenrock crmd[3366]:  warning: update_failcount: 
Updating failcount for iscsi-engine-target on glenrock after failed 
stop: rc=1 (update=INFINITY, time=1443364559)
Sep 27 15:35:59 glenrock attrd[3364]:   notice: attrd_perform_update: 
Sent update 203: last-failure-iscsi-engine-target=1443364559
Sep 27 15:35:59 glenrock crmd[3366]:  warning: update_failcount: 
Updating failcount for iscsi-engine-target on glenrock after failed 
stop: rc=1 (update=INFINITY, time=1443364559)
Sep 27 15:35:59 glenrock attrd[3364]:   notice: attrd_trigger_update: 
Sending flush op to all hosts for: last-failure-iscsi-engine-target 
(1443364559)
Sep 27 15:35:59 glenrock crmd[3366]:   notice: run_graph: Transition 53 
(Complete=8, Pending=0, Fired=0, Skipped=3, Incomplete=0, 
Source=/var/lib/pacemaker/pengine/pe-input-536.bz2): Stopped
Sep 27 15:35:59 glenrock attrd[3364]:   notice: attrd_perform_update: 
Sent update 205: last-failure-iscsi-engine-target=1443364559
Sep 27 15:35:59 glenrock attrd[3364]:   notice: attrd_trigger_update: 
Sending flush op to all hosts for: last-failure-iscsi-engine-target 
(1443364559)
Sep 27 15:35:59 glenrock attrd[3364]:   notice: attrd_perform_update: 
Sent update 207: last-failure-iscsi-engine-target=1443364559
Sep 27 15:35:59 glenrock pengine[3365]:   notice: unpack_config: On loss 
of CCM Quorum: Ignore
Sep 27 15:35:59 glenrock pengine[3365]:  warning: unpack_rsc_op_failure: 
Processing failed op stop for iscsi-engine-target on glenrock: unknown 
error (1)
Sep 27 15:35:59 glenrock pengine[3365]:  warning: unpack_rsc_op_failure: 
Processing failed op stop for iscsi-engine-target on glenrock: unknown 
error (1)
Sep 27 15:35:59 glenrock pengine[3365]:  warning: 
common_apply_stickiness: Forcing iscsi-engine-target away from glenrock 
after 1000000 failures (max=1000000)
Sep 27 15:35:59 glenrock pengine[3365]:   notice: process_pe_message: 
Calculated Transition 54: /var/lib/pacemaker/pengine/pe-input-537.bz2
Sep 27 15:35:59 glenrock crmd[3366]:   notice: run_graph: Transition 54 
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, 
Source=/var/lib/pacemaker/pengine/pe-input-537.bz2): Complete
Sep 27 15:35:59 glenrock crmd[3366]:   notice: do_state_transition: 
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS 
cause=C_FSA_INTERNAL origin=notify_crmd ]

I've even deleted all resources for the engine stuff and started again, 
but the same still happens! My oVirt engine is down all this time even 
though the big "vms" lun has been absolutely fine.

Also it's really odd that the iSCSITarget resource is not available is 
Centos 6.x, I had to copy the resource script from an FTP site, ugh!

Any help would be most gratefully received.

Best wishes

Alex

More information about the Users mailing list