<div dir="ltr"><div>Hi Everyone, <br></div><div>I have experience a weird issue last night where our cluster try to failover due to an "Unkown interface"</div><div><br></div><div>Look like when the IPaddr2 monitor try to perform a status on eth0, it didn't find the device. Both node are VM. I haven't found any reason as why eth0 would have "disapear"</div><div><br></div><div><div><LOG NODE1></div><div>Sep 29 21:25:04 node-01 IPaddr2(vip_v207_174)[4082]: ERROR: Unknown interface [eth0] No such device.</div><div>Sep 29 21:25:04 node-01 IPaddr2(vip_v207_174)[4082]: ERROR: [findif] failed</div><div>Sep 29 21:25:05 node-01 crmd[3369]: notice: process_lrm_event: Operation vip_v207_174_monitor_10000: not configured (node=node-01, call=91, rc=6, cib-update=73, confirmed=false)</div><div>Sep 29 21:25:06 node-01 attrd[3367]: notice: attrd_cs_dispatch: Update relayed from node-02</div><div>Sep 29 21:25:06 node-01 attrd[3367]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-vip_v207_174 (2)</div><div>Sep 29 21:25:06 node-01 attrd[3367]: notice: attrd_perform_update: Sent update 41: fail-count-vip_v207_174=2</div><div>Sep 29 21:25:06 node-01 attrd[3367]: notice: attrd_cs_dispatch: Update relayed from node-02</div><div>Sep 29 21:25:06 node-01 attrd[3367]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-vip_v207_174 (1443576306)</div><div>Sep 29 21:25:06 node-01 attrd[3367]: notice: attrd_perform_update: Sent update 43: last-failure-vip_v207_174=1443576306</div><div>Sep 29 21:25:07 node-01 crmd[3369]: notice: process_lrm_event: Operation fwcorp-mailto-sysadmin_stop_0: ok (node=node-01, call=110, rc=0, cib-update=74, confirmed=true)</div><div>Sep 29 21:25:07 node-01 crmd[3369]: notice: process_lrm_event: Operation change-default-fw_stop_0: ok (node=node-01, call=112, rc=0, cib-update=75, confirmed=true)</div><div>Sep 29 21:25:07 node-01 IPaddr2(vip_v254_230)[4259]: INFO: IP status = ok, IP_CIP=</div><div>Sep 29 21:25:07 node-01 crmd[3369]: notice: process_lrm_event: Operation vip_v254_230_stop_0: ok (node=node-01, call=114, rc=0, cib-update=76, confirmed=true)</div><div>Sep 29 21:25:07 node-01 IPaddr2(vip_v27_1)[4313]: INFO: IP status = ok, IP_CIP=</div><div>Sep 29 21:25:07 node-01 crmd[3369]: notice: process_lrm_event: Operation vip_v27_1_stop_0: ok (node=node-01, call=116, rc=0, cib-update=77, confirmed=true)</div><div>Sep 29 21:25:07 node-01 IPaddr2(vip_v26_1)[4366]: INFO: IP status = ok, IP_CIP=</div><div>Sep 29 21:25:07 node-01 crmd[3369]: notice: process_lrm_event: Operation vip_v26_1_stop_0: ok (node=node-01, call=118, rc=0, cib-update=78, confirmed=true)</div><div>Sep 29 21:25:07 node-01 IPaddr2(vip_v207_174)[4419]: INFO: IP status = ok, IP_CIP=</div><div>Sep 29 21:25:07 node-01 crmd[3369]: notice: process_lrm_event: Operation vip_v207_174_stop_0: ok (node=node-01, call=120, rc=0, cib-update=79, confirmed=true)</div><div></LOG NODE1><br></div><div><br></div><div><LOG NODE2><br></div><div>Sep 29 21:22:48 node-02 crmd[3241]: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]</div><div>Sep 29 21:22:48 node-02 pengine[3240]: notice: update_validation: pacemaker-1.2-style configuration is also valid for pacemaker-1.3</div><div>Sep 29 21:22:48 node-02 pengine[3240]: notice: update_validation: Upgrading pacemaker-1.3-style configuration to pacemaker-2.0 with upgrade-1.3.xsl</div><div>Sep 29 21:22:48 node-02 pengine[3240]: notice: update_validation: Transformed the configuration from pacemaker-1.2 to pacemaker-2.0</div><div>Sep 29 21:22:48 node-02 pengine[3240]: notice: unpack_config: On loss of CCM Quorum: Ignore</div><div>Sep 29 21:22:48 node-02 crmd[3241]: notice: run_graph: Transition 14769 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-786.bz2): Complete</div><div>Sep 29 21:22:48 node-02 crmd[3241]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]</div><div>Sep 29 21:22:48 node-02 pengine[3240]: notice: process_pe_message: Calculated Transition 14769: /var/lib/pacemaker/pengine/pe-input-786.bz2</div><div>Sep 29 21:25:06 node-02 crmd[3241]: warning: update_failcount: Updating failcount for vip_v207_174 on node-01 after failed monitor: rc=6 (update=value++, time=1443576306)</div><div>Sep 29 21:25:06 node-02 crmd[3241]: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]</div><div>Sep 29 21:25:06 node-02 pengine[3240]: notice: update_validation: pacemaker-1.2-style configuration is also valid for pacemaker-1.3</div><div>Sep 29 21:25:06 node-02 pengine[3240]: notice: update_validation: Upgrading pacemaker-1.3-style configuration to pacemaker-2.0 with upgrade-1.3.xsl</div><div>Sep 29 21:25:06 node-02 pengine[3240]: notice: update_validation: Transformed the configuration from pacemaker-1.2 to pacemaker-2.0</div><div>Sep 29 21:25:06 node-02 pengine[3240]: notice: unpack_config: On loss of CCM Quorum: Ignore</div><div>Sep 29 21:25:06 node-02 pengine[3240]: warning: unpack_rsc_op_failure: Processing failed op monitor for vip_v207_174 on node-01: not configured (6)</div><div>Sep 29 21:25:06 node-02 pengine[3240]: error: unpack_rsc_op: Preventing vip_v207_174 from re-starting anywhere: operation monitor failed 'not configured' (6)</div><div>Sep 29 21:25:06 node-02 pengine[3240]: notice: LogActions: Stop vip_v207_174#011(node-01)</div><div>Sep 29 21:25:06 node-02 pengine[3240]: notice: LogActions: Stop vip_v26_1#011(node-01)</div><div>Sep 29 21:25:06 node-02 pengine[3240]: notice: LogActions: Stop vip_v27_1#011(node-01)</div><div>Sep 29 21:25:06 node-02 pengine[3240]: notice: LogActions: Stop vip_v254_230#011(node-01)</div><div>Sep 29 21:25:06 node-02 pengine[3240]: notice: LogActions: Stop change-default-fw#011(node-01)</div><div>Sep 29 21:25:06 node-02 pengine[3240]: notice: LogActions: Stop fwcorp-mailto-sysadmin#011(node-01)</div><div>Sep 29 21:25:06 node-02 pengine[3240]: notice: process_pe_message: Calculated Transition 14770: /var/lib/pacemaker/pengine/pe-input-787.bz2</div><div>Sep 29 21:25:06 node-02 crmd[3241]: notice: te_rsc_command: Initiating action 16: stop fwcorp-mailto-sysadmin_stop_0 on node-01</div><div>Sep 29 21:25:06 node-02 crmd[3241]: notice: abort_transition_graph: Transition aborted by status-node-01-fail-count-vip_v207_174, fail-count-vip_v207_174=2: Transient attribute change (modify cib=0.94.107, source=te_update_diff:391, path=/cib/status/node_state[@id='node-01']/transient_attributes[@id='node-01']/instance_attributes[@id='status-node-01']/nvpair[@id='status-node-01-fail-count-vip_v207_174'], 0)</div><div>Sep 29 21:25:07 node-02 crmd[3241]: notice: run_graph: Transition 14770 (Complete=2, Pending=0, Fired=0, Skipped=7, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-787.bz2): Stopped</div><div>Sep 29 21:25:07 node-02 pengine[3240]: notice: update_validation: pacemaker-1.2-style configuration is also valid for pacemaker-1.3</div><div>Sep 29 21:25:07 node-02 pengine[3240]: notice: update_validation: Upgrading pacemaker-1.3-style configuration to pacemaker-2.0 with upgrade-1.3.xsl</div><div>Sep 29 21:25:07 node-02 pengine[3240]: notice: update_validation: Transformed the configuration from pacemaker-1.2 to pacemaker-2.0</div><div>Sep 29 21:25:07 node-02 pengine[3240]: notice: unpack_config: On loss of CCM Quorum: Ignore</div><div>Sep 29 21:25:07 node-02 pengine[3240]: warning: unpack_rsc_op_failure: Processing failed op monitor for vip_v207_174 on node-01: not configured (6)</div><div>Sep 29 21:25:07 node-02 pengine[3240]: error: unpack_rsc_op: Preventing vip_v207_174 from re-starting anywhere: operation monitor failed 'not configured' (6)</div><div>Sep 29 21:25:07 node-02 pengine[3240]: notice: LogActions: Stop vip_v207_174#011(node-01)</div><div>Sep 29 21:25:07 node-02 pengine[3240]: notice: LogActions: Stop vip_v26_1#011(node-01)</div><div>Sep 29 21:25:07 node-02 pengine[3240]: notice: LogActions: Stop vip_v27_1#011(node-01)</div><div>Sep 29 21:25:07 node-02 pengine[3240]: notice: LogActions: Stop vip_v254_230#011(node-01)</div><div>Sep 29 21:25:07 node-02 pengine[3240]: notice: LogActions: Stop change-default-fw#011(node-01)</div><div>Sep 29 21:25:07 node-02 crmd[3241]: notice: te_rsc_command: Initiating action 14: stop change-default-fw_stop_0 on node-01</div><div>Sep 29 21:25:07 node-02 pengine[3240]: notice: process_pe_message: Calculated Transition 14771: /var/lib/pacemaker/pengine/pe-input-788.bz2</div><div>Sep 29 21:25:07 node-02 crmd[3241]: notice: te_rsc_command: Initiating action 13: stop vip_v254_230_stop_0 on node-01</div><div>Sep 29 21:25:07 node-02 crmd[3241]: notice: te_rsc_command: Initiating action 12: stop vip_v27_1_stop_0 on node-01</div><div>Sep 29 21:25:07 node-02 crmd[3241]: notice: te_rsc_command: Initiating action 11: stop vip_v26_1_stop_0 on node-01</div><div>Sep 29 21:25:07 node-02 crmd[3241]: notice: te_rsc_command: Initiating action 3: stop vip_v207_174_stop_0 on node-01</div><div>Sep 29 21:25:07 node-02 crmd[3241]: notice: run_graph: Transition 14771 (Complete=8, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-788.bz2): Complete</div><div>Sep 29 21:25:07 node-02 crmd[3241]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]</div></div><div></LOG NODE2><br></div><div><br></div><div>I know that I found some post that say to run sysctl -w net.ipv4.conf.all.promote_secondaries=1 to avoid secondary nic to be remove when primary is gone, but in this case the eth0 has a single nic that is manage through IPaddr2 within crm configuration</div><div><br></div><div>Here's the configuration or node: </div><div><div><br></div><div><CONFIGURATION><br></div><div>Cluster Name: nodecluster1</div><div>Corosync Nodes:</div><div> node-01 node-02</div><div>Pacemaker Nodes:</div><div> node-01 node-02</div><div><br></div><div>Resources:</div><div> Group: lbpcivip</div><div> Resource: vip_v207_174 (class=ocf provider=heartbeat type=IPaddr2)</div><div> Attributes: ip=x.x.x.174 cidr_netmask=27 broadcast=x.x.x.191 nic=eth0</div><div> Operations: monitor interval=10s (vip_v207_174-monitor-interval-10s)</div><div> Resource: vip_v26_1 (class=ocf provider=heartbeat type=IPaddr2)</div><div> Attributes: ip=x.x.26.1</div><div> Operations: monitor interval=10s (vip_v26_1-monitor-interval-10s)</div><div> Resource: vip_v27_1 (class=ocf provider=heartbeat type=IPaddr2)</div><div> Attributes: ip=x.x.27.1</div><div> Operations: monitor interval=10s (vip_v27_1-monitor-interval-10s)</div><div> Resource: vip_v254_230 (class=ocf provider=heartbeat type=IPaddr2)</div><div> Attributes: ip=x.x.254.230</div><div> Operations: monitor interval=10s (vip_v254_230-monitor-interval-10s)</div><div> Resource: change-default-fw (class=lsb type=fwdefaultgw)</div><div> Operations: monitor interval=60s (change-default-fw-monitor-interval-60s)</div><div> Resource: fwcorp-mailto-sysadmin (class=ocf provider=heartbeat type=MailTo)</div><div> Attributes: email=<a href="mailto:its@touchtunes.com">its@touchtunes.com</a> subject="[node - Clustered services]"</div><div> Operations: monitor interval=60s (fwcorp-mailto-sysadmin-monitor-interval-60s)</div><div><br></div><div>Stonith Devices:</div><div>Fencing Levels:</div><div><br></div><div>Location Constraints:</div><div>Ordering Constraints:</div><div>Colocation Constraints:</div><div><br></div><div>Cluster Properties:</div><div> cluster-infrastructure: cman</div><div> dc-version: 1.1.11-97629de</div><div> last-lrm-refresh: 1412269491</div><div> no-quorum-policy: ignore</div><div> stonith-enabled: false</div></div><div></CONFIGURATION><br></div><div><br></div><div>Has anyone have suggestion on how I can solve this issue? Why did the failover from node1 to node2 didn't work ? </div><div><br></div><div>If more information is require let me know, any suggestion would be appreciated!</div><div><br></div><div>Thanx! </div><div><br></div><div><br></div><div><br></div><div><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div><div>--</div><div> !!!!!</div><div> ( o o )</div><div> --------------oOO----(_)----OOo--------------</div><div> Luc Paulin</div><div> email: paulinster(at)<a href="http://gmail.com" target="_blank">gmail.com</a></div><div> Skype: paulinster</div></div><div><br></div></div></div></div>
</div></div>