===================================================== Corosync started ----------------------------------------------------- ============ Last updated: Thu Mar 3 11:41:06 2011 Stack: openais Current DC: armani - partition with quorum Version: 1.0.9-da7075976b5ff0bee71074385f8fd02f296ec8a3 2 Nodes configured, 2 expected votes 2 Resources configured. ============ Online: [ bulgari armani ] failover-ip (ocf::heartbeat:IPaddr): Started bulgari Clone Set: cloneKamailio Started: [ bulgari armani ] Operations: * Node armani: kamailio:1: migration-threshold=2 + (4) start: rc=0 (ok) + (5) monitor: interval=10000ms rc=0 (ok) * Node bulgari: kamailio:0: migration-threshold=2 + (5) start: rc=0 (ok) + (7) monitor: interval=10000ms rc=0 (ok) failover-ip: migration-threshold=1000000 + (4) start: rc=0 (ok) + (6) monitor: interval=3000ms rc=0 (ok) ===================================================== armani: /etc/init.d/kamailio stop Kamailio will be restarted immediately, failure-count increased, IP resides on bulgari ----------------------------------------------------- Online: [ bulgari armani ] failover-ip (ocf::heartbeat:IPaddr): Started bulgari Clone Set: cloneKamailio Started: [ bulgari armani ] Operations: * Node armani: kamailio:1: migration-threshold=2 fail-count=1 last-failure='Thu Mar 3 11:47:36 2011' + (6) stop: rc=0 (ok) + (7) start: rc=0 (ok) + (8) monitor: interval=10000ms rc=0 (ok) * Node bulgari: kamailio:0: migration-threshold=2 + (5) start: rc=0 (ok) + (7) monitor: interval=10000ms rc=0 (ok) failover-ip: migration-threshold=1000000 + (4) start: rc=0 (ok) + (6) monitor: interval=3000ms rc=0 (ok) ===================================================== bulgari: /etc/init.d/kamailio stop Kamailio will be restarted immediately, failure-count increased, IP resides on bulgari ----------------------------------------------------- Online: [ bulgari armani ] failover-ip (ocf::heartbeat:IPaddr): Started bulgari Clone Set: cloneKamailio Started: [ bulgari armani ] Operations: * Node armani: kamailio:1: migration-threshold=2 fail-count=1 last-failure='Thu Mar 3 11:47:36 2011' + (6) stop: rc=0 (ok) + (7) start: rc=0 (ok) + (8) monitor: interval=10000ms rc=0 (ok) * Node bulgari: kamailio:0: migration-threshold=2 fail-count=1 last-failure='Thu Mar 3 11:49:47 2011' + (8) stop: rc=0 (ok) + (9) start: rc=0 (ok) + (10) monitor: interval=10000ms rc=0 (ok) failover-ip: migration-threshold=1000000 + (4) start: rc=0 (ok) + (6) monitor: interval=3000ms rc=0 (ok) ===================================================== armani: /etc/init.d/kamailio stop Kamailio failed second time, failure-count increased, will stay stopped for 30 seconds, IP resides on bulgari ----------------------------------------------------- Online: [ bulgari armani ] failover-ip (ocf::heartbeat:IPaddr): Started bulgari Clone Set: cloneKamailio Started: [ bulgari ] Stopped: [ kamailio:1 ] Operations: * Node armani: kamailio:1: migration-threshold=2 fail-count=2 last-failure='Thu Mar 3 11:50:27 2011' + (7) start: rc=0 (ok) + (8) monitor: interval=10000ms rc=7 (not running) + (9) stop: rc=0 (ok) * Node bulgari: kamailio:0: migration-threshold=2 fail-count=1 last-failure='Thu Mar 3 11:49:47 2011' + (8) stop: rc=0 (ok) + (9) start: rc=0 (ok) + (10) monitor: interval=10000ms rc=0 (ok) failover-ip: migration-threshold=1000000 + (4) start: rc=0 (ok) + (6) monitor: interval=3000ms rc=0 (ok) Failed actions: kamailio:1_monitor_10000 (node=armani, call=8, rc=7, status=complete): not running ===================================================== while Kamailio is still stopped on armani: bulgari: /etc/init.d/kamailio stop Kamailio failed second time on bulgari, failure-count increased, will stay stopped for 30 seconds, IP is stoppped ----------------------------------------------------- Online: [ bulgari armani ] Operations: * Node armani: kamailio:1: migration-threshold=2 fail-count=2 last-failure='Thu Mar 3 11:50:27 2011' + (7) start: rc=0 (ok) + (8) monitor: interval=10000ms rc=7 (not running) + (9) stop: rc=0 (ok) * Node bulgari: kamailio:0: migration-threshold=2 fail-count=2 last-failure='Thu Mar 3 11:50:37 2011' + (9) start: rc=0 (ok) + (10) monitor: interval=10000ms rc=7 (not running) + (11) stop: rc=0 (ok) failover-ip: migration-threshold=1000000 + (4) start: rc=0 (ok) + (6) monitor: interval=3000ms rc=0 (ok) + (12) stop: rc=0 (ok) Failed actions: kamailio:1_monitor_10000 (node=armani, call=8, rc=7, status=complete): not running kamailio:0_monitor_10000 (node=bulgari, call=10, rc=7, status=complete): not running ===================================================== failure-timeout triggers and starts Kamailio on armani, the cloned resource is "swapped" between the two nodes IP is started on armani ----------------------------------------------------- Online: [ bulgari armani ] failover-ip (ocf::heartbeat:IPaddr): Started armani Clone Set: cloneKamailio Started: [ armani ] Stopped: [ kamailio:1 ] Operations: * Node armani: kamailio:1: migration-threshold=2 fail-count=2 last-failure='Thu Mar 3 11:50:27 2011' + (7) start: rc=0 (ok) + (8) monitor: interval=10000ms rc=7 (not running) + (9) stop: rc=0 (ok) failover-ip: migration-threshold=1000000 + (10) start: rc=0 (ok) + (12) monitor: interval=3000ms rc=0 (ok) kamailio:0: migration-threshold=2 last-failure='Thu Mar 3 11:50:27 2011' + (11) start: rc=0 (ok) + (13) monitor: interval=10000ms rc=0 (ok) * Node bulgari: kamailio:0: migration-threshold=2 fail-count=2 last-failure='Thu Mar 3 11:50:37 2011' + (9) start: rc=0 (ok) + (10) monitor: interval=10000ms rc=7 (not running) + (11) stop: rc=0 (ok) failover-ip: migration-threshold=1000000 + (4) start: rc=0 (ok) + (6) monitor: interval=3000ms rc=0 (ok) + (12) stop: rc=0 (ok) Failed actions: kamailio:0_monitor_10000 (node=bulgari, call=10, rc=7, status=complete): not running ===================================================== failure-timeout triggers and starts Kamailio on bulgari, IP resides on armani ----------------------------------------------------- Online: [ bulgari armani ] failover-ip (ocf::heartbeat:IPaddr): Started armani Clone Set: cloneKamailio Started: [ armani bulgari ] Operations: * Node armani: kamailio:1: migration-threshold=2 fail-count=2 last-failure='Thu Mar 3 11:50:27 2011' + (7) start: rc=0 (ok) + (8) monitor: interval=10000ms rc=7 (not running) + (9) stop: rc=0 (ok) failover-ip: migration-threshold=1000000 + (10) start: rc=0 (ok) + (12) monitor: interval=3000ms rc=0 (ok) kamailio:0: migration-threshold=2 last-failure='Thu Mar 3 11:50:27 2011' + (11) start: rc=0 (ok) + (13) monitor: interval=10000ms rc=0 (ok) * Node bulgari: kamailio:0: migration-threshold=2 fail-count=2 last-failure='Thu Mar 3 11:50:37 2011' + (9) start: rc=0 (ok) + (10) monitor: interval=10000ms rc=7 (not running) + (11) stop: rc=0 (ok) failover-ip: migration-threshold=1000000 + (4) start: rc=0 (ok) + (6) monitor: interval=3000ms rc=0 (ok) + (12) stop: rc=0 (ok) kamailio:1: migration-threshold=2 last-failure='Thu Mar 3 11:50:37 2011' + (13) start: rc=0 (ok) + (14) monitor: interval=10000ms rc=0 (ok) ===================================================== now, regardless how often I stop Kamailio on armani, armani: /etc/init.d/kamailio stop failure-count for kamailio:0 is increased but Kamailio always will be re-started immediately (not waiting failure-timeout) and IP will never migrate to bulgari ----------------------------------------------------- Online: [ bulgari armani ] failover-ip (ocf::heartbeat:IPaddr): Started armani Clone Set: cloneKamailio Started: [ armani bulgari ] Operations: * Node armani: kamailio:1: migration-threshold=2 fail-count=2 last-failure='Thu Mar 3 11:50:27 2011' + (7) start: rc=0 (ok) + (8) monitor: interval=10000ms rc=7 (not running) + (9) stop: rc=0 (ok) failover-ip: migration-threshold=1000000 + (10) start: rc=0 (ok) + (12) monitor: interval=3000ms rc=0 (ok) kamailio:0: migration-threshold=2 fail-count=1 last-failure='Thu Mar 3 11:50:27 2011' + (14) stop: rc=0 (ok) + (15) start: rc=0 (ok) + (16) monitor: interval=10000ms rc=0 (ok) * Node bulgari: kamailio:0: migration-threshold=2 fail-count=2 last-failure='Thu Mar 3 11:50:37 2011' + (9) start: rc=0 (ok) + (10) monitor: interval=10000ms rc=7 (not running) + (11) stop: rc=0 (ok) failover-ip: migration-threshold=1000000 + (4) start: rc=0 (ok) + (6) monitor: interval=3000ms rc=0 (ok) + (12) stop: rc=0 (ok) kamailio:1: migration-threshold=2 last-failure='Thu Mar 3 11:50:37 2011' + (13) start: rc=0 (ok) + (14) monitor: interval=10000ms rc=0 (ok) ===================================================== armani: /etc/init.d/kamailio stop kamailio:0: fail-count=2, no migration... ----------------------------------------------------- Online: [ bulgari armani ] failover-ip (ocf::heartbeat:IPaddr): Started armani Clone Set: cloneKamailio Started: [ armani bulgari ] Operations: * Node armani: kamailio:1: migration-threshold=2 fail-count=2 last-failure='Thu Mar 3 11:50:27 2011' + (7) start: rc=0 (ok) + (8) monitor: interval=10000ms rc=7 (not running) + (9) stop: rc=0 (ok) failover-ip: migration-threshold=1000000 + (10) start: rc=0 (ok) + (12) monitor: interval=3000ms rc=0 (ok) kamailio:0: migration-threshold=2 fail-count=2 last-failure='Thu Mar 3 11:50:27 2011' + (17) stop: rc=0 (ok) + (18) start: rc=0 (ok) + (19) monitor: interval=10000ms rc=0 (ok) * Node bulgari: kamailio:0: migration-threshold=2 fail-count=2 last-failure='Thu Mar 3 11:50:37 2011' + (9) start: rc=0 (ok) + (10) monitor: interval=10000ms rc=7 (not running) + (11) stop: rc=0 (ok) failover-ip: migration-threshold=1000000 + (4) start: rc=0 (ok) + (6) monitor: interval=3000ms rc=0 (ok) + (12) stop: rc=0 (ok) kamailio:1: migration-threshold=2 last-failure='Thu Mar 3 11:50:37 2011' + (13) start: rc=0 (ok) + (14) monitor: interval=10000ms rc=0 (ok) ===================================================== armani: /etc/init.d/kamailio stop kamailio:0: fail-count=3, no migration... ----------------------------------------------------- Online: [ bulgari armani ] failover-ip (ocf::heartbeat:IPaddr): Started armani Clone Set: cloneKamailio Started: [ armani bulgari ] Operations: * Node armani: kamailio:1: migration-threshold=2 fail-count=2 last-failure='Thu Mar 3 11:50:27 2011' + (7) start: rc=0 (ok) + (8) monitor: interval=10000ms rc=7 (not running) + (9) stop: rc=0 (ok) failover-ip: migration-threshold=1000000 + (10) start: rc=0 (ok) + (12) monitor: interval=3000ms rc=0 (ok) kamailio:0: migration-threshold=2 fail-count=3 last-failure='Thu Mar 3 11:50:27 2011' + (20) stop: rc=0 (ok) + (21) start: rc=0 (ok) + (22) monitor: interval=10000ms rc=0 (ok) * Node bulgari: kamailio:0: migration-threshold=2 fail-count=2 last-failure='Thu Mar 3 11:50:37 2011' + (9) start: rc=0 (ok) + (10) monitor: interval=10000ms rc=7 (not running) + (11) stop: rc=0 (ok) failover-ip: migration-threshold=1000000 + (4) start: rc=0 (ok) + (6) monitor: interval=3000ms rc=0 (ok) + (12) stop: rc=0 (ok) kamailio:1: migration-threshold=2 last-failure='Thu Mar 3 11:50:37 2011' + (13) start: rc=0 (ok) + (14) monitor: interval=10000ms rc=0 (ok) ===================================================== armani: /etc/init.d/kamailio stop kamailio:0: fail-count=4, no migration... ----------------------------------------------------- Online: [ bulgari armani ] failover-ip (ocf::heartbeat:IPaddr): Started armani Clone Set: cloneKamailio Started: [ armani bulgari ] Operations: * Node armani: kamailio:1: migration-threshold=2 fail-count=2 last-failure='Thu Mar 3 11:50:27 2011' + (7) start: rc=0 (ok) + (8) monitor: interval=10000ms rc=7 (not running) + (9) stop: rc=0 (ok) failover-ip: migration-threshold=1000000 + (10) start: rc=0 (ok) + (12) monitor: interval=3000ms rc=0 (ok) kamailio:0: migration-threshold=2 fail-count=4 last-failure='Thu Mar 3 11:50:27 2011' + (23) stop: rc=0 (ok) + (24) start: rc=0 (ok) + (25) monitor: interval=10000ms rc=0 (ok) * Node bulgari: kamailio:0: migration-threshold=2 fail-count=2 last-failure='Thu Mar 3 11:50:37 2011' + (9) start: rc=0 (ok) + (10) monitor: interval=10000ms rc=7 (not running) + (11) stop: rc=0 (ok) failover-ip: migration-threshold=1000000 + (4) start: rc=0 (ok) + (6) monitor: interval=3000ms rc=0 (ok) + (12) stop: rc=0 (ok) kamailio:1: migration-threshold=2 last-failure='Thu Mar 3 11:50:37 2011' + (13) start: rc=0 (ok) + (14) monitor: interval=10000ms rc=0 (ok)