Dec 24 11:25:33 corosync [MAIN  ] Corosync Cluster Engine ('1.1.1'): started and ready to provide service.
Dec 24 11:25:33 corosync [MAIN  ] Corosync built-in features:
Dec 24 11:25:33 corosync [MAIN  ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
Dec 24 11:25:33 corosync [TOTEM ] Initializing transport (UDP/IP).
Dec 24 11:25:33 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Dec 24 11:25:33 corosync [TOTEM ] Initializing transport (UDP/IP).
Dec 24 11:25:33 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Dec 24 11:25:33 corosync [MAIN  ] Compatibility mode set to whitetank.  Using V1 and V2 of the synchronization engine.
Dec 24 11:25:33 corosync [TOTEM ] The network interface [192.168.2.2] is now up.
Dec 24 11:25:33 corosync [pcmk  ] info: process_ais_conf: Reading configure
Dec 24 11:25:33 corosync [pcmk  ] info: config_find_init: Local handle: 5650605097994944513 for logging
Dec 24 11:25:33 corosync [pcmk  ] info: config_find_next: Processing additional logging options...
Dec 24 11:25:33 corosync [pcmk  ] info: get_config_opt: Found 'off' for option: debug
Dec 24 11:25:33 corosync [pcmk  ] info: get_config_opt: Defaulting to 'off' for option: to_file
Dec 24 11:25:33 corosync [pcmk  ] info: get_config_opt: Defaulting to 'daemon' for option: syslog_facility
Dec 24 11:25:33 corosync [pcmk  ] info: config_find_init: Local handle: 2730409743423111170 for service
Dec 24 11:25:33 corosync [pcmk  ] info: config_find_next: Processing additional service options...
Dec 24 11:25:33 corosync [pcmk  ] info: get_config_opt: Defaulting to 'no' for option: use_logd
Dec 24 11:25:33 corosync [pcmk  ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd
Dec 24 11:25:33 corosync [pcmk  ] info: pcmk_startup: CRM: Initialized
Dec 24 11:25:33 corosync [pcmk  ] Logging: Initialized pcmk_startup
Dec 24 11:25:33 corosync [pcmk  ] info: pcmk_startup: Maximum core file size is: 18446744073709551615
Dec 24 11:25:33 corosync [pcmk  ] info: pcmk_startup: Service: 9
Dec 24 11:25:33 corosync [pcmk  ] info: pcmk_startup: Local hostname: node2.itactics.com
Dec 24 11:25:33 corosync [pcmk  ] info: pcmk_update_nodeid: Local node id: 33728704
Dec 24 11:25:33 corosync [pcmk  ] info: update_member: Creating entry for node 33728704 born on 0
Dec 24 11:25:33 corosync [pcmk  ] info: update_member: 0x54d830 Node 33728704 now known as node2.itactics.com (was: (null))
Dec 24 11:25:33 corosync [pcmk  ] info: update_member: Node node2.itactics.com now has 1 quorum votes (was 0)
Dec 24 11:25:33 corosync [pcmk  ] info: update_member: Node 33728704/node2.itactics.com is now: member
Dec 24 11:25:33 corosync [pcmk  ] info: spawn_child: Forked child 7707 for process stonithd
Dec 24 11:25:33 corosync [pcmk  ] info: spawn_child: Forked child 7708 for process cib
Dec 24 11:25:33 corosync [pcmk  ] info: spawn_child: Forked child 7709 for process lrmd
Dec 24 11:25:33 corosync [pcmk  ] info: spawn_child: Forked child 7710 for process attrd
Dec 24 11:25:33 corosync [pcmk  ] info: spawn_child: Forked child 7711 for process pengine
Dec 24 11:25:33 corosync [pcmk  ] info: spawn_child: Forked child 7712 for process crmd
Dec 24 11:25:33 corosync [SERV  ] Service engine loaded: Pacemaker Cluster Manager 1.0.5
Dec 24 11:25:33 corosync [SERV  ] Service engine loaded: corosync extended virtual synchrony service
Dec 24 11:25:33 corosync [SERV  ] Service engine loaded: corosync configuration service
Dec 24 11:25:33 corosync [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01
Dec 24 11:25:33 corosync [SERV  ] Service engine loaded: corosync cluster config database access v1.01
Dec 24 11:25:33 corosync [SERV  ] Service engine loaded: corosync profile loading service
Dec 24 11:25:33 corosync [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Dec 24 11:25:33 corosync [TOTEM ] The network interface [192.168.2.2] is now up.
Dec 24 11:25:33 corosync [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 4: memb=0, new=0, lost=0
Dec 24 11:25:33 corosync [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 4: memb=1, new=1, lost=0
Dec 24 11:25:33 corosync [pcmk  ] info: pcmk_peer_update: NEW:  node2.itactics.com 33728704
Dec 24 11:25:33 corosync [pcmk  ] info: pcmk_peer_update: MEMB: node2.itactics.com 33728704
Dec 24 11:25:33 corosync [pcmk  ] info: update_member: Node node2.itactics.com now has process list: 00000000000000000000000000013312 (78610)
Dec 24 11:25:33 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Dec 24 11:25:33 corosync [TOTEM ] Marking seqid 10 ringid 0 interface 192.168.2.2 FAULTY - adminisrtative intervention required.
Dec 24 11:25:33 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Dec 24 11:25:33 corosync [pcmk  ] info: pcmk_ipc: Recorded connection 0x551ac0 for attrd/7710
Dec 24 11:26:34 corosync [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 8: memb=1, new=0, lost=0
Dec 24 11:26:34 corosync [pcmk  ] info: pcmk_peer_update: memb: node2.itactics.com 33728704
Dec 24 11:26:34 corosync [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 8: memb=2, new=1, lost=0
Dec 24 11:26:34 corosync [pcmk  ] info: update_member: Creating entry for node 16951488 born on 8
Dec 24 11:26:34 corosync [pcmk  ] info: update_member: Node 16951488/unknown is now: member
Dec 24 11:26:34 corosync [pcmk  ] info: pcmk_peer_update: NEW:  .pending. 16951488
Dec 24 11:26:34 corosync [pcmk  ] info: pcmk_peer_update: MEMB: .pending. 16951488
Dec 24 11:26:34 corosync [pcmk  ] info: pcmk_peer_update: MEMB: node2.itactics.com 33728704
Dec 24 11:26:34 corosync [pcmk  ] info: send_member_notification: Sending membership update 8 to 0 children
Dec 24 11:26:34 corosync [pcmk  ] info: update_member: 0x54d830 Node 33728704 ((null)) born on: 8
Dec 24 11:26:34 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Dec 24 11:26:34 corosync [pcmk  ] info: update_member: 0x550f30 Node 16951488 (node1.itactics.com) born on: 8
Dec 24 11:26:34 corosync [pcmk  ] info: update_member: 0x550f30 Node 16951488 now known as node1.itactics.com (was: (null))
Dec 24 11:26:34 corosync [pcmk  ] info: update_member: Node node1.itactics.com now has process list: 00000000000000000000000000013312 (78610)
Dec 24 11:26:34 corosync [pcmk  ] info: update_member: Node node1.itactics.com now has 1 quorum votes (was 0)
Dec 24 11:26:34 corosync [pcmk  ] info: send_member_notification: Sending membership update 8 to 0 children
Dec 24 11:26:34 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Dec 24 11:26:35 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.crmd failed: unknown (rc=-2)
Dec 24 11:26:36 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.crmd failed: unknown (rc=-2)
Dec 24 11:27:18 corosync [pcmk  ] info: pcmk_ipc_exit: Client attrd (conn=0x551ac0, async-conn=0x551ac0) left
Dec 24 11:27:18 corosync [pcmk  ] ERROR: pcmk_wait_dispatch: Child process attrd exited (pid=7710, rc=100)
Dec 24 11:27:18 corosync [pcmk  ] notice: pcmk_wait_dispatch: Child process attrd no longer wishes to be respawned
Dec 24 11:27:18 corosync [pcmk  ] info: update_member: Node node2.itactics.com now has process list: 00000000000000000000000000012312 (74514)
Dec 24 11:27:18 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.crmd failed: unknown (rc=-2)
Dec 24 11:28:46 corosync [pcmk  ] ERROR: pcmk_wait_dispatch: Child process pengine terminated with signal 9 (pid=7711, core=false)
Dec 24 11:28:46 corosync [pcmk  ] notice: pcmk_wait_dispatch: Respawning failed child process: pengine
Dec 24 11:28:46 corosync [pcmk  ] info: spawn_child: Forked child 7924 for process pengine
Dec 24 11:33:47 corosync [MAIN  ] Corosync Cluster Engine ('1.1.1'): started and ready to provide service.
Dec 24 11:33:47 corosync [MAIN  ] Corosync built-in features:
Dec 24 11:33:47 corosync [MAIN  ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
Dec 24 11:33:47 corosync [TOTEM ] Initializing transport (UDP/IP).
Dec 24 11:33:47 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Dec 24 11:33:47 corosync [TOTEM ] Initializing transport (UDP/IP).
Dec 24 11:33:47 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Dec 24 11:33:47 corosync [MAIN  ] Compatibility mode set to whitetank.  Using V1 and V2 of the synchronization engine.
Dec 24 11:33:48 corosync [TOTEM ] The network interface [192.168.2.2] is now up.
Dec 24 11:33:48 corosync [pcmk  ] info: process_ais_conf: Reading configure
Dec 24 11:33:48 corosync [pcmk  ] info: config_find_init: Local handle: 4730966301143465985 for logging
Dec 24 11:33:48 corosync [pcmk  ] info: config_find_next: Processing additional logging options...
Dec 24 11:33:48 corosync [pcmk  ] info: get_config_opt: Found 'off' for option: debug
Dec 24 11:33:48 corosync [pcmk  ] info: get_config_opt: Defaulting to 'off' for option: to_file
Dec 24 11:33:48 corosync [pcmk  ] info: get_config_opt: Defaulting to 'daemon' for option: syslog_facility
Dec 24 11:33:48 corosync [pcmk  ] info: config_find_init: Local handle: 7739444317642555394 for service
Dec 24 11:33:48 corosync [pcmk  ] info: config_find_next: Processing additional service options...
Dec 24 11:33:48 corosync [pcmk  ] info: get_config_opt: Defaulting to 'no' for option: use_logd
Dec 24 11:33:48 corosync [pcmk  ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd
Dec 24 11:33:48 corosync [pcmk  ] info: pcmk_startup: CRM: Initialized
Dec 24 11:33:48 corosync [pcmk  ] Logging: Initialized pcmk_startup
Dec 24 11:33:48 corosync [pcmk  ] info: pcmk_startup: Maximum core file size is: 18446744073709551615
Dec 24 11:33:48 corosync [pcmk  ] info: pcmk_startup: Service: 9
Dec 24 11:33:48 corosync [pcmk  ] info: pcmk_startup: Local hostname: node2.itactics.com
Dec 24 11:33:48 corosync [pcmk  ] info: pcmk_update_nodeid: Local node id: 33728704
Dec 24 11:33:48 corosync [pcmk  ] info: update_member: Creating entry for node 33728704 born on 0
Dec 24 11:33:48 corosync [pcmk  ] info: update_member: 0x54d570 Node 33728704 now known as node2.itactics.com (was: (null))
Dec 24 11:33:48 corosync [pcmk  ] info: update_member: Node node2.itactics.com now has 1 quorum votes (was 0)
Dec 24 11:33:48 corosync [pcmk  ] info: update_member: Node 33728704/node2.itactics.com is now: member
Dec 24 11:33:48 corosync [pcmk  ] info: spawn_child: Forked child 8248 for process stonithd
Dec 24 11:33:48 corosync [pcmk  ] info: spawn_child: Forked child 8249 for process cib
Dec 24 11:33:48 corosync [pcmk  ] info: spawn_child: Forked child 8250 for process lrmd
Dec 24 11:33:48 corosync [pcmk  ] info: spawn_child: Forked child 8251 for process attrd
Dec 24 11:33:48 corosync [pcmk  ] info: spawn_child: Forked child 8252 for process pengine
Dec 24 11:33:48 corosync [pcmk  ] info: spawn_child: Forked child 8253 for process crmd
Dec 24 11:33:48 corosync [SERV  ] Service engine loaded: Pacemaker Cluster Manager 1.0.5
Dec 24 11:33:48 corosync [SERV  ] Service engine loaded: corosync extended virtual synchrony service
Dec 24 11:33:48 corosync [SERV  ] Service engine loaded: corosync configuration service
Dec 24 11:33:48 corosync [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01
Dec 24 11:33:48 corosync [SERV  ] Service engine loaded: corosync cluster config database access v1.01
Dec 24 11:33:48 corosync [SERV  ] Service engine loaded: corosync profile loading service
Dec 24 11:33:48 corosync [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Dec 24 11:33:48 corosync [TOTEM ] The network interface [192.168.2.2] is now up.
Dec 24 11:33:48 corosync [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 12: memb=0, new=0, lost=0
Dec 24 11:33:48 corosync [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 12: memb=1, new=1, lost=0
Dec 24 11:33:48 corosync [pcmk  ] info: pcmk_peer_update: NEW:  node2.itactics.com 33728704
Dec 24 11:33:48 corosync [pcmk  ] info: pcmk_peer_update: MEMB: node2.itactics.com 33728704
Dec 24 11:33:48 corosync [pcmk  ] info: update_member: Node node2.itactics.com now has process list: 00000000000000000000000000013312 (78610)
Dec 24 11:33:48 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Dec 24 11:33:48 corosync [TOTEM ] Marking seqid 10 ringid 0 interface 192.168.2.2 FAULTY - adminisrtative intervention required.
Dec 24 11:33:48 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Dec 24 11:33:48 corosync [pcmk  ] info: pcmk_ipc: Recorded connection 0x551ba0 for attrd/8251
Dec 24 11:34:21 corosync [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 20: memb=1, new=0, lost=0
Dec 24 11:34:21 corosync [pcmk  ] info: pcmk_peer_update: memb: node2.itactics.com 33728704
Dec 24 11:34:21 corosync [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 20: memb=2, new=1, lost=0
Dec 24 11:34:21 corosync [pcmk  ] info: update_member: Creating entry for node 16951488 born on 20
Dec 24 11:34:21 corosync [pcmk  ] info: update_member: Node 16951488/unknown is now: member
Dec 24 11:34:21 corosync [pcmk  ] info: pcmk_peer_update: NEW:  .pending. 16951488
Dec 24 11:34:21 corosync [pcmk  ] info: pcmk_peer_update: MEMB: .pending. 16951488
Dec 24 11:34:21 corosync [pcmk  ] info: pcmk_peer_update: MEMB: node2.itactics.com 33728704
Dec 24 11:34:21 corosync [pcmk  ] info: send_member_notification: Sending membership update 20 to 0 children
Dec 24 11:34:21 corosync [pcmk  ] info: update_member: 0x54d570 Node 33728704 ((null)) born on: 20
Dec 24 11:34:21 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Dec 24 11:34:21 corosync [pcmk  ] info: update_member: 0x550b20 Node 16951488 (node1.itactics.com) born on: 20
Dec 24 11:34:21 corosync [pcmk  ] info: update_member: 0x550b20 Node 16951488 now known as node1.itactics.com (was: (null))
Dec 24 11:34:21 corosync [pcmk  ] info: update_member: Node node1.itactics.com now has process list: 00000000000000000000000000013312 (78610)
Dec 24 11:34:21 corosync [pcmk  ] info: update_member: Node node1.itactics.com now has 1 quorum votes (was 0)
Dec 24 11:34:21 corosync [pcmk  ] info: send_member_notification: Sending membership update 20 to 0 children
Dec 24 11:34:21 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Dec 24 11:34:22 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.crmd failed: unknown (rc=-2)
Dec 24 11:34:23 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.crmd failed: unknown (rc=-2)
Dec 24 11:35:16 corosync [pcmk  ] info: pcmk_ipc: Recorded connection 0x553010 for crmd/8253
Dec 24 11:35:16 corosync [pcmk  ] info: pcmk_ipc: Sending membership update 20 to crmd
Dec 24 11:35:16 corosync [pcmk  ] info: pcmk_ipc_exit: Client crmd (conn=0x553010, async-conn=0x553010) left
Dec 24 11:35:16 corosync [pcmk  ] ERROR: pcmk_wait_dispatch: Child process crmd exited (pid=8253, rc=2)
Dec 24 11:35:16 corosync [pcmk  ] notice: pcmk_wait_dispatch: Respawning failed child process: crmd
Dec 24 11:35:16 corosync [pcmk  ] info: spawn_child: Forked child 8393 for process crmd
Dec 24 11:35:33 corosync [pcmk  ] info: pcmk_ipc_exit: Client attrd (conn=0x551ba0, async-conn=0x551ba0) left
Dec 24 11:35:33 corosync [pcmk  ] ERROR: pcmk_wait_dispatch: Child process attrd exited (pid=8251, rc=100)
Dec 24 11:35:33 corosync [pcmk  ] notice: pcmk_wait_dispatch: Child process attrd no longer wishes to be respawned
Dec 24 11:35:33 corosync [pcmk  ] info: update_member: Node node2.itactics.com now has process list: 00000000000000000000000000012312 (74514)
Dec 24 11:35:33 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.crmd failed: unknown (rc=-2)
Dec 24 11:37:23 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.crmd failed: unknown (rc=-2)
Dec 24 11:39:23 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.cib failed: unknown (rc=-2)
Dec 24 11:39:23 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.crmd failed: unknown (rc=-2)
Dec 24 11:42:23 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.cib failed: unknown (rc=-2)
Dec 24 11:42:23 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.cib failed: unknown (rc=-2)
Dec 24 11:42:23 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.cib failed: unknown (rc=-2)
Dec 24 11:42:23 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.attrd failed: unknown (rc=-2)
Dec 24 11:42:23 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.attrd failed: unknown (rc=-2)
Dec 24 11:42:23 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.cib failed: unknown (rc=-2)
Dec 24 11:42:23 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.cib failed: unknown (rc=-2)
Dec 24 11:42:23 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.cib failed: unknown (rc=-2)
Dec 24 11:42:23 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.cib failed: unknown (rc=-2)
Dec 24 11:42:24 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.cib failed: unknown (rc=-2)
Dec 24 11:42:24 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.cib failed: unknown (rc=-2)
Dec 24 11:42:33 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.cib failed: unknown (rc=-2)
Dec 24 11:42:33 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.attrd failed: unknown (rc=-2)
Dec 24 11:42:33 corosync [pcmk  ] WARN: route_ais_message: Sending message to local.cib failed: unknown (rc=-2)
Dec 24 11:50:11 corosync [TOTEM ] A processor failed, forming new configuration.
Dec 24 11:50:13 corosync [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 24: memb=1, new=0, lost=1
Dec 24 11:50:13 corosync [pcmk  ] info: pcmk_peer_update: memb: node2.itactics.com 33728704
Dec 24 11:50:13 corosync [pcmk  ] info: pcmk_peer_update: lost: node1.itactics.com 16951488
Dec 24 11:50:13 corosync [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 24: memb=1, new=0, lost=0
Dec 24 11:50:13 corosync [pcmk  ] info: pcmk_peer_update: MEMB: node2.itactics.com 33728704
Dec 24 11:50:13 corosync [pcmk  ] info: ais_mark_unseen_peer_dead: Node node1.itactics.com was not seen in the previous transition
Dec 24 11:50:13 corosync [pcmk  ] info: update_member: Node 16951488/node1.itactics.com is now: lost
Dec 24 11:50:13 corosync [pcmk  ] info: send_member_notification: Sending membership update 24 to 0 children
Dec 24 11:50:13 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Dec 24 11:50:13 corosync [MAIN  ] Completed service synchronization, ready to provide service.