Apr 10 14:06:25 corosync [MAIN ] Corosync Cluster Engine ('1.4.1'): started and ready to provide service. Apr 10 14:06:25 corosync [MAIN ] Corosync built-in features: nss dbus rdma snmp Apr 10 14:06:25 corosync [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'. Apr 10 14:06:25 corosync [TOTEM ] Initializing transport (UDP/IP Multicast). Apr 10 14:06:25 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0). Apr 10 14:06:25 corosync [TOTEM ] The network interface [172.16.68.117] is now up. Set r/w permissions for uid=0, gid=0 on /var/log/cluster/corosync.log Apr 10 14:06:25 corosync [pcmk ] info: process_ais_conf: Reading configure Apr 10 14:06:25 corosync [pcmk ] ERROR: process_ais_conf: You have configured a cluster using the Pacemaker plugin for Corosync. The plugin is not supported in this environment and will be removed very soon. Apr 10 14:06:25 corosync [pcmk ] ERROR: process_ais_conf: Please see Chapter 8 of 'Clusters from Scratch' (http://www.clusterlabs.org/doc) for details on using Pacemaker with CMAN Apr 10 14:06:25 corosync [pcmk ] info: config_find_init: Local handle: 4730966301143465985 for logging Apr 10 14:06:25 corosync [pcmk ] info: config_find_next: Processing additional logging options... Apr 10 14:06:25 corosync [pcmk ] info: get_config_opt: Found 'off' for option: debug Apr 10 14:06:25 corosync [pcmk ] info: get_config_opt: Found 'yes' for option: to_logfile Apr 10 14:06:25 corosync [pcmk ] info: get_config_opt: Found '/var/log/cluster/corosync.log' for option: logfile Apr 10 14:06:26 corosync [pcmk ] info: get_config_opt: Found 'yes' for option: to_syslog Apr 10 14:06:26 corosync [pcmk ] info: get_config_opt: Defaulting to 'daemon' for option: syslog_facility Apr 10 14:06:26 corosync [pcmk ] info: config_find_init: Local handle: 7739444317642555394 for quorum Apr 10 14:06:26 corosync [pcmk ] info: config_find_next: No additional configuration supplied for: quorum Apr 10 14:06:26 corosync [pcmk ] info: get_config_opt: No default for option: provider Apr 10 14:06:26 corosync [pcmk ] info: config_find_init: Local handle: 5650605097994944515 for service Apr 10 14:06:26 corosync [pcmk ] info: config_find_next: Processing additional service options... Apr 10 14:06:26 corosync [pcmk ] info: get_config_opt: Found '1' for option: ver Apr 10 14:06:26 corosync [pcmk ] info: process_ais_conf: Enabling MCP mode: Use the Pacemaker init script to complete Pacemaker startup Apr 10 14:06:26 corosync [pcmk ] info: get_config_opt: Defaulting to 'pcmk' for option: clustername Apr 10 14:06:26 corosync [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_logd Apr 10 14:06:26 corosync [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd Apr 10 14:06:26 corosync [pcmk ] info: pcmk_startup: CRM: Initialized Apr 10 14:06:26 corosync [pcmk ] Logging: Initialized pcmk_startup Apr 10 14:06:26 corosync [pcmk ] info: pcmk_startup: Maximum core file size is: 18446744073709551615 Apr 10 14:06:26 corosync [pcmk ] info: pcmk_startup: Service: 9 Apr 10 14:06:26 corosync [pcmk ] info: pcmk_startup: Local hostname: vsanqa4 Apr 10 14:06:26 corosync [pcmk ] info: pcmk_update_nodeid: Local node id: 1967394988 Apr 10 14:06:26 corosync [pcmk ] info: update_member: Creating entry for node 1967394988 born on 0 Apr 10 14:06:26 corosync [pcmk ] info: update_member: 0x2756590 Node 1967394988 now known as vsanqa4 (was: (null)) Apr 10 14:06:26 corosync [pcmk ] info: update_member: Node vsanqa4 now has 1 quorum votes (was 0) Apr 10 14:06:26 corosync [pcmk ] info: update_member: Node 1967394988/vsanqa4 is now: member Apr 10 14:06:26 corosync [SERV ] Service engine loaded: Pacemaker Cluster Manager 1.1.8 Apr 10 14:06:26 corosync [SERV ] Service engine loaded: corosync extended virtual synchrony service Apr 10 14:06:26 corosync [SERV ] Service engine loaded: corosync configuration service Apr 10 14:06:26 corosync [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 Apr 10 14:06:26 corosync [SERV ] Service engine loaded: corosync cluster config database access v1.01 Apr 10 14:06:26 corosync [SERV ] Service engine loaded: corosync profile loading service Apr 10 14:06:26 corosync [SERV ] Service engine loaded: corosync cluster quorum service v0.1 Apr 10 14:06:26 corosync [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine. Apr 10 14:06:26 corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 166048: memb=0, new=0, lost=0 Apr 10 14:06:26 corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 166048: memb=1, new=1, lost=0 Apr 10 14:06:26 corosync [pcmk ] info: pcmk_peer_update: NEW: vsanqa4 1967394988 Apr 10 14:06:26 corosync [pcmk ] info: pcmk_peer_update: MEMB: vsanqa4 1967394988 Apr 10 14:06:26 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed. Apr 10 14:06:26 corosync [CPG ] chosen downlist: sender r(0) ip(172.16.68.117) ; members(old:0 left:0) Apr 10 14:06:26 corosync [MAIN ] Completed service synchronization, ready to provide service. Set r/w permissions for uid=495, gid=489 on /var/log/cluster/corosync.log Apr 10 14:06:31 [3380] vsanqa4 pacemakerd: notice: main: Starting Pacemaker 1.1.8-7.el6 (Build: 394e906): generated-manpages agent-manpages ascii-docs publican-docs ncurses libqb-logging libqb-ipc corosync-plugin cman Apr 10 14:06:31 [3380] vsanqa4 pacemakerd: info: main: Maximum core file size is: 18446744073709551615 Apr 10 14:06:31 [3380] vsanqa4 pacemakerd: info: qb_ipcs_us_publish: server name: pacemakerd Apr 10 14:06:31 [3380] vsanqa4 pacemakerd: notice: get_local_node_name: Defaulting to uname(2).nodename for the local classic openais (with plugin) node name Apr 10 14:06:31 [3380] vsanqa4 pacemakerd: notice: update_node_processes: 0xf39f70 Node 1967394988 now known as vsanqa4, was: Apr 10 14:06:31 [3380] vsanqa4 pacemakerd: info: start_child: Forked child 3386 for process cib Apr 10 14:06:31 [3380] vsanqa4 pacemakerd: info: start_child: Forked child 3387 for process stonith-ng Apr 10 14:06:31 [3380] vsanqa4 pacemakerd: info: start_child: Forked child 3388 for process lrmd Apr 10 14:06:31 [3380] vsanqa4 pacemakerd: info: start_child: Forked child 3389 for process attrd Apr 10 14:06:31 [3380] vsanqa4 pacemakerd: info: start_child: Forked child 3390 for process pengine Apr 10 14:06:31 [3380] vsanqa4 pacemakerd: info: start_child: Forked child 3391 for process crmd Apr 10 14:06:31 [3380] vsanqa4 pacemakerd: info: main: Starting mainloop Apr 10 14:06:31 [3387] vsanqa4 stonith-ng: info: crm_log_init: Changed active directory to /var/lib/pacemaker/cores/root Apr 10 14:06:31 [3387] vsanqa4 stonith-ng: info: get_cluster_type: Cluster type is: 'openais' Apr 10 14:06:31 [3387] vsanqa4 stonith-ng: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin) Apr 10 14:06:31 [3387] vsanqa4 stonith-ng: info: init_cs_connection_classic: Creating connection to our Corosync plugin Apr 10 14:06:31 [3386] vsanqa4 cib: info: crm_log_init: Cannot change active directory to /var/lib/pacemaker/cores/hacluster: Permission denied (13) Apr 10 14:06:31 [3388] vsanqa4 lrmd: info: crm_log_init: Changed active directory to /var/lib/pacemaker/cores/root Apr 10 14:06:31 [3388] vsanqa4 lrmd: info: qb_ipcs_us_publish: server name: lrmd Apr 10 14:06:31 [3388] vsanqa4 lrmd: info: main: Starting Apr 10 14:06:31 [3389] vsanqa4 attrd: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin) Apr 10 14:06:31 [3387] vsanqa4 stonith-ng: info: init_cs_connection_classic: AIS connection established Apr 10 14:06:31 corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x27688e0 for stonith-ng/0 Apr 10 14:06:31 [3387] vsanqa4 stonith-ng: info: get_ais_nodeid: Server details: id=1967394988 uname=vsanqa4 cname=pcmk Apr 10 14:06:31 [3387] vsanqa4 stonith-ng: info: init_cs_connection_once: Connection to 'classic openais (with plugin)': established Apr 10 14:06:31 [3387] vsanqa4 stonith-ng: info: crm_get_peer: Node vsanqa4 now has id: 2195325672 Apr 10 14:06:31 [3387] vsanqa4 stonith-ng: info: crm_get_peer: Node 2195325672 is now known as vsanqa4 Apr 10 14:06:31 [3387] vsanqa4 stonith-ng: info: crm_get_peer: Node 2195325672 has uuid vsanqa4 Apr 10 14:06:31 [3390] vsanqa4 pengine: info: crm_log_init: Cannot change active directory to /var/lib/pacemaker/cores/hacluster: Permission denied (13) Apr 10 14:06:31 [3390] vsanqa4 pengine: info: qb_ipcs_us_publish: server name: pengine Apr 10 14:06:31 [3390] vsanqa4 pengine: info: main: Starting pengine Apr 10 14:06:31 [3387] vsanqa4 stonith-ng: info: crm_ipc_connect: Could not establish cib_rw connection: Connection refused (111) Apr 10 14:06:31 corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x276cc40 for attrd/0 Apr 10 14:06:31 [3391] vsanqa4 crmd: info: crm_log_init: Cannot change active directory to /var/lib/pacemaker/cores/hacluster: Permission denied (13) Apr 10 14:06:31 [3391] vsanqa4 crmd: notice: main: CRM Git Version: 394e906 Apr 10 14:06:31 [3389] vsanqa4 attrd: notice: main: Starting mainloop... Apr 10 14:06:31 [3391] vsanqa4 crmd: info: get_cluster_type: Cluster type is: 'openais' Apr 10 14:06:31 [3391] vsanqa4 crmd: info: crm_ipc_connect: Could not establish cib_shm connection: Connection refused (111) Apr 10 14:06:31 [3386] vsanqa4 cib: notice: main: Using legacy config location: /var/lib/heartbeat/crm Apr 10 14:06:31 [3386] vsanqa4 cib: info: get_cluster_type: Cluster type is: 'openais' Apr 10 14:06:31 [3386] vsanqa4 cib: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) Apr 10 14:06:31 [3386] vsanqa4 cib: info: validate_with_relaxng: Creating RNG parser context Apr 10 14:06:31 [3386] vsanqa4 cib: info: startCib: CIB Initialization completed successfully Apr 10 14:06:31 [3386] vsanqa4 cib: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin) Apr 10 14:06:31 [3386] vsanqa4 cib: info: init_cs_connection_classic: Creating connection to our Corosync plugin Apr 10 14:06:31 [3386] vsanqa4 cib: info: init_cs_connection_classic: AIS connection established Apr 10 14:06:31 corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x2771470 for cib/0 Apr 10 14:06:31 corosync [pcmk ] info: pcmk_ipc: Sending membership update 166048 to cib Apr 10 14:06:31 [3386] vsanqa4 cib: info: get_ais_nodeid: Server details: id=1967394988 uname=vsanqa4 cname=pcmk Apr 10 14:06:31 [3386] vsanqa4 cib: info: init_cs_connection_once: Connection to 'classic openais (with plugin)': established Apr 10 14:06:31 [3386] vsanqa4 cib: info: qb_ipcs_us_publish: server name: cib_ro Apr 10 14:06:31 [3386] vsanqa4 cib: info: qb_ipcs_us_publish: server name: cib_rw Apr 10 14:06:31 [3386] vsanqa4 cib: info: qb_ipcs_us_publish: server name: cib_shm Apr 10 14:06:31 [3386] vsanqa4 cib: info: cib_init: Starting cib mainloop Apr 10 14:06:31 [3386] vsanqa4 cib: info: ais_dispatch_message: Membership 166048: quorum still lost Apr 10 14:06:31 [3386] vsanqa4 cib: info: crm_get_peer: Node vsanqa4 now has id: 1967394988 Apr 10 14:06:31 [3386] vsanqa4 cib: info: crm_get_peer: Node 1967394988 is now known as vsanqa4 Apr 10 14:06:31 [3386] vsanqa4 cib: info: crm_get_peer: Node 1967394988 has uuid vsanqa4 Apr 10 14:06:31 [3386] vsanqa4 cib: notice: crm_update_peer_state: crm_update_ais_node: Node vsanqa4[1967394988] - state is now member Apr 10 14:06:31 [3386] vsanqa4 cib: info: crm_update_peer: crm_update_ais_node: Node vsanqa4: id=1967394988 state=member addr=r(0) ip(172.16.68.117) (new) votes=1 (new) born=0 seen=166048 proc=00000000000000000000000000000000 Apr 10 14:06:31 [3386] vsanqa4 cib: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa4[1967394988] - unknown is now (null) Apr 10 14:06:32 [3387] vsanqa4 stonith-ng: notice: setup_cib: Watching for stonith topology changes Apr 10 14:06:32 [3387] vsanqa4 stonith-ng: info: qb_ipcs_us_publish: server name: stonith-ng Apr 10 14:06:32 [3387] vsanqa4 stonith-ng: info: main: Starting stonith-ng mainloop Apr 10 14:06:32 [3387] vsanqa4 stonith-ng: info: crm_get_peer: Node vsanqa4 now has id: 1967394988 Apr 10 14:06:32 [3387] vsanqa4 stonith-ng: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa4[1967394988] - unknown is now (null) Apr 10 14:06:32 [3391] vsanqa4 crmd: info: do_cib_control: CIB connection established Apr 10 14:06:32 [3391] vsanqa4 crmd: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin) Apr 10 14:06:32 [3391] vsanqa4 crmd: info: init_cs_connection_classic: Creating connection to our Corosync plugin Apr 10 14:06:32 [3391] vsanqa4 crmd: info: init_cs_connection_classic: AIS connection established Apr 10 14:06:32 corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x27760e0 for crmd/0 Apr 10 14:06:32 corosync [pcmk ] info: pcmk_ipc: Sending membership update 166048 to crmd Apr 10 14:06:32 [3391] vsanqa4 crmd: info: get_ais_nodeid: Server details: id=1967394988 uname=vsanqa4 cname=pcmk Apr 10 14:06:32 [3391] vsanqa4 crmd: info: init_cs_connection_once: Connection to 'classic openais (with plugin)': established Apr 10 14:06:32 [3391] vsanqa4 crmd: info: do_ha_control: Connected to the cluster Apr 10 14:06:32 [3391] vsanqa4 crmd: info: lrmd_api_connect: Connecting to lrmd Apr 10 14:06:32 [3388] vsanqa4 lrmd: info: lrmd_ipc_accept: Accepting client connection: 0x16149f0 pid=3391 for uid=495 gid=0 Apr 10 14:06:32 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/3, version=0.666.1): OK (rc=0) Apr 10 14:06:32 [3391] vsanqa4 crmd: info: do_started: Delaying start, no membership data (0000000000100000) Apr 10 14:06:32 [3391] vsanqa4 crmd: info: do_started: Delaying start, no membership data (0000000000100000) Apr 10 14:06:32 [3391] vsanqa4 crmd: info: ais_dispatch_message: Membership 166048: quorum still lost Apr 10 14:06:32 [3391] vsanqa4 crmd: info: crm_get_peer: Node vsanqa4 now has id: 1967394988 Apr 10 14:06:32 [3391] vsanqa4 crmd: info: crm_get_peer: Node 1967394988 is now known as vsanqa4 Apr 10 14:06:32 [3391] vsanqa4 crmd: info: peer_update_callback: vsanqa4 is now (null) Apr 10 14:06:32 [3391] vsanqa4 crmd: info: crm_get_peer: Node 1967394988 has uuid vsanqa4 Apr 10 14:06:32 [3391] vsanqa4 crmd: notice: crm_update_peer_state: crm_update_ais_node: Node vsanqa4[1967394988] - state is now member Apr 10 14:06:32 [3391] vsanqa4 crmd: info: peer_update_callback: vsanqa4 is now member (was (null)) Apr 10 14:06:32 [3391] vsanqa4 crmd: info: crm_update_peer: crm_update_ais_node: Node vsanqa4: id=1967394988 state=member addr=r(0) ip(172.16.68.117) (new) votes=1 (new) born=0 seen=166048 proc=00000000000000000000000000000000 Apr 10 14:06:32 [3391] vsanqa4 crmd: info: ais_dispatch_message: Membership 166048: quorum still lost Apr 10 14:06:32 [3391] vsanqa4 crmd: error: crmd_ais_dispatch: Recieving messages from a node we think is dead: vsanqa4[1967394988] Apr 10 14:06:32 [3391] vsanqa4 crmd: info: crm_update_peer_proc: crmd_ais_dispatch: Node vsanqa4[1967394988] - ais is now online Apr 10 14:06:32 [3391] vsanqa4 crmd: info: peer_update_callback: Client vsanqa4/peer now has status [offline] (DC=) Apr 10 14:06:32 [3391] vsanqa4 crmd: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa4[1967394988] - unknown is now (null) Apr 10 14:06:32 [3391] vsanqa4 crmd: info: peer_update_callback: Client vsanqa4/peer now has status [online] (DC=) Apr 10 14:06:32 [3391] vsanqa4 crmd: info: qb_ipcs_us_publish: server name: crmd Apr 10 14:06:32 [3391] vsanqa4 crmd: notice: do_started: The local CRM is operational Apr 10 14:06:32 [3391] vsanqa4 crmd: info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ] Apr 10 14:06:34 [3387] vsanqa4 stonith-ng: info: stonith_command: Processed register from crmd.3391: OK (0) Apr 10 14:06:34 [3387] vsanqa4 stonith-ng: info: stonith_command: Processed st_notify from crmd.3391: OK (0) Apr 10 14:06:34 [3387] vsanqa4 stonith-ng: info: stonith_command: Processed st_notify from crmd.3391: OK (0) Apr 10 14:06:44 corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 166056: memb=1, new=0, lost=0 Apr 10 14:06:44 corosync [pcmk ] info: pcmk_peer_update: memb: vsanqa4 1967394988 Apr 10 14:06:44 corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 166056: memb=2, new=1, lost=0 Apr 10 14:06:44 corosync [pcmk ] info: update_member: Creating entry for node 1950617772 born on 166056 Apr 10 14:06:44 corosync [pcmk ] info: update_member: Node 1950617772/unknown is now: member Apr 10 14:06:44 corosync [pcmk ] info: pcmk_peer_update: NEW: .pending. 1950617772 Apr 10 14:06:44 corosync [pcmk ] info: pcmk_peer_update: MEMB: .pending. 1950617772 Apr 10 14:06:44 corosync [pcmk ] info: pcmk_peer_update: MEMB: vsanqa4 1967394988 Apr 10 14:06:44 corosync [pcmk ] info: send_member_notification: Sending membership update 166056 to 2 children Apr 10 14:06:44 [3391] vsanqa4 crmd: info: ais_dispatch_message: Membership 166056: quorum still lost Apr 10 14:06:44 corosync [pcmk ] info: update_member: 0x2756590 Node 1967394988 ((null)) born on: 166048 Apr 10 14:06:44 [3391] vsanqa4 crmd: info: crm_get_peer: Node now has id: 1950617772 Apr 10 14:06:44 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed. Apr 10 14:06:44 [3391] vsanqa4 crmd: notice: crm_update_peer_state: crm_update_ais_node: Node (null)[1950617772] - state is now member Apr 10 14:06:44 [3391] vsanqa4 crmd: info: crm_update_peer: crm_update_ais_node: Node (null): id=1950617772 state=member addr=r(0) ip(172.16.68.116) (new) votes=0 born=0 seen=166056 proc=00000000000000000000000000000000 Apr 10 14:06:44 [3386] vsanqa4 cib: info: ais_dispatch_message: Membership 166056: quorum still lost Apr 10 14:06:44 [3386] vsanqa4 cib: info: crm_get_peer: Node now has id: 1950617772 Apr 10 14:06:44 [3386] vsanqa4 cib: notice: crm_update_peer_state: crm_update_ais_node: Node (null)[1950617772] - state is now member Apr 10 14:06:44 [3386] vsanqa4 cib: info: crm_update_peer: crm_update_ais_node: Node (null): id=1950617772 state=member addr=r(0) ip(172.16.68.116) (new) votes=0 born=0 seen=166056 proc=00000000000000000000000000000000 Apr 10 14:06:44 corosync [pcmk ] info: update_member: 0x277a860 Node 1950617772 (vsanqa3) born on: 166056 Apr 10 14:06:44 corosync [pcmk ] info: update_member: 0x277a860 Node 1950617772 now known as vsanqa3 (was: (null)) Apr 10 14:06:44 corosync [pcmk ] info: update_member: Node vsanqa3 now has 1 quorum votes (was 0) Apr 10 14:06:44 corosync [pcmk ] info: send_member_notification: Sending membership update 166056 to 2 children Apr 10 14:06:44 [3386] vsanqa4 cib: notice: ais_dispatch_message: Membership 166056: quorum acquired Apr 10 14:06:44 [3386] vsanqa4 cib: info: crm_get_peer: Node 1950617772 is now known as vsanqa3 Apr 10 14:06:44 [3386] vsanqa4 cib: info: crm_get_peer: Node 1950617772 has uuid vsanqa3 Apr 10 14:06:44 [3386] vsanqa4 cib: info: crm_update_peer: crm_update_ais_node: Node vsanqa3: id=1950617772 state=member addr=r(0) ip(172.16.68.116) votes=1 (new) born=166056 seen=166056 proc=00000000000000000000000000000000 Apr 10 14:06:44 [3391] vsanqa4 crmd: notice: ais_dispatch_message: Membership 166056: quorum acquired Apr 10 14:06:44 [3391] vsanqa4 crmd: info: crm_get_peer: Node 1950617772 is now known as vsanqa3 Apr 10 14:06:44 [3391] vsanqa4 crmd: info: peer_update_callback: vsanqa3 is now member Apr 10 14:06:44 [3391] vsanqa4 crmd: info: crm_get_peer: Node 1950617772 has uuid vsanqa3 Apr 10 14:06:44 [3391] vsanqa4 crmd: info: crm_update_peer: crm_update_ais_node: Node vsanqa3: id=1950617772 state=member addr=r(0) ip(172.16.68.116) votes=1 (new) born=166056 seen=166056 proc=00000000000000000000000000000000 Apr 10 14:06:44 corosync [CPG ] chosen downlist: sender r(0) ip(172.16.68.116) ; members(old:1 left:0) Apr 10 14:06:44 corosync [MAIN ] Completed service synchronization, ready to provide service. Apr 10 14:06:50 [3380] vsanqa4 pacemakerd: notice: update_node_processes: 0xf385e0 Node 1950617772 now known as vsanqa3, was: Apr 10 14:06:50 [3387] vsanqa4 stonith-ng: info: crm_get_peer: Node vsanqa3 now has id: 1950617772 Apr 10 14:06:50 [3387] vsanqa4 stonith-ng: info: crm_get_peer: Node 1950617772 is now known as vsanqa3 Apr 10 14:06:50 [3391] vsanqa4 crmd: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - ais is now (null) Apr 10 14:06:50 [3391] vsanqa4 crmd: info: peer_update_callback: Client vsanqa3/peer now has status [offline] (DC=) Apr 10 14:06:50 [3386] vsanqa4 cib: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - ais is now (null) Apr 10 14:06:50 [3387] vsanqa4 stonith-ng: info: crm_get_peer: Node 1950617772 has uuid vsanqa3 Apr 10 14:06:50 [3387] vsanqa4 stonith-ng: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - ais is now (null) Apr 10 14:06:50 [3386] vsanqa4 cib: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3387] vsanqa4 stonith-ng: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3386] vsanqa4 cib: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3391] vsanqa4 crmd: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3387] vsanqa4 stonith-ng: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3391] vsanqa4 crmd: info: peer_update_callback: Client vsanqa3/peer now has status [offline] (DC=) Apr 10 14:06:50 [3391] vsanqa4 crmd: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3391] vsanqa4 crmd: info: peer_update_callback: Client vsanqa3/peer now has status [offline] (DC=) Apr 10 14:06:50 [3387] vsanqa4 stonith-ng: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3391] vsanqa4 crmd: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3391] vsanqa4 crmd: info: peer_update_callback: Client vsanqa3/peer now has status [offline] (DC=) Apr 10 14:06:50 [3386] vsanqa4 cib: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3387] vsanqa4 stonith-ng: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3386] vsanqa4 cib: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3386] vsanqa4 cib: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3387] vsanqa4 stonith-ng: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3391] vsanqa4 crmd: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3391] vsanqa4 crmd: info: peer_update_callback: Client vsanqa3/peer now has status [offline] (DC=) Apr 10 14:06:50 [3391] vsanqa4 crmd: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3391] vsanqa4 crmd: info: peer_update_callback: Client vsanqa3/peer now has status [offline] (DC=) Apr 10 14:06:50 [3387] vsanqa4 stonith-ng: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3386] vsanqa4 cib: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3391] vsanqa4 crmd: info: crm_update_peer_proc: pcmk_mcp_dispatch: Node vsanqa3[1950617772] - unknown is now (null) Apr 10 14:06:50 [3391] vsanqa4 crmd: info: peer_update_callback: Client vsanqa3/peer now has status [online] (DC=) Apr 10 14:06:51 [3386] vsanqa4 cib: warning: cib_process_diff: Diff 0.666.0 -> 0.666.1 from vsanqa3 not applied to 0.666.1: current "num_updates" is greater than required Apr 10 14:06:53 [3391] vsanqa4 crmd: info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped (20000ms) Apr 10 14:06:53 [3391] vsanqa4 crmd: warning: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING Apr 10 14:06:53 [3391] vsanqa4 crmd: info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped ] Apr 10 14:06:53 [3391] vsanqa4 crmd: notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ] Apr 10 14:06:53 [3391] vsanqa4 crmd: info: do_te_control: Registering TE UUID: 3cc2762e-4c5f-4962-a00c-9ba349fd3e5d Apr 10 14:06:53 [3391] vsanqa4 crmd: info: set_graph_functions: Setting custom graph functions Apr 10 14:06:53 [3391] vsanqa4 crmd: info: do_dc_takeover: Taking over DC status for this partition Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_readwrite: We are now in R/W mode Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/6, version=0.666.2): OK (rc=0) Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/7, version=0.666.3): OK (rc=0) Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/9, version=0.666.4): OK (rc=0) Apr 10 14:06:53 [3391] vsanqa4 crmd: info: join_make_offer: Making join offers based on membership 166056 Apr 10 14:06:53 [3391] vsanqa4 crmd: info: do_dc_join_offer_all: join-1: Waiting on 2 outstanding join acks Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/11, version=0.666.5): OK (rc=0) Apr 10 14:06:53 [3391] vsanqa4 crmd: info: ais_dispatch_message: Membership 166056: quorum retained Apr 10 14:06:53 [3391] vsanqa4 crmd: info: crmd_ais_dispatch: Setting expected votes to 2 Apr 10 14:06:53 [3391] vsanqa4 crmd: info: update_dc: Set DC to vsanqa4 (3.0.7) Apr 10 14:06:53 [3391] vsanqa4 crmd: info: ais_dispatch_message: Membership 166056: quorum retained Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/14, version=0.666.6): OK (rc=0) Apr 10 14:06:53 [3391] vsanqa4 crmd: info: crmd_ais_dispatch: Setting expected votes to 2 Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/17, version=0.666.7): OK (rc=0) Apr 10 14:06:53 [3391] vsanqa4 crmd: info: crm_update_peer_expected: do_dc_join_filter_offer: Node vsanqa4[1967394988] - expected state is now member Apr 10 14:06:53 [3391] vsanqa4 crmd: info: crm_update_peer_expected: do_dc_join_filter_offer: Node vsanqa3[1950617772] - expected state is now member Apr 10 14:06:53 [3391] vsanqa4 crmd: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] Apr 10 14:06:53 [3391] vsanqa4 crmd: info: do_dc_join_finalize: join-1: Syncing the CIB from vsanqa3 to the rest of the cluster Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_replace: Digest matched on replace from vsanqa3: 2ff09d71673a375be326050869596cb9 Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_replace: Replaced 0.666.7 with 0.666.7 from vsanqa3 Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=vsanqa3/vsanqa3/18, version=0.666.7): OK (rc=0) Apr 10 14:06:53 [3391] vsanqa4 crmd: info: erase_status_tag: Deleting xpath: //node_state[@uname='vsanqa4']/transient_attributes Apr 10 14:06:53 [3391] vsanqa4 crmd: info: update_attrd: Connecting to attrd... 5 retries remaining Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/19, version=0.666.8): OK (rc=0) Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/20, version=0.666.9): OK (rc=0) Apr 10 14:06:53 [3391] vsanqa4 crmd: info: do_dc_join_ack: join-1: Updating node state to member for vsanqa4 Apr 10 14:06:53 [3391] vsanqa4 crmd: info: erase_status_tag: Deleting xpath: //node_state[@uname='vsanqa4']/lrm Apr 10 14:06:53 [3391] vsanqa4 crmd: info: do_dc_join_ack: join-1: Updating node state to member for vsanqa3 Apr 10 14:06:53 [3391] vsanqa4 crmd: info: erase_status_tag: Deleting xpath: //node_state[@uname='vsanqa3']/lrm Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='vsanqa4']/transient_attributes (origin=local/crmd/21, version=0.666.10): OK (rc=0) Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='vsanqa4']/lrm (origin=local/crmd/22, version=0.666.11): OK (rc=0) Apr 10 14:06:53 [3391] vsanqa4 crmd: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] Apr 10 14:06:53 [3391] vsanqa4 crmd: info: abort_transition_graph: do_te_invoke:156 - Triggered transition abort (complete=1) : Peer Cancelled Apr 10 14:06:53 [3389] vsanqa4 attrd: notice: attrd_local_callback: Sending full refresh (origin=crmd) Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='vsanqa3']/lrm (origin=local/crmd/24, version=0.666.13): OK (rc=0) Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/26, version=0.666.15): OK (rc=0) Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/28, version=0.666.17): OK (rc=0) Apr 10 14:06:53 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='vsanqa3']/transient_attributes (origin=vsanqa3/crmd/8, version=0.666.18): OK (rc=0) Apr 10 14:06:54 [3390] vsanqa4 pengine: info: unpack_config: Startup probes: enabled Apr 10 14:06:54 [3390] vsanqa4 pengine: notice: unpack_config: On loss of CCM Quorum: Ignore Apr 10 14:06:54 [3390] vsanqa4 pengine: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Apr 10 14:06:54 [3390] vsanqa4 pengine: info: unpack_domains: Unpacking domains Apr 10 14:06:54 [3390] vsanqa4 pengine: info: determine_online_status: Node vsanqa4 is online Apr 10 14:06:54 [3390] vsanqa4 pengine: info: determine_online_status: Node vsanqa3 is online Apr 10 14:06:54 [3390] vsanqa4 pengine: notice: process_pe_message: Calculated Transition 0: /var/lib/pacemaker/pengine/pe-input-389.bz2 Apr 10 14:06:54 [3391] vsanqa4 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Apr 10 14:06:54 [3391] vsanqa4 crmd: info: do_te_invoke: Processing graph 0 (ref=pe_calc-dc-1365628014-11) derived from /var/lib/pacemaker/pengine/pe-input-389.bz2 Apr 10 14:06:54 [3391] vsanqa4 crmd: info: te_rsc_command: Initiating action 3: probe_complete probe_complete on vsanqa4 (local) - no waiting Apr 10 14:06:54 [3391] vsanqa4 crmd: info: te_rsc_command: Initiating action 2: probe_complete probe_complete on vsanqa3 - no waiting Apr 10 14:06:54 [3389] vsanqa4 attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) Apr 10 14:06:54 [3391] vsanqa4 crmd: notice: run_graph: Transition 0 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-389.bz2): Complete Apr 10 14:06:54 [3391] vsanqa4 crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Apr 10 14:06:54 [3389] vsanqa4 attrd: notice: attrd_perform_update: Sent update 4: probe_complete=true Apr 10 14:06:54 [3389] vsanqa4 attrd: notice: attrd_perform_update: Sent update 6: probe_complete=true Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_replace_notify: Replaced: 0.666.21 -> 0.667.1 from Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: Diff: --- 0.666.21 Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: Diff: +++ 0.667.1 de1df9bdb3733dd14f8aec4c395ccc82 Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: -- Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3391] vsanqa4 crmd: info: abort_transition_graph: te_update_diff:126 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.667.1) : Non-status change Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3391] vsanqa4 crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3391] vsanqa4 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced ] Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_replace for section 'all' (origin=local/cibadmin/2, version=0.667.1): OK (rc=0) Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/31, version=0.667.2): OK (rc=0) Apr 10 14:11:43 [3391] vsanqa4 crmd: notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ] Apr 10 14:11:43 [3391] vsanqa4 crmd: info: do_dc_takeover: Taking over DC status for this partition Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/34, version=0.667.4): OK (rc=0) Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/35, version=0.667.5): OK (rc=0) Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/37, version=0.667.6): OK (rc=0) Apr 10 14:11:43 [3391] vsanqa4 crmd: info: do_dc_join_offer_all: join-2: Waiting on 2 outstanding join acks Apr 10 14:11:43 [3391] vsanqa4 crmd: info: ais_dispatch_message: Membership 166056: quorum retained Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/39, version=0.667.7): OK (rc=0) Apr 10 14:11:43 [3391] vsanqa4 crmd: info: crmd_ais_dispatch: Setting expected votes to 2 Apr 10 14:11:43 [3391] vsanqa4 crmd: info: update_dc: Set DC to vsanqa4 (3.0.7) Apr 10 14:11:43 [3391] vsanqa4 crmd: info: ais_dispatch_message: Membership 166056: quorum retained Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/42, version=0.667.9): OK (rc=0) Apr 10 14:11:43 [3391] vsanqa4 crmd: info: crmd_ais_dispatch: Setting expected votes to 2 Apr 10 14:11:43 [3391] vsanqa4 crmd: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] Apr 10 14:11:43 [3391] vsanqa4 crmd: info: do_dc_join_finalize: join-2: Syncing the CIB from vsanqa4 to the rest of the cluster Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/45, version=0.667.10): OK (rc=0) Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/46, version=0.667.10): OK (rc=0) Apr 10 14:11:43 corosync [TOTEM ] Retransmit List: 89 8a 8b 8c Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/47, version=0.667.11): OK (rc=0) Apr 10 14:11:43 [3391] vsanqa4 crmd: info: do_dc_join_ack: join-2: Updating node state to member for vsanqa4 Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/48, version=0.667.12): OK (rc=0) Apr 10 14:11:43 [3391] vsanqa4 crmd: info: erase_status_tag: Deleting xpath: //node_state[@uname='vsanqa4']/lrm Apr 10 14:11:43 [3391] vsanqa4 crmd: info: do_dc_join_ack: join-2: Updating node state to member for vsanqa3 Apr 10 14:11:43 [3391] vsanqa4 crmd: info: erase_status_tag: Deleting xpath: //node_state[@uname='vsanqa3']/lrm Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='vsanqa4']/lrm (origin=local/crmd/49, version=0.667.13): OK (rc=0) Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='vsanqa3']/lrm (origin=local/crmd/51, version=0.667.15): OK (rc=0) Apr 10 14:11:43 [3391] vsanqa4 crmd: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] Apr 10 14:11:43 [3389] vsanqa4 attrd: notice: attrd_local_callback: Sending full refresh (origin=crmd) Apr 10 14:11:43 [3389] vsanqa4 attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) Apr 10 14:11:43 [3391] vsanqa4 crmd: info: abort_transition_graph: do_te_invoke:156 - Triggered transition abort (complete=1) : Peer Cancelled Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/53, version=0.667.17): OK (rc=0) Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/55, version=0.667.19): OK (rc=0) Apr 10 14:11:43 corosync [TOTEM ] Retransmit List: a3 Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: Diff: --- 0.667.23 Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: Diff: +++ 0.668.1 cbf56573a1b5df9055cc72cd3ea68f53 Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: -- Apr 10 14:11:43 [3386] vsanqa4 cib: notice: cib:diff: ++ Apr 10 14:11:43 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_replace for section resources (origin=local/cibadmin/2, version=0.668.1): OK (rc=0) Apr 10 14:11:44 [3391] vsanqa4 crmd: info: abort_transition_graph: te_update_diff:126 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.668.1) : Non-status change Apr 10 14:11:44 [3390] vsanqa4 pengine: info: unpack_config: Startup probes: enabled Apr 10 14:11:44 [3390] vsanqa4 pengine: notice: unpack_config: On loss of CCM Quorum: Ignore Apr 10 14:11:44 [3390] vsanqa4 pengine: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Apr 10 14:11:44 [3390] vsanqa4 pengine: info: unpack_domains: Unpacking domains Apr 10 14:11:44 [3390] vsanqa4 pengine: info: determine_online_status: Node vsanqa4 is online Apr 10 14:11:44 [3390] vsanqa4 pengine: info: determine_online_status: Node vsanqa3 is online Apr 10 14:11:44 [3390] vsanqa4 pengine: info: clone_print: Master/Slave Set: ms-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e [vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e] Apr 10 14:11:44 [3390] vsanqa4 pengine: info: short_print: Stopped: [ vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0 vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:1 ] Apr 10 14:11:44 [3390] vsanqa4 pengine: info: master_color: ms-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e: Promoted 0 instances of a possible 1 to master Apr 10 14:11:44 [3390] vsanqa4 pengine: info: RecurringOp: Start recurring monitor (31s) for vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0 on vsanqa3 Apr 10 14:11:44 [3390] vsanqa4 pengine: info: RecurringOp: Start recurring monitor (31s) for vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:1 on vsanqa4 Apr 10 14:11:44 [3390] vsanqa4 pengine: info: RecurringOp: Start recurring monitor (31s) for vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0 on vsanqa3 Apr 10 14:11:44 [3390] vsanqa4 pengine: info: RecurringOp: Start recurring monitor (31s) for vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:1 on vsanqa4 Apr 10 14:11:44 [3390] vsanqa4 pengine: notice: LogActions: Start vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0 (vsanqa3) Apr 10 14:11:44 [3390] vsanqa4 pengine: notice: LogActions: Start vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:1 (vsanqa4) Apr 10 14:11:44 [3390] vsanqa4 pengine: notice: process_pe_message: Calculated Transition 1: /var/lib/pacemaker/pengine/pe-input-390.bz2 Apr 10 14:11:44 [3391] vsanqa4 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Apr 10 14:11:44 [3391] vsanqa4 crmd: info: do_te_invoke: Processing graph 1 (ref=pe_calc-dc-1365628304-21) derived from /var/lib/pacemaker/pengine/pe-input-390.bz2 Apr 10 14:11:44 [3391] vsanqa4 crmd: info: te_rsc_command: Initiating action 4: monitor vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0_monitor_0 on vsanqa3 Apr 10 14:11:44 [3391] vsanqa4 crmd: info: te_rsc_command: Initiating action 6: monitor vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:1_monitor_0 on vsanqa4 (local) Apr 10 14:11:44 [3388] vsanqa4 lrmd: info: process_lrmd_get_rsc_info: Resource 'vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e' not found (0 active resources) Apr 10 14:11:44 [3388] vsanqa4 lrmd: info: process_lrmd_get_rsc_info: Resource 'vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:1' not found (0 active resources) Apr 10 14:11:44 [3388] vsanqa4 lrmd: info: process_lrmd_rsc_register: Added 'vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e' to the rsc list (1 active resources) Apr 10 14:11:45 [3391] vsanqa4 crmd: info: services_os_action_execute: Managed vgc-cm-agent.ocf_meta-data_0 process 3756 exited with rc=0 Apr 10 14:11:45 [3391] vsanqa4 crmd: notice: process_lrm_event: LRM operation vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_0 (call=6, rc=7, cib-update=58, confirmed=true) not running Apr 10 14:11:45 [3391] vsanqa4 crmd: info: te_rsc_command: Initiating action 5: probe_complete probe_complete on vsanqa4 (local) - no waiting Apr 10 14:11:45 [3391] vsanqa4 crmd: info: te_rsc_command: Initiating action 3: probe_complete probe_complete on vsanqa3 - no waiting Apr 10 14:11:45 [3391] vsanqa4 crmd: info: te_rsc_command: Initiating action 7: start vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0_start_0 on vsanqa3 Apr 10 14:11:45 [3391] vsanqa4 crmd: info: te_rsc_command: Initiating action 9: start vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:1_start_0 on vsanqa4 (local) vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[3760]: 2013/04/10_14:11:45 INFO: vhacm_start vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[3760]: 2013/04/10_14:11:45 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[3760]: 2013/04/10_14:11:45 INFO: vhacm_monitor: Resource is SECONDARY Apr 10 14:11:45 [3391] vsanqa4 crmd: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=0, tag=nvpair, id=status-vsanqa3-master-vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e, name=master-vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e, value=2, magic=NA, cib=0.668.4) : Transient attribute: update Apr 10 14:11:45 [3389] vsanqa4 attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: master-vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e (1) Apr 10 14:11:45 [3389] vsanqa4 attrd: notice: attrd_perform_update: Sent update 18: master-vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e=1 Apr 10 14:11:45 [3391] vsanqa4 crmd: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=0, tag=nvpair, id=status-vsanqa4-master-vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e, name=master-vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e, value=1, magic=NA, cib=0.668.6) : Transient attribute: update vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[3760]: 2013/04/10_14:11:45 INFO: vhacm_monitor: returning 0 Apr 10 14:11:45 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_start_0:3760 [ 2013/04/10_14:11:45 INFO: vhacm_start ] Apr 10 14:11:45 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_start_0:3760 [ 2013/04/10_14:11:45 INFO: vhacm_monitor ] Apr 10 14:11:45 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_start_0:3760 [ 2013/04/10_14:11:45 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:11:45 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_start_0:3760 [ 2013/04/10_14:11:45 INFO: vhacm_monitor: returning 0 ] Apr 10 14:11:45 [3391] vsanqa4 crmd: notice: process_lrm_event: LRM operation vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_start_0 (call=9, rc=0, cib-update=59, confirmed=true) ok Apr 10 14:11:45 [3391] vsanqa4 crmd: notice: run_graph: Transition 1 (Complete=9, Pending=0, Fired=0, Skipped=2, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-390.bz2): Stopped Apr 10 14:11:45 [3391] vsanqa4 crmd: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Apr 10 14:11:45 [3390] vsanqa4 pengine: info: unpack_config: Startup probes: enabled Apr 10 14:11:45 [3390] vsanqa4 pengine: notice: unpack_config: On loss of CCM Quorum: Ignore Apr 10 14:11:45 [3390] vsanqa4 pengine: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Apr 10 14:11:45 [3390] vsanqa4 pengine: info: unpack_domains: Unpacking domains Apr 10 14:11:45 [3390] vsanqa4 pengine: info: determine_online_status: Node vsanqa4 is online Apr 10 14:11:45 [3390] vsanqa4 pengine: info: determine_online_status: Node vsanqa3 is online Apr 10 14:11:45 [3390] vsanqa4 pengine: info: find_anonymous_clone: Internally renamed vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e on vsanqa4 to vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0 Apr 10 14:11:45 [3390] vsanqa4 pengine: info: find_anonymous_clone: Internally renamed vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e on vsanqa3 to vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:1 Apr 10 14:11:45 [3390] vsanqa4 pengine: info: clone_print: Master/Slave Set: ms-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e [vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e] Apr 10 14:11:45 [3390] vsanqa4 pengine: info: short_print: Slaves: [ vsanqa3 vsanqa4 ] Apr 10 14:11:45 [3390] vsanqa4 pengine: info: master_color: Promoting vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:1 (Slave vsanqa3) Apr 10 14:11:45 [3390] vsanqa4 pengine: info: master_color: ms-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e: Promoted 1 instances of a possible 1 to master Apr 10 14:11:45 [3390] vsanqa4 pengine: info: RecurringOp: Start recurring monitor (31s) for vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0 on vsanqa4 Apr 10 14:11:45 [3390] vsanqa4 pengine: info: RecurringOp: Start recurring monitor (30s) for vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:1 on vsanqa3 Apr 10 14:11:45 [3390] vsanqa4 pengine: info: RecurringOp: Start recurring monitor (31s) for vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0 on vsanqa4 Apr 10 14:11:45 [3390] vsanqa4 pengine: info: RecurringOp: Start recurring monitor (30s) for vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:1 on vsanqa3 Apr 10 14:11:45 [3390] vsanqa4 pengine: info: LogActions: Leave vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0 (Slave vsanqa4) Apr 10 14:11:45 [3390] vsanqa4 pengine: notice: LogActions: Promote vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:1 (Slave -> Master vsanqa3) Apr 10 14:11:45 [3390] vsanqa4 pengine: notice: process_pe_message: Calculated Transition 2: /var/lib/pacemaker/pengine/pe-input-391.bz2 Apr 10 14:11:45 [3391] vsanqa4 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Apr 10 14:11:45 [3391] vsanqa4 crmd: info: do_te_invoke: Processing graph 2 (ref=pe_calc-dc-1365628305-28) derived from /var/lib/pacemaker/pengine/pe-input-391.bz2 Apr 10 14:11:45 [3391] vsanqa4 crmd: info: te_rsc_command: Initiating action 7: monitor vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000 on vsanqa4 (local) Apr 10 14:11:45 [3391] vsanqa4 crmd: info: te_rsc_command: Initiating action 10: promote vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_promote_0 on vsanqa3 vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[3847]: 2013/04/10_14:11:45 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[3847]: 2013/04/10_14:11:45 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[3847]: 2013/04/10_14:11:45 INFO: vhacm_monitor: returning 0 Apr 10 14:11:45 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:3847 [ 2013/04/10_14:11:45 INFO: vhacm_monitor ] Apr 10 14:11:45 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:3847 [ 2013/04/10_14:11:45 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:11:45 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:3847 [ 2013/04/10_14:11:45 INFO: vhacm_monitor: returning 0 ] Apr 10 14:11:45 [3391] vsanqa4 crmd: notice: process_lrm_event: LRM operation vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000 (call=12, rc=0, cib-update=61, confirmed=false) ok Apr 10 14:11:45 [3391] vsanqa4 crmd: info: te_rsc_command: Initiating action 11: monitor vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_30000 on vsanqa3 Apr 10 14:11:46 [3391] vsanqa4 crmd: notice: run_graph: Transition 2 (Complete=5, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-391.bz2): Complete Apr 10 14:11:46 [3391] vsanqa4 crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[3931]: 2013/04/10_14:12:16 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[3931]: 2013/04/10_14:12:16 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[3931]: 2013/04/10_14:12:16 INFO: vhacm_monitor: returning 0 Apr 10 14:12:16 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:3931 [ 2013/04/10_14:12:16 INFO: vhacm_monitor ] Apr 10 14:12:16 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:3931 [ 2013/04/10_14:12:16 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:12:16 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:3931 [ 2013/04/10_14:12:16 INFO: vhacm_monitor: returning 0 ] Apr 10 14:12:32 corosync [TOTEM ] A processor failed, forming new configuration. Apr 10 14:12:38 corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 166060: memb=1, new=0, lost=1 Apr 10 14:12:38 corosync [pcmk ] info: pcmk_peer_update: memb: vsanqa4 1967394988 Apr 10 14:12:38 corosync [pcmk ] info: pcmk_peer_update: lost: vsanqa3 1950617772 Apr 10 14:12:38 corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 166060: memb=1, new=0, lost=0 Apr 10 14:12:38 corosync [pcmk ] info: pcmk_peer_update: MEMB: vsanqa4 1967394988 Apr 10 14:12:38 corosync [pcmk ] info: ais_mark_unseen_peer_dead: Node vsanqa3 was not seen in the previous transition Apr 10 14:12:38 corosync [pcmk ] info: update_member: Node 1950617772/vsanqa3 is now: lost Apr 10 14:12:38 corosync [pcmk ] info: send_member_notification: Sending membership update 166060 to 2 children Apr 10 14:12:38 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed. Apr 10 14:12:38 [3386] vsanqa4 cib: notice: ais_dispatch_message: Membership 166060: quorum lost Apr 10 14:12:38 [3391] vsanqa4 crmd: notice: ais_dispatch_message: Membership 166060: quorum lost Apr 10 14:12:38 [3386] vsanqa4 cib: notice: crm_update_peer_state: crm_update_ais_node: Node vsanqa3[1950617772] - state is now lost Apr 10 14:12:38 [3391] vsanqa4 crmd: notice: crm_update_peer_state: crm_update_ais_node: Node vsanqa3[1950617772] - state is now lost Apr 10 14:12:38 [3391] vsanqa4 crmd: info: peer_update_callback: vsanqa3 is now lost (was member) Apr 10 14:12:38 corosync [CPG ] chosen downlist: sender r(0) ip(172.16.68.117) ; members(old:2 left:1) Apr 10 14:12:38 corosync [MAIN ] Completed service synchronization, ready to provide service. Apr 10 14:12:38 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/62, version=0.668.12): OK (rc=0) Apr 10 14:12:38 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/64, version=0.668.14): OK (rc=0) Apr 10 14:12:38 [3391] vsanqa4 crmd: info: crmd_ais_dispatch: Setting expected votes to 2 Apr 10 14:12:38 [3386] vsanqa4 cib: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/66, version=0.668.15): OK (rc=0) vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4016]: 2013/04/10_14:12:47 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4016]: 2013/04/10_14:12:47 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4016]: 2013/04/10_14:12:47 INFO: vhacm_monitor: returning 0 Apr 10 14:12:47 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4016 [ 2013/04/10_14:12:47 INFO: vhacm_monitor ] Apr 10 14:12:47 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4016 [ 2013/04/10_14:12:47 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:12:47 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4016 [ 2013/04/10_14:12:47 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4099]: 2013/04/10_14:13:18 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4099]: 2013/04/10_14:13:19 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4099]: 2013/04/10_14:13:19 INFO: vhacm_monitor: returning 0 Apr 10 14:13:19 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4099 [ 2013/04/10_14:13:18 INFO: vhacm_monitor ] Apr 10 14:13:19 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4099 [ 2013/04/10_14:13:19 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:13:19 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4099 [ 2013/04/10_14:13:19 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4185]: 2013/04/10_14:13:50 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4185]: 2013/04/10_14:13:50 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4185]: 2013/04/10_14:13:50 INFO: vhacm_monitor: returning 0 Apr 10 14:13:50 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4185 [ 2013/04/10_14:13:50 INFO: vhacm_monitor ] Apr 10 14:13:50 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4185 [ 2013/04/10_14:13:50 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:13:50 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4185 [ 2013/04/10_14:13:50 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4268]: 2013/04/10_14:14:21 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4268]: 2013/04/10_14:14:21 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4268]: 2013/04/10_14:14:21 INFO: vhacm_monitor: returning 0 Apr 10 14:14:21 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4268 [ 2013/04/10_14:14:21 INFO: vhacm_monitor ] Apr 10 14:14:21 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4268 [ 2013/04/10_14:14:21 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:14:21 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4268 [ 2013/04/10_14:14:21 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4351]: 2013/04/10_14:14:52 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4351]: 2013/04/10_14:14:52 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4351]: 2013/04/10_14:14:52 INFO: vhacm_monitor: returning 0 Apr 10 14:14:52 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4351 [ 2013/04/10_14:14:52 INFO: vhacm_monitor ] Apr 10 14:14:52 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4351 [ 2013/04/10_14:14:52 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:14:52 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4351 [ 2013/04/10_14:14:52 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4436]: 2013/04/10_14:15:23 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4436]: 2013/04/10_14:15:23 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4436]: 2013/04/10_14:15:23 INFO: vhacm_monitor: returning 0 Apr 10 14:15:23 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4436 [ 2013/04/10_14:15:23 INFO: vhacm_monitor ] Apr 10 14:15:23 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4436 [ 2013/04/10_14:15:23 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:15:23 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4436 [ 2013/04/10_14:15:23 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4520]: 2013/04/10_14:15:54 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4520]: 2013/04/10_14:15:54 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4520]: 2013/04/10_14:15:54 INFO: vhacm_monitor: returning 0 Apr 10 14:15:54 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4520 [ 2013/04/10_14:15:54 INFO: vhacm_monitor ] Apr 10 14:15:54 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4520 [ 2013/04/10_14:15:54 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:15:54 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4520 [ 2013/04/10_14:15:54 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4605]: 2013/04/10_14:16:25 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4605]: 2013/04/10_14:16:25 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4605]: 2013/04/10_14:16:25 INFO: vhacm_monitor: returning 0 Apr 10 14:16:25 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4605 [ 2013/04/10_14:16:25 INFO: vhacm_monitor ] Apr 10 14:16:25 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4605 [ 2013/04/10_14:16:25 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:16:25 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4605 [ 2013/04/10_14:16:25 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4687]: 2013/04/10_14:16:56 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4687]: 2013/04/10_14:16:56 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4687]: 2013/04/10_14:16:56 INFO: vhacm_monitor: returning 0 Apr 10 14:16:56 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4687 [ 2013/04/10_14:16:56 INFO: vhacm_monitor ] Apr 10 14:16:56 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4687 [ 2013/04/10_14:16:56 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:16:56 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4687 [ 2013/04/10_14:16:56 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4770]: 2013/04/10_14:17:27 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4770]: 2013/04/10_14:17:27 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4770]: 2013/04/10_14:17:27 INFO: vhacm_monitor: returning 0 Apr 10 14:17:27 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4770 [ 2013/04/10_14:17:27 INFO: vhacm_monitor ] Apr 10 14:17:27 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4770 [ 2013/04/10_14:17:27 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:17:27 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4770 [ 2013/04/10_14:17:27 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4855]: 2013/04/10_14:17:59 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4855]: 2013/04/10_14:17:59 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4855]: 2013/04/10_14:17:59 INFO: vhacm_monitor: returning 0 Apr 10 14:17:59 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4855 [ 2013/04/10_14:17:59 INFO: vhacm_monitor ] Apr 10 14:17:59 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4855 [ 2013/04/10_14:17:59 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:17:59 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4855 [ 2013/04/10_14:17:59 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4938]: 2013/04/10_14:18:30 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4938]: 2013/04/10_14:18:30 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[4938]: 2013/04/10_14:18:30 INFO: vhacm_monitor: returning 0 Apr 10 14:18:30 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4938 [ 2013/04/10_14:18:30 INFO: vhacm_monitor ] Apr 10 14:18:30 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4938 [ 2013/04/10_14:18:30 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:18:30 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:4938 [ 2013/04/10_14:18:30 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5021]: 2013/04/10_14:19:01 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5021]: 2013/04/10_14:19:01 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5021]: 2013/04/10_14:19:01 INFO: vhacm_monitor: returning 0 Apr 10 14:19:01 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5021 [ 2013/04/10_14:19:01 INFO: vhacm_monitor ] Apr 10 14:19:01 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5021 [ 2013/04/10_14:19:01 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:19:01 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5021 [ 2013/04/10_14:19:01 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5103]: 2013/04/10_14:19:32 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5103]: 2013/04/10_14:19:32 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5103]: 2013/04/10_14:19:32 INFO: vhacm_monitor: returning 0 Apr 10 14:19:32 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5103 [ 2013/04/10_14:19:32 INFO: vhacm_monitor ] Apr 10 14:19:32 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5103 [ 2013/04/10_14:19:32 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:19:32 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5103 [ 2013/04/10_14:19:32 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5188]: 2013/04/10_14:20:03 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5188]: 2013/04/10_14:20:03 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5188]: 2013/04/10_14:20:03 INFO: vhacm_monitor: returning 0 Apr 10 14:20:03 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5188 [ 2013/04/10_14:20:03 INFO: vhacm_monitor ] Apr 10 14:20:03 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5188 [ 2013/04/10_14:20:03 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:20:03 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5188 [ 2013/04/10_14:20:03 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5271]: 2013/04/10_14:20:34 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5271]: 2013/04/10_14:20:34 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5271]: 2013/04/10_14:20:34 INFO: vhacm_monitor: returning 0 Apr 10 14:20:34 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5271 [ 2013/04/10_14:20:34 INFO: vhacm_monitor ] Apr 10 14:20:34 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5271 [ 2013/04/10_14:20:34 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:20:34 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5271 [ 2013/04/10_14:20:34 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5354]: 2013/04/10_14:21:05 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5354]: 2013/04/10_14:21:05 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5354]: 2013/04/10_14:21:05 INFO: vhacm_monitor: returning 0 Apr 10 14:21:05 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5354 [ 2013/04/10_14:21:05 INFO: vhacm_monitor ] Apr 10 14:21:05 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5354 [ 2013/04/10_14:21:05 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:21:05 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5354 [ 2013/04/10_14:21:05 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5437]: 2013/04/10_14:21:36 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5437]: 2013/04/10_14:21:36 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5437]: 2013/04/10_14:21:36 INFO: vhacm_monitor: returning 0 Apr 10 14:21:36 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5437 [ 2013/04/10_14:21:36 INFO: vhacm_monitor ] Apr 10 14:21:36 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5437 [ 2013/04/10_14:21:36 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:21:36 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5437 [ 2013/04/10_14:21:36 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5519]: 2013/04/10_14:22:07 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5519]: 2013/04/10_14:22:07 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5519]: 2013/04/10_14:22:08 INFO: vhacm_monitor: returning 0 Apr 10 14:22:08 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5519 [ 2013/04/10_14:22:07 INFO: vhacm_monitor ] Apr 10 14:22:08 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5519 [ 2013/04/10_14:22:07 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:22:08 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5519 [ 2013/04/10_14:22:08 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5603]: 2013/04/10_14:22:39 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5603]: 2013/04/10_14:22:39 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5603]: 2013/04/10_14:22:39 INFO: vhacm_monitor: returning 0 Apr 10 14:22:39 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5603 [ 2013/04/10_14:22:39 INFO: vhacm_monitor ] Apr 10 14:22:39 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5603 [ 2013/04/10_14:22:39 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:22:39 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5603 [ 2013/04/10_14:22:39 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5685]: 2013/04/10_14:23:10 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5685]: 2013/04/10_14:23:10 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5685]: 2013/04/10_14:23:10 INFO: vhacm_monitor: returning 0 Apr 10 14:23:10 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5685 [ 2013/04/10_14:23:10 INFO: vhacm_monitor ] Apr 10 14:23:10 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5685 [ 2013/04/10_14:23:10 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:23:10 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5685 [ 2013/04/10_14:23:10 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5768]: 2013/04/10_14:23:41 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5768]: 2013/04/10_14:23:41 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5768]: 2013/04/10_14:23:41 INFO: vhacm_monitor: returning 0 Apr 10 14:23:41 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5768 [ 2013/04/10_14:23:41 INFO: vhacm_monitor ] Apr 10 14:23:41 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5768 [ 2013/04/10_14:23:41 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:23:41 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5768 [ 2013/04/10_14:23:41 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5851]: 2013/04/10_14:24:12 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5851]: 2013/04/10_14:24:12 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5851]: 2013/04/10_14:24:12 INFO: vhacm_monitor: returning 0 Apr 10 14:24:12 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5851 [ 2013/04/10_14:24:12 INFO: vhacm_monitor ] Apr 10 14:24:12 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5851 [ 2013/04/10_14:24:12 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:24:12 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5851 [ 2013/04/10_14:24:12 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5959]: 2013/04/10_14:24:43 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5959]: 2013/04/10_14:24:43 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[5959]: 2013/04/10_14:24:43 INFO: vhacm_monitor: returning 0 Apr 10 14:24:43 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5959 [ 2013/04/10_14:24:43 INFO: vhacm_monitor ] Apr 10 14:24:43 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5959 [ 2013/04/10_14:24:43 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:24:43 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:5959 [ 2013/04/10_14:24:43 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[6042]: 2013/04/10_14:25:14 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[6042]: 2013/04/10_14:25:14 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[6042]: 2013/04/10_14:25:14 INFO: vhacm_monitor: returning 0 Apr 10 14:25:14 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:6042 [ 2013/04/10_14:25:14 INFO: vhacm_monitor ] Apr 10 14:25:14 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:6042 [ 2013/04/10_14:25:14 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:25:14 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:6042 [ 2013/04/10_14:25:14 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[6124]: 2013/04/10_14:25:45 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[6124]: 2013/04/10_14:25:45 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[6124]: 2013/04/10_14:25:45 INFO: vhacm_monitor: returning 0 Apr 10 14:25:45 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:6124 [ 2013/04/10_14:25:45 INFO: vhacm_monitor ] Apr 10 14:25:45 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:6124 [ 2013/04/10_14:25:45 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:25:45 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:6124 [ 2013/04/10_14:25:45 INFO: vhacm_monitor: returning 0 ] vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[6207]: 2013/04/10_14:26:16 INFO: vhacm_monitor vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[6207]: 2013/04/10_14:26:16 INFO: vhacm_monitor: Resource is SECONDARY vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[6207]: 2013/04/10_14:26:16 INFO: vhacm_monitor: returning 0 Apr 10 14:26:16 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:6207 [ 2013/04/10_14:26:16 INFO: vhacm_monitor ] Apr 10 14:26:16 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:6207 [ 2013/04/10_14:26:16 INFO: vhacm_monitor: Resource is SECONDARY ] Apr 10 14:26:16 [3388] vsanqa4 lrmd: notice: operation_finished: vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000:6207 [ 2013/04/10_14:26:16 INFO: vhacm_monitor: returning 0 ] Apr 10 14:26:46 [3391] vsanqa4 crmd: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped (900000ms) Apr 10 14:26:46 [3391] vsanqa4 crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Apr 10 14:26:46 [3391] vsanqa4 crmd: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Apr 10 14:26:46 [3390] vsanqa4 pengine: info: unpack_config: Startup probes: enabled Apr 10 14:26:46 [3390] vsanqa4 pengine: notice: unpack_config: On loss of CCM Quorum: Ignore Apr 10 14:26:46 [3390] vsanqa4 pengine: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Apr 10 14:26:46 [3390] vsanqa4 pengine: info: unpack_domains: Unpacking domains Apr 10 14:26:46 [3390] vsanqa4 pengine: info: determine_online_status: Node vsanqa4 is online Apr 10 14:26:46 [3390] vsanqa4 pengine: info: find_anonymous_clone: Internally renamed vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e on vsanqa4 to vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0 Apr 10 14:26:46 [3390] vsanqa4 pengine: info: clone_print: Master/Slave Set: ms-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e [vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e] Apr 10 14:26:46 [3390] vsanqa4 pengine: info: short_print: Slaves: [ vsanqa4 ] Apr 10 14:26:46 [3390] vsanqa4 pengine: info: short_print: Stopped: [ vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:1 ] Apr 10 14:26:46 [3390] vsanqa4 pengine: info: native_color: Resource vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:1 cannot run anywhere Apr 10 14:26:46 [3390] vsanqa4 pengine: info: master_color: Promoting vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0 (Slave vsanqa4) Apr 10 14:26:46 [3390] vsanqa4 pengine: info: master_color: ms-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e: Promoted 1 instances of a possible 1 to master Apr 10 14:26:46 [3390] vsanqa4 pengine: info: RecurringOp: Start recurring monitor (30s) for vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0 on vsanqa4 Apr 10 14:26:46 [3390] vsanqa4 pengine: info: RecurringOp: Cancelling action vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0_monitor_31000 (Slave vs. Master) Apr 10 14:26:46 [3390] vsanqa4 pengine: info: RecurringOp: Start recurring monitor (30s) for vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0 on vsanqa4 Apr 10 14:26:46 [3390] vsanqa4 pengine: info: RecurringOp: Cancelling action vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0_monitor_31000 (Slave vs. Master) Apr 10 14:26:46 [3390] vsanqa4 pengine: notice: LogActions: Promote vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0 (Slave -> Master vsanqa4) Apr 10 14:26:46 [3390] vsanqa4 pengine: info: LogActions: Leave vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:1 (Stopped) Apr 10 14:26:46 [3390] vsanqa4 pengine: notice: process_pe_message: Calculated Transition 3: /var/lib/pacemaker/pengine/pe-input-392.bz2 Apr 10 14:26:46 [3391] vsanqa4 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Apr 10 14:26:46 [3391] vsanqa4 crmd: info: do_te_invoke: Processing graph 3 (ref=pe_calc-dc-1365629206-33) derived from /var/lib/pacemaker/pengine/pe-input-392.bz2 Apr 10 14:26:46 [3391] vsanqa4 crmd: info: te_rsc_command: Initiating action 1: cancel vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_cancel_31000 on vsanqa4 (local) Apr 10 14:26:46 [3388] vsanqa4 lrmd: info: cancel_recurring_action: Cancelling operation vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000 Apr 10 14:26:46 [3391] vsanqa4 crmd: info: process_lrm_event: LRM operation vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000 (call=12, status=1, cib-update=0, confirmed=false) Cancelled Apr 10 14:26:46 [3391] vsanqa4 crmd: info: te_rsc_command: Initiating action 7: promote vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_promote_0 on vsanqa4 (local) Apr 10 14:26:46 [3391] vsanqa4 crmd: info: abort_transition_graph: te_update_diff:271 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e_monitor_31000, magic=0:0;7:2:0:3cc2762e-4c5f-4962-a00c-9ba349fd3e5d, cib=0.668.16) : Resource op removal vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[6290]: 2013/04/10_14:26:46 INFO: vhacm_promote vgc-cm-agent.ocf(vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e)[6290]: 2013/04/10_14:26:46 INFO: vhacm_monitor