Hi,<br><br>Following is the oracle instance startup problem.<br><br>I am able to successfully start oracle listener using LCMC. However when i tried to start oracle instance the start process seems to continue for ever and ever. The database version used is oracle 10gR2. Please find the logs and version details specified below. Please help me in this regards.<br>
<br><br>pacemaker : pacemaker 1.1.7-6-el6.x86_64<br>corosync : corosync 1.4.1-7.el6_3.1.x86_64<br>LCMC : 1.4.5<br><br><br><br><b>following is the system log.</b><br><br>Jan 9 09:23:18 VirtKirchoff pengine[7561]: notice: LogActions: Start res_oracle_1#011(VirtKirchoff)<br>
Jan 9 09:23:18 VirtKirchoff crmd[7562]: info: te_rsc_command: Initiating action 71: start res_oracle_1_start_0 on VirtKirchoff (local)<br>Jan 9 09:23:18 VirtKirchoff lrmd: [7559]: info: rsc:res_oracle_1:33: start<br>
Jan 9 09:23:18 VirtKirchoff crmd[7562]: info: handle_failcount_op: Removing failcount for res_oracle_1<br>Jan 9 09:23:18 VirtKirchoff attrd[7560]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-res_oracle_1 (<null>)<br>
Jan 9 09:23:18 VirtKirchoff cib[7557]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='VirtKirchoff']//lrm_resource[@id='res_oracle_1']/lrm_rsc_op[@id='res_oracle_1_last_failure_0'] (origin=local/crmd/99, version=0.267.12): ok (rc=0)<br>
Jan 9 09:23:18 VirtKirchoff crmd[7562]: info: abort_transition_graph: te_update_diff:320 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=res_oracle_1_last_failure_0, magic=0:1;71:8:0:b1f3b4d6-6433-4642-91c6-810b06f62afc, cib=0.267.12) : Resource op removal<br>
Jan 9 09:23:18 VirtKirchoff attrd[7560]: notice: attrd_perform_update: Sent delete 47: node=VirtKirchoff, attr=fail-count-res_oracle_1, id=<n/a>, set=(null), section=status<br>Jan 9 09:23:18 VirtKirchoff attrd[7560]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-res_oracle_1 (<null>)<br>
Jan 9 09:23:18 VirtKirchoff attrd[7560]: notice: attrd_perform_update: Sent delete 49: node=VirtKirchoff, attr=last-failure-res_oracle_1, id=<n/a>, set=(null), section=status<br>Jan 9 09:23:18 VirtKirchoff attrd[7560]: notice: attrd_perform_update: Sent delete -22: node=VirtKirchoff, attr=fail-count-res_oracle_1, id=<n/a>, set=(null), section=status<br>
Jan 9 09:23:18 VirtKirchoff attrd[7560]: notice: attrd_perform_update: Sent delete -22: node=VirtKirchoff, attr=last-failure-res_oracle_1, id=<n/a>, set=(null), section=status<br>Jan 9 09:23:19 VirtKirchoff oracle(res_oracle_1)[13746]: INFO: ORA-01081 error found, trying to cleanup oracle (dbstart_mount output: ORA-01081: cannot start already-running ORACLE - shut it down first)<br>
Jan 9 09:23:19 VirtKirchoff lrmd: [7559]: info: RA output: (res_oracle_1:start:stderr) ls: cannot access /u01/app/oracle/product/10.2.0/db_1/dbs/lk*: No such file or directory<br>Jan 9 09:23:19 VirtKirchoff oracle(res_oracle_1)[13746]: ERROR: oracle ORCL can not be mounted (status: OPEN)<br>
Jan 9 09:23:19 VirtKirchoff crmd[7562]: info: process_lrm_event: LRM operation res_oracle_1_start_0 (call=33, rc=1, cib-update=100, confirmed=true) unknown error<br>Jan 9 09:23:19 VirtKirchoff crmd[7562]: warning: status_from_rc: Action 71 (res_oracle_1_start_0) on VirtKirchoff failed (target: 0 vs. rc: 1): Error<br>
Jan 9 09:23:19 VirtKirchoff crmd[7562]: warning: update_failcount: Updating failcount for res_oracle_1 on VirtKirchoff after failed start: rc=1 (update=value++, time=1357703599)<br>Jan 9 09:23:19 VirtKirchoff crmd[7562]: info: abort_transition_graph: match_graph_event:277 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=res_oracle_1_last_failure_0, magic=0:1;71:12:0:b1f3b4d6-6433-4642-91c6-810b06f62afc, cib=0.267.15) : Event failed<br>
Jan 9 09:23:19 VirtKirchoff attrd[7560]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-res_oracle_1 (1)<br>Jan 9 09:23:19 VirtKirchoff attrd[7560]: notice: attrd_perform_update: Sent update 54: fail-count-res_oracle_1=1<br>
Jan 9 09:23:19 VirtKirchoff attrd[7560]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-res_oracle_1 (1357703599)<br>Jan 9 09:23:19 VirtKirchoff crmd[7562]: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair, id=status-VirtKirchoff-fail-count-res_oracle_1, name=fail-count-res_oracle_1, value=1, magic=NA, cib=0.267.16) : Transient attribute: update<br>
Jan 9 09:23:19 VirtKirchoff attrd[7560]: notice: attrd_perform_update: Sent update 57: last-failure-res_oracle_1=1357703599<br>Jan 9 09:23:19 VirtKirchoff crmd[7562]: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair, id=status-VirtKirchoff-last-failure-res_oracle_1, name=last-failure-res_oracle_1, value=1357703599, magic=NA, cib=0.267.17) : Transient attribute: update<br>
Jan 9 09:23:19 VirtKirchoff pengine[7561]: warning: unpack_rsc_op: Processing failed op res_oracle_1_last_failure_0 on VirtKirchoff: unknown error (1)<br>Jan 9 09:23:19 VirtKirchoff pengine[7561]: notice: LogActions: Recover res_oracle_1#011(Started VirtKirchoff)<br>
Jan 9 09:23:19 VirtKirchoff pengine[7561]: warning: unpack_rsc_op: Processing failed op res_oracle_1_last_failure_0 on VirtKirchoff: unknown error (1)<br>Jan 9 09:23:19 VirtKirchoff pengine[7561]: warning: common_apply_stickiness: Forcing res_oracle_1 away from VirtKirchoff after 1 failures (max=1)<br>
Jan 9 09:23:19 VirtKirchoff pengine[7561]: notice: LogActions: Stop res_oracle_1#011(VirtKirchoff)<br>Jan 9 09:23:19 VirtKirchoff pengine[7561]: warning: unpack_rsc_op: Processing failed op res_oracle_1_last_failure_0 on VirtKirchoff: unknown error (1)<br>
Jan 9 09:23:19 VirtKirchoff pengine[7561]: warning: common_apply_stickiness: Forcing res_oracle_1 away from VirtKirchoff after 1 failures (max=1)<br>Jan 9 09:23:19 VirtKirchoff pengine[7561]: notice: LogActions: Stop res_oracle_1#011(VirtKirchoff)<br>
Jan 9 09:23:19 VirtKirchoff crmd[7562]: info: te_rsc_command: Initiating action 4: stop res_oracle_1_stop_0 on VirtKirchoff (local)<br>Jan 9 09:23:19 VirtKirchoff lrmd: [7559]: info: rsc:res_oracle_1:34: stop<br>Jan 9 09:23:19 VirtKirchoff oracle(res_oracle_1)[13870]: INFO: Oracle instance ORCL already stopped<br>
Jan 9 09:23:19 VirtKirchoff crmd[7562]: info: process_lrm_event: LRM operation res_oracle_1_stop_0 (call=34, rc=0, cib-update=104, confirmed=true) ok<br><br><b>Following is corosync log</b> <br><br><br>Jan 09 09:30:25 [7557] VirtKirchoff cib: info: cib:diff: + </instance_attributes><br>
Jan 09 09:30:25 [7557] VirtKirchoff cib: info: cib:diff: + </node><br>Jan 09 09:30:25 [7557] VirtKirchoff cib: info: cib:diff: + </nodes><br>Jan 09 09:30:25 [7557] VirtKirchoff cib: info: cib:diff: + </configuration><br>
Jan 09 09:30:25 [7557] VirtKirchoff cib: info: cib:diff: + </cib><br>Jan 09 09:30:25 [7557] VirtKirchoff cib: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crm_attribute/4, version=0.269.1): ok (rc=0)<br>
Jan 09 09:30:25 [7557] VirtKirchoff cib: info: set_crm_log_level: New log level: 3 0<br>Jan 09 09:30:25 [7561] VirtKirchoff pengine: notice: unpack_config: On loss of CCM Quorum: Ignore<br>Jan 09 09:30:25 [7561] VirtKirchoff pengine: warning: unpack_nodes: Blind faith: not fencing unseen nodes<br>
Jan 09 09:30:25 [7561] VirtKirchoff pengine: notice: unpack_rsc_op: Operation monitor found resource res_drbd_1:0 active on VirtTesla<br>Jan 09 09:30:25 [7561] VirtKirchoff pengine: notice: unpack_rsc_op: Operation monitor found resource res_drbd_2:0 active on VirtTesla<br>
Jan 09 09:30:25 [7561] VirtKirchoff pengine: notice: LogActions: Start res_drbd_1:1 (VirtTesla)<br>Jan 09 09:30:25 [7561] VirtKirchoff pengine: notice: LogActions: Start res_drbd_2:1 (VirtTesla)<br>
Jan 09 09:30:25 [7562] VirtKirchoff crmd: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<br>Jan 09 09:30:25 [7562] VirtKirchoff crmd: info: do_te_invoke: Processing graph 26 (ref=pe_calc-dc-1357704025-87) derived from /var/lib/pengine/pe-input-3344.bz2<br>
Jan 09 09:30:25 [7562] VirtKirchoff crmd: info: te_rsc_command: Initiating action 80: notify res_drbd_1:0_pre_notify_start_0 on VirtKirchoff (local)<br>Jan 09 09:30:25 VirtKirchoff lrmd: [7559]: info: rsc:res_drbd_1:0:35: notify<br>
Jan 09 09:30:25 [7562] VirtKirchoff crmd: info: te_rsc_command: Initiating action 91: notify res_drbd_2:0_pre_notify_start_0 on VirtKirchoff (local)<br>Jan 09 09:30:25 VirtKirchoff lrmd: [7559]: info: rsc:res_drbd_2:0:36: notify<br>
Jan 09 09:30:25 [7561] VirtKirchoff pengine: notice: process_pe_message: Transition 26: PEngine Input stored in: /var/lib/pengine/pe-input-3344.bz2<br>Jan 09 09:30:25 VirtKirchoff lrmd: [7559]: info: RA output: (res_drbd_1:0:notify:stdout) <br>
<br>Jan 09 09:30:25 [7562] VirtKirchoff crmd: info: process_lrm_event: LRM operation res_drbd_1:0_notify_0 (call=35, rc=0, cib-update=0, confirmed=true) ok<br>Jan 09 09:30:25 [7562] VirtKirchoff crmd: info: te_rsc_command: Initiating action 12: start res_drbd_1:1_start_0 on VirtTesla<br>
Jan 09 09:30:25 VirtKirchoff lrmd: [7559]: info: RA output: (res_drbd_2:0:notify:stdout) drbdsetup disk-options 1 --set-defaults --resync-rate=250M --c-max-rate=1024000K --c-min-rate=250M <br><br>Jan 09 09:30:25 [7562] VirtKirchoff crmd: info: process_lrm_event: LRM operation res_drbd_2:0_notify_0 (call=36, rc=0, cib-update=0, confirmed=true) ok<br>
Jan 09 09:30:25 [7562] VirtKirchoff crmd: info: te_rsc_command: Initiating action 44: start res_drbd_2:1_start_0 on VirtTesla<br>Jan 09 09:30:25 [7562] VirtKirchoff crmd: info: te_rsc_command: Initiating action 81: notify res_drbd_1:0_post_notify_start_0 on VirtKirchoff (local)<br>
Jan 09 09:30:25 VirtKirchoff lrmd: [7559]: info: rsc:res_drbd_1:0:37: notify<br>Jan 09 09:30:25 [7562] VirtKirchoff crmd: info: te_rsc_command: Initiating action 82: notify res_drbd_1:1_post_notify_start_0 on VirtTesla<br>
Jan 09 09:30:25 [7562] VirtKirchoff crmd: info: te_rsc_command: Initiating action 92: notify res_drbd_2:0_post_notify_start_0 on VirtKirchoff (local)<br>Jan 09 09:30:25 VirtKirchoff lrmd: [7559]: info: rsc:res_drbd_2:0:38: notify<br>
Jan 09 09:30:25 [7562] VirtKirchoff crmd: info: te_rsc_command: Initiating action 93: notify res_drbd_2:1_post_notify_start_0 on VirtTesla<br>Jan 09 09:30:25 VirtKirchoff lrmd: [7559]: info: RA output: (res_drbd_1:0:notify:stdout) <br>
<br>Jan 09 09:30:25 [7562] VirtKirchoff crmd: info: process_lrm_event: LRM operation res_drbd_1:0_notify_0 (call=37, rc=0, cib-update=0, confirmed=true) ok<br>Jan 09 09:30:25 VirtKirchoff lrmd: [7559]: info: RA output: (res_drbd_2:0:notify:stdout) <br>
<br>Jan 09 09:30:25 [7562] VirtKirchoff crmd: info: process_lrm_event: LRM operation res_drbd_2:0_notify_0 (call=38, rc=0, cib-update=0, confirmed=true) ok<br>Jan 09 09:30:26 [7562] VirtKirchoff crmd: info: te_rsc_command: Initiating action 13: monitor res_drbd_1:1_monitor_10000 on VirtTesla<br>
Jan 09 09:30:26 [7562] VirtKirchoff crmd: info: te_rsc_command: Initiating action 45: monitor res_drbd_2:1_monitor_10000 on VirtTesla<br>Jan 09 09:30:26 [7562] VirtKirchoff crmd: notice: run_graph: ==== Transition 26 (Complete=22, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-3344.bz2): Complete<br>
Jan 09 09:30:26 [7562] VirtKirchoff crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<br>Jan 09 09:30:56 [7562] VirtKirchoff crmd: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair, id=status-VirtTesla-master-res_drbd_2.1, name=master-res_drbd_2:1, value=5, magic=NA, cib=0.269.6) : Transient attribute: update<br>
Jan 09 09:30:56 [7562] VirtKirchoff crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]<br>Jan 09 09:30:56 [7561] VirtKirchoff pengine: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
Jan 09 09:30:56 [7561] VirtKirchoff pengine: warning: unpack_nodes: Blind faith: not fencing unseen nodes<br>Jan 09 09:30:56 [7561] VirtKirchoff pengine: notice: unpack_rsc_op: Operation monitor found resource res_drbd_1:0 active on VirtTesla<br>
Jan 09 09:30:56 [7561] VirtKirchoff pengine: notice: unpack_rsc_op: Operation monitor found resource res_drbd_2:0 active on VirtTesla<br>Jan 09 09:30:56 [7562] VirtKirchoff crmd: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<br>
Jan 09 09:30:56 [7562] VirtKirchoff crmd: info: do_te_invoke: Processing graph 27 (ref=pe_calc-dc-1357704056-102) derived from /var/lib/pengine/pe-input-3345.bz2<br>Jan 09 09:30:56 [7562] VirtKirchoff crmd: notice: run_graph: ==== Transition 27 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-3345.bz2): Complete<br>
Jan 09 09:30:56 [7562] VirtKirchoff crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<br>Jan 09 09:30:56 [7561] VirtKirchoff pengine: notice: process_pe_message: Transition 27: PEngine Input stored in: /var/lib/pengine/pe-input-3345.bz2<br>
Jan 09 09:31:56 [7562] VirtKirchoff crmd: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped (60000ms)<br>Jan 09 09:31:56 [7562] VirtKirchoff crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]<br>
Jan 09 09:31:56 [7562] VirtKirchoff crmd: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED<br>Jan 09 09:31:56 [7561] VirtKirchoff pengine: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
Jan 09 09:31:56 [7561] VirtKirchoff pengine: warning: unpack_nodes: Blind faith: not fencing unseen nodes<br>Jan 09 09:31:56 [7561] VirtKirchoff pengine: notice: unpack_rsc_op: Operation monitor found resource res_drbd_1:0 active on VirtTesla<br>
Jan 09 09:31:56 [7561] VirtKirchoff pengine: notice: unpack_rsc_op: Operation monitor found resource res_drbd_2:0 active on VirtTesla<br>Jan 09 09:31:56 [7561] VirtKirchoff pengine: notice: process_pe_message: Transition 28: PEngine Input stored in: /var/lib/pengine/pe-input-3345.bz2<br>
Jan 09 09:31:56 [7562] VirtKirchoff crmd: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<br>Jan 09 09:31:56 [7562] VirtKirchoff crmd: info: do_te_invoke: Processing graph 28 (ref=pe_calc-dc-1357704116-103) derived from /var/lib/pengine/pe-input-3345.bz2<br>
Jan 09 09:31:56 [7562] VirtKirchoff crmd: notice: run_graph: ==== Transition 28 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-3345.bz2): Complete<br>Jan 09 09:31:56 [7562] VirtKirchoff crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<br>
<br><br><br>Thanks & Regards,<br>sucheta<br><br><div></div>