<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
pacemaker 1.1.12-11.12<br>
openais 1.1.4-5.24.5 <br>
corosync 1.4.7-0.23.5<br>
<br>
Its a two node active/passive cluster and we just upgraded the SLES
11 SP 3 to SLES 11 SP 4(nothing else) but when we try to start the
cluster service we get the following error:<br>
<br>
"Totem is unable to form a cluster because of an operating system or
network fault."<br>
<br>
Firewall is stopped and disabled on both the nodes. Both nodes can
ping/ssh/vnc each other.<br>
<br>
corosync.conf:<br>
aisexec {<br>
group: root<br>
user: root<br>
} <br>
service {<br>
use_mgmtd: yes<br>
use_logd: yes<br>
ver: 0<br>
name: pacemaker<br>
}<br>
totem {<br>
rrp_mode: none<br>
join: 60<br>
max_messages: 20<br>
vsftype: none<br>
token: 5000<br>
consensus: 6000<br>
<br>
interface {<br>
bindnetaddr: 192.168.150.0<br>
<br>
member {<br>
memberaddr: 192.168.150.12<br>
}<br>
member {<br>
memberaddr: 192.168.150.13<br>
}<br>
mcastport: 5405<br>
<br>
ringnumber: 0<br>
<br>
}<br>
secauth: off<br>
version: 2<br>
transport: udpu<br>
token_retransmits_before_loss_const: 10<br>
clear_node_high_bit: new<br>
}<br>
logging {<br>
to_logfile: no<br>
to_syslog: yes<br>
debug: off<br>
timestamp: off<br>
to_stderr: no<br>
fileline: off<br>
syslog_facility: daemon<br>
}<br>
amf {<br>
mode: disable<br>
}<br>
<br>
/var/log/messages:<br>
Apr 6 17:51:49 prd1 corosync[8672]: [MAIN ] Corosync Cluster
Engine ('1.4.7'): started and ready to provide service.<br>
Apr 6 17:51:49 prd1 corosync[8672]: [MAIN ] Corosync built-in
features: nss<br>
Apr 6 17:51:49 prd1 corosync[8672]: [MAIN ] Successfully
configured openais services to load<br>
Apr 6 17:51:49 prd1 corosync[8672]: [MAIN ] Successfully read
main configuration file '/etc/corosync/corosync.conf'.<br>
Apr 6 17:51:49 prd1 corosync[8672]: [TOTEM ] Initializing
transport (UDP/IP Unicast).<br>
Apr 6 17:51:49 prd1 corosync[8672]: [TOTEM ] Initializing
transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).<br>
Apr 6 17:51:49 prd1 corosync[8672]: [TOTEM ] The network interface
is down.<br>
Apr 6 17:51:49 prd1 corosync[8672]: [SERV ] Service engine
loaded: openais cluster membership service B.01.01<br>
Apr 6 17:51:49 prd1 corosync[8672]: [SERV ] Service engine
loaded: openais event service B.01.01<br>
Apr 6 17:51:49 prd1 corosync[8672]: [SERV ] Service engine
loaded: openais checkpoint service B.01.01<br>
Apr 6 17:51:49 prd1 corosync[8672]: [SERV ] Service engine
loaded: openais availability management framework B.01.01<br>
Apr 6 17:51:49 prd1 corosync[8672]: [SERV ] Service engine
loaded: openais message service B.03.01<br>
Apr 6 17:51:49 prd1 corosync[8672]: [SERV ] Service engine
loaded: openais distributed locking service B.03.01<br>
Apr 6 17:51:49 prd1 corosync[8672]: [SERV ] Service engine
loaded: openais timer service A.01.01<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info:
process_ais_conf: Reading configure<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info:
config_find_init: Local handle: 7685269064754659330 for logging<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info:
config_find_next: Processing additional logging options...<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: get_config_opt:
Found 'off' for option: debug<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: get_config_opt:
Found 'no' for option: to_logfile<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: get_config_opt:
Found 'yes' for option: to_syslog<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: get_config_opt:
Found 'daemon' for option: syslog_facility<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info:
config_find_init: Local handle: 8535092201842016259 for quorum<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info:
config_find_next: No additional configuration supplied for: quorum<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: get_config_opt:
No default for option: provider<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info:
config_find_init: Local handle: 8054506479773810692 for service<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info:
config_find_next: Processing additional service options...<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info:
config_find_next: Processing additional service options...<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info:
config_find_next: Processing additional service options...<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info:
config_find_next: Processing additional service options...<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info:
config_find_next: Processing additional service options...<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info:
config_find_next: Processing additional service options...<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info:
config_find_next: Processing additional service options...<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info:
config_find_next: Processing additional service options...<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: get_config_opt:
Found '0' for option: ver<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: get_config_opt:
Defaulting to 'pcmk' for option: clustername<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: get_config_opt:
Found 'yes' for option: use_logd<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: get_config_opt:
Found 'yes' for option: use_mgmtd<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: pcmk_startup:
CRM: Initialized<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] Logging: Initialized
pcmk_startup<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: pcmk_startup:
Maximum core file size is: 18446744073709551615<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: pcmk_startup:
Service: 9<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: pcmk_startup:
Local hostname: prd1<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info:
pcmk_update_nodeid: Local node id: 2130706433<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: update_member:
Creating entry for node 2130706433 born on 0<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: update_member:
0x64c9c0 Node 2130706433 now known as prd1 (was: (null))<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: update_member:
Node prd1 now has 1 quorum votes (was 0)<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: update_member:
Node 2130706433/prd1 is now: member<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: spawn_child:
Using uid=90 and group=90 for process cib<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: spawn_child:
Forked child 8677 for process cib<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: spawn_child:
Forked child 8678 for process stonith-ng<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: spawn_child:
Forked child 8679 for process lrmd<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: spawn_child:
Using uid=90 and group=90 for process attrd<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: spawn_child:
Forked child 8680 for process attrd<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: spawn_child:
Using uid=90 and group=90 for process pengine<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: spawn_child:
Forked child 8681 for process pengine<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: spawn_child:
Using uid=90 and group=90 for process crmd<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: spawn_child:
Forked child 8682 for process crmd<br>
Apr 6 17:51:49 prd1 corosync[8672]: [pcmk ] info: spawn_child:
Forked child 8683 for process mgmtd<br>
Apr 6 17:51:49 prd1 corosync[8672]: [SERV ] Service engine
loaded: Pacemaker Cluster Manager 1.1.12<br>
Apr 6 17:51:49 prd1 corosync[8672]: [SERV ] Service engine
loaded: corosync extended virtual synchrony service<br>
Apr 6 17:51:49 prd1 corosync[8672]: [SERV ] Service engine
loaded: corosync configuration service<br>
Apr 6 17:51:49 prd1 corosync[8672]: [SERV ] Service engine
loaded: corosync cluster closed process group service v1.01<br>
Apr 6 17:51:49 prd1 corosync[8672]: [SERV ] Service engine
loaded: corosync cluster config database access v1.01<br>
Apr 6 17:51:49 prd1 corosync[8672]: [SERV ] Service engine
loaded: corosync profile loading service<br>
Apr 6 17:51:49 prd1 corosync[8672]: [SERV ] Service engine
loaded: corosync cluster quorum service v0.1<br>
Apr 6 17:51:49 prd1 corosync[8672]: [MAIN ] Compatibility mode
set to whitetank. Using V1 and V2 of the synchronization engine.<br>
Apr 6 17:51:49 prd1 corosync[8672]: [TOTEM ] adding new UDPU
member {192.168.150.12}<br>
Apr 6 17:51:49 prd1 corosync[8672]: [TOTEM ] adding new UDPU
member {192.168.150.13}<br>
Apr 6 17:51:50 prd1 lrmd[8679]: notice: crm_add_logfile:
Additional logging available in /var/log/pacemaker.log<br>
Apr 6 17:51:50 prd1 mgmtd: [8683]: info: Pacemaker-mgmt Git
Version: 969d213<br>
Apr 6 17:51:50 prd1 mgmtd: [8683]: WARN: Core dumps could be lost
if multiple dumps occur.<br>
Apr 6 17:51:50 prd1 mgmtd: [8683]: WARN: Consider setting
non-default value in /proc/sys/kernel/core_pattern (or equivalent)
for maximum supportability<br>
Apr 6 17:51:50 prd1 mgmtd: [8683]: WARN: Consider setting
/proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum
supportability<br>
Apr 6 17:51:50 prd1 attrd[8680]: notice: crm_add_logfile:
Additional logging available in /var/log/pacemaker.log<br>
Apr 6 17:51:50 prd1 pengine[8681]: notice: crm_add_logfile:
Additional logging available in /var/log/pacemaker.log<br>
Apr 6 17:51:50 prd1 attrd[8680]: notice: crm_cluster_connect:
Connecting to cluster infrastructure: classic openais (with plugin)<br>
Apr 6 17:51:50 prd1 cib[8677]: notice: crm_add_logfile:
Additional logging available in /var/log/pacemaker.log<br>
Apr 6 17:51:50 prd1 crmd[8682]: notice: crm_add_logfile:
Additional logging available in /var/log/pacemaker.log<br>
Apr 6 17:51:50 prd1 attrd[8680]: notice: get_node_name:
Defaulting to uname -n for the local classic openais (with plugin)
node name<br>
Apr 6 17:51:50 prd1 corosync[8672]: [pcmk ] info: pcmk_ipc:
Recorded connection 0x7f944c04acf0 for attrd/8680<br>
Apr 6 17:51:50 prd1 crmd[8682]: notice: main: CRM Git Version:
f47ea56<br>
Apr 6 17:51:50 prd1 attrd[8680]: notice: get_node_name:
Defaulting to uname -n for the local classic openais (with plugin)
node name<br>
Apr 6 17:51:50 prd1 attrd[8680]: notice: main: Starting
mainloop...<br>
Apr 6 17:51:50 prd1 stonith-ng[8678]: notice: crm_add_logfile:
Additional logging available in /var/log/pacemaker.log<br>
Apr 6 17:51:50 prd1 stonith-ng[8678]: notice:
crm_cluster_connect: Connecting to cluster infrastructure: classic
openais (with plugin)<br>
Apr 6 17:51:50 prd1 stonith-ng[8678]: notice: get_node_name:
Defaulting to uname -n for the local classic openais (with plugin)
node name<br>
Apr 6 17:51:50 prd1 corosync[8672]: [pcmk ] info: pcmk_ipc:
Recorded connection 0x658190 for stonith-ng/8678<br>
Apr 6 17:51:50 prd1 corosync[8672]: [pcmk ] info: update_member:
Node prd1 now has process list: 00000000000000000000000000151312
(1381138)<br>
Apr 6 17:51:50 prd1 corosync[8672]: [pcmk ] info: pcmk_ipc:
Sending membership update 0 to stonith-ng<br>
Apr 6 17:51:50 prd1 stonith-ng[8678]: notice: get_node_name:
Defaulting to uname -n for the local classic openais (with plugin)
node name<br>
Apr 6 17:51:50 prd1 cib[8677]: notice: crm_cluster_connect:
Connecting to cluster infrastructure: classic openais (with plugin)<br>
Apr 6 17:51:50 prd1 cib[8677]: notice: get_node_name: Defaulting
to uname -n for the local classic openais (with plugin) node name<br>
Apr 6 17:51:50 prd1 corosync[8672]: [pcmk ] info: pcmk_ipc:
Recorded connection 0x65d450 for cib/8677<br>
Apr 6 17:51:50 prd1 corosync[8672]: [pcmk ] info: pcmk_ipc:
Sending membership update 0 to cib<br>
Apr 6 17:51:50 prd1 cib[8677]: notice: get_node_name: Defaulting
to uname -n for the local classic openais (with plugin) node name<br>
Apr 6 17:51:50 prd1 cib[8677]: notice: crm_update_peer_state:
cib_peer_update_callback: Node prd1[2130706433] - state is now lost
(was (null))<br>
Apr 6 17:51:50 prd1 cib[8677]: notice: crm_update_peer_state:
plugin_handle_membership: Node prd1[2130706433] - state is now
member (was lost)<br>
Apr 6 17:51:50 prd1 mgmtd: [8683]: info: Started.<br>
Apr 6 17:51:51 prd1 crmd[8682]: notice: crm_cluster_connect:
Connecting to cluster infrastructure: classic openais (with plugin)<br>
Apr 6 17:51:51 prd1 crmd[8682]: notice: get_node_name: Defaulting
to uname -n for the local classic openais (with plugin) node name<br>
Apr 6 17:51:51 prd1 corosync[8672]: [pcmk ] info: pcmk_ipc:
Recorded connection 0x661b00 for crmd/8682<br>
Apr 6 17:51:51 prd1 corosync[8672]: [pcmk ] info: pcmk_ipc:
Sending membership update 0 to crmd<br>
Apr 6 17:51:51 prd1 crmd[8682]: notice: get_node_name: Defaulting
to uname -n for the local classic openais (with plugin) node name<br>
Apr 6 17:51:51 prd1 stonith-ng[8678]: notice: setup_cib: Watching
for stonith topology changes<br>
Apr 6 17:51:51 prd1 stonith-ng[8678]: notice:
crm_update_peer_state: st_peer_update_callback: Node
prd1[2130706433] - state is now lost (was (null))<br>
Apr 6 17:51:51 prd1 stonith-ng[8678]: notice:
crm_update_peer_state: plugin_handle_membership: Node
prd1[2130706433] - state is now member (was lost)<br>
Apr 6 17:51:51 prd1 crmd[8682]: notice: crm_update_peer_state:
plugin_handle_membership: Node prd1[2130706433] - state is now
member (was (null))<br>
Apr 6 17:51:51 prd1 crmd[8682]: notice: do_started: The local CRM
is operational<br>
Apr 6 17:51:51 prd1 crmd[8682]: notice: do_state_transition:
State transition S_STARTING -> S_PENDING [ input=I_PENDING
cause=C_FSA_INTERNAL origin=do_started ]<br>
Apr 6 17:51:51 prd1 stonith-ng[8678]: notice: unpack_config: On
loss of CCM Quorum: Ignore<br>
Apr 6 17:52:12 prd1 crmd[8682]: warning: do_log: FSA: Input
I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING<br>
Apr 6 17:52:35 prd1 corosync[8672]: [MAIN ] Totem is unable to
form a cluster because of an operating system or network fault. The
most common cause of this message is that the local firewall is
configured improperly.<br>
Apr 6 17:52:36 prd1 corosync[8672]: [MAIN ] Totem is unable to
form a cluster because of an operating system or network fault. The
most common cause of this message is that the local firewall is
configured improperly.<br>
<br>
<br>
<div class="moz-signature">-- <br>
<font size="1">Regards,<br>
<br>
</font>
<font size="2">Muhammad Sharfuddin<br>
</font>
<font size="1">
<a href="http://www.nds.com.pk"></a></font><br>
<br>
</div>
</body>
</html>