[ClusterLabs] Odd clvmd error - clvmd: Unable to create DLM lockspace for CLVM: Address already in use

Digimer lists at alteeve.ca
Thu Sep 24 23:09:58 UTC 2015


I had a RHEL 6.7, cman + rgmanager cluster that I've built many times
before. Oddly, I just hit this error:

====
[root at node2 ~]# /etc/init.d/clvmd start
Starting clvmd: clvmd could not connect to cluster manager
Consult syslog for more information
====

syslog:
====
Sep 24 23:00:30 node2 kernel: dlm: Using SCTP for communications
Sep 24 23:00:30 node2 clvmd: Unable to create DLM lockspace for CLVM:
Address already in use
Sep 24 23:00:30 node2 kernel: dlm: Can't bind to port 21064 addr number 1
Sep 24 23:00:30 node2 kernel: dlm: cannot start dlm lowcomms -98
====

There are no iptables rules:

====
[root at node2 ~]# iptables-save
====

And there are no DLM lockspaces, either:

====
[root at node2 ~]# dlm_tool ls
[root at node2 ~]#
====

I tried withdrawing the node from the cluster entirely, the started cman
alone and tried to start clvmd, same issue.

Pinging between the two nodes seems OK:

====
[root at node1 ~]# uname -n
node1.ccrs.bcn
[root at node1 ~]# ping -c 2 node1.ccrs.bcn
PING node1.bcn (10.20.10.1) 56(84) bytes of data.
64 bytes from node1.bcn (10.20.10.1): icmp_seq=1 ttl=64 time=0.015 ms
64 bytes from node1.bcn (10.20.10.1): icmp_seq=2 ttl=64 time=0.017 ms

--- node1.bcn ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.015/0.016/0.017/0.001 ms
====
[root at node2 ~]# uname -n
node2.ccrs.bcn
[root at node2 ~]# ping -c 2 node1.ccrs.bcn
PING node1.bcn (10.20.10.1) 56(84) bytes of data.
64 bytes from node1.bcn (10.20.10.1): icmp_seq=1 ttl=64 time=0.079 ms
64 bytes from node1.bcn (10.20.10.1): icmp_seq=2 ttl=64 time=0.076 ms

--- node1.bcn ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.076/0.077/0.079/0.008 ms
====

I have RRP configured and pings work on the second network, too:

====
[root at node1 ~]# corosync-objctl |grep ring -A 5
totem.interface.ringnumber=0
totem.interface.bindnetaddr=10.20.10.1
totem.interface.mcastaddr=239.192.100.163
totem.interface.mcastport=5405
totem.interface.member.memberaddr=node1.ccrs.bcn
totem.interface.member.memberaddr=node2.ccrs.bcn
totem.interface.ringnumber=1
totem.interface.bindnetaddr=10.10.10.1
totem.interface.mcastaddr=239.192.100.164
totem.interface.mcastport=5405
totem.interface.member.memberaddr=node1.sn
totem.interface.member.memberaddr=node2.sn

[root at node1 ~]# ping -c 2 node2.sn
PING node2.sn (10.10.10.2) 56(84) bytes of data.
64 bytes from node2.sn (10.10.10.2): icmp_seq=1 ttl=64 time=0.111 ms
64 bytes from node2.sn (10.10.10.2): icmp_seq=2 ttl=64 time=0.120 ms

--- node2.sn ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.111/0.115/0.120/0.011 ms
====
[root at node2 ~]# ping -c 2 node1.sn
PING node1.sn (10.10.10.1) 56(84) bytes of data.
64 bytes from node1.sn (10.10.10.1): icmp_seq=1 ttl=64 time=0.079 ms
64 bytes from node1.sn (10.10.10.1): icmp_seq=2 ttl=64 time=0.171 ms

--- node1.sn ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.079/0.125/0.171/0.046 ms
====

Here is the cluster.conf:

====
[root at node1 ~]# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster name="ccrs" config_version="1">
	<cman expected_votes="1" two_node="1" transport="udpu" />
	<clusternodes>
		<clusternode name="node1.ccrs.bcn" nodeid="1">
			<altname name="node1.sn" />
			<fence>
				<method name="ipmi">
					<device name="ipmi_n01" ipaddr="10.250.199.15" login="admin"
passwd="secret" delay="15" action="reboot" />
				</method>
				<method name="pdu">
					<device name="pdu01" port="1" action="reboot" />
					<device name="pdu02" port="1" action="reboot" />
				</method>
			</fence>
		</clusternode>
		<clusternode name="node2.ccrs.bcn" nodeid="2">
			<altname name="node2.sn" />
			<fence>
				<method name="ipmi">
					<device name="ipmi_n02" ipaddr="10.250.199.17" login="admin"
passwd="secret" action="reboot" />
				</method>
				<method name="pdu">
					<device name="pdu01" port="2" action="reboot" />
					<device name="pdu02" port="2" action="reboot" />
				</method>
			</fence>
		</clusternode>
	</clusternodes>
	<fencedevices>
		<fencedevice name="ipmi_n01" agent="fence_ipmilan" />
		<fencedevice name="ipmi_n02" agent="fence_ipmilan" />
		<fencedevice name="pdu01" agent="fence_raritan_snmp" ipaddr="pdu1A" />
		<fencedevice name="pdu02" agent="fence_raritan_snmp" ipaddr="pdu1B" />
		<fencedevice name="pdu03" agent="fence_raritan_snmp" ipaddr="pdu2A" />
		<fencedevice name="pdu04" agent="fence_raritan_snmp" ipaddr="pdu2B" />
	</fencedevices>
	<fence_daemon post_join_delay="30" />
	<totem rrp_mode="passive" secauth="off"/>
	<rm log_level="5">
		<resources>
			<script file="/etc/init.d/drbd" name="drbd"/>
			<script file="/etc/init.d/wait-for-drbd" name="wait-for-drbd"/>
			<script file="/etc/init.d/clvmd" name="clvmd"/>
			<clusterfs device="/dev/node1_vg0/shared" force_unmount="1"
fstype="gfs2" mountpoint="/shared" name="sharedfs" />
			<script file="/etc/init.d/libvirtd" name="libvirtd"/>
		</resources>
		<failoverdomains>
			<failoverdomain name="only_n01" nofailback="1" ordered="0"
restricted="1">
				<failoverdomainnode name="node1.ccrs.bcn"/>
			</failoverdomain>
			<failoverdomain name="only_n02" nofailback="1" ordered="0"
restricted="1">
				<failoverdomainnode name="node2.ccrs.bcn"/>
			</failoverdomain>
			<failoverdomain name="primary_n01" nofailback="1" ordered="1"
restricted="1">
				<failoverdomainnode name="node1.ccrs.bcn" priority="1"/>
				<failoverdomainnode name="node2.ccrs.bcn" priority="2"/>
			</failoverdomain>
			<failoverdomain name="primary_n02" nofailback="1" ordered="1"
restricted="1">
				<failoverdomainnode name="node1.ccrs.bcn" priority="2"/>
				<failoverdomainnode name="node2.ccrs.bcn" priority="1"/>
			</failoverdomain>
		</failoverdomains>
		<service name="storage_n01" autostart="1" domain="only_n01"
exclusive="0" recovery="restart">
			<script ref="drbd">
				<script ref="wait-for-drbd">
					<script ref="clvmd">
						<clusterfs ref="sharedfs"/>
					</script>
				</script>
			</script>
		</service>
		<service name="storage_n02" autostart="1" domain="only_n02"
exclusive="0" recovery="restart">
			<script ref="drbd">
				<script ref="wait-for-drbd">
					<script ref="clvmd">
						<clusterfs ref="sharedfs"/>
					</script>
				</script>
			</script>
		</service>
		<service name="libvirtd_n01" autostart="1" domain="only_n01"
exclusive="0" recovery="restart">
			<script ref="libvirtd"/>
		</service>
		<service name="libvirtd_n02" autostart="1" domain="only_n02"
exclusive="0" recovery="restart">
			<script ref="libvirtd"/>
		</service>
	</rm>
</cluster>
====

Nothing special there at all.

While writing this email though, I saw this on the other node:

====
Sep 24 23:03:39 node1 corosync[4770]:   [TOTEM ] Retransmit List: 14e
Sep 24 23:03:39 node1 corosync[4770]:   [TOTEM ] Retransmit List: 14e
Sep 24 23:03:49 node1 corosync[4770]:   [TOTEM ] Retransmit List: 158
Sep 24 23:03:49 node1 corosync[4770]:   [TOTEM ] Retransmit List: 15a
Sep 24 23:03:49 node1 corosync[4770]:   [TOTEM ] Retransmit List: 15a
Sep 24 23:03:59 node1 corosync[4770]:   [TOTEM ] Retransmit List: 161
Sep 24 23:03:59 node1 corosync[4770]:   [TOTEM ] Retransmit List: 161
Sep 24 23:03:59 node1 corosync[4770]:   [TOTEM ] Retransmit List: 161
Sep 24 23:03:59 node1 corosync[4770]:   [TOTEM ] Retransmit List: 161 163
Sep 24 23:03:59 node1 corosync[4770]:   [TOTEM ] Retransmit List: 163
Sep 24 23:04:19 node1 corosync[4770]:   [TOTEM ] Retransmit List: 177
Sep 24 23:04:19 node1 corosync[4770]:   [TOTEM ] Retransmit List: 177
Sep 24 23:04:19 node1 corosync[4770]:   [TOTEM ] Retransmit List: 179
Sep 24 23:04:19 node1 corosync[4770]:   [TOTEM ] Retransmit List: 179
Sep 24 23:04:29 node1 corosync[4770]:   [TOTEM ] Retransmit List: 181
Sep 24 23:04:29 node1 corosync[4770]:   [TOTEM ] Retransmit List: 181
Sep 24 23:04:29 node1 corosync[4770]:   [TOTEM ] Retransmit List: 181
Sep 24 23:04:29 node1 corosync[4770]:   [TOTEM ] Retransmit List: 183
Sep 24 23:04:29 node1 corosync[4770]:   [TOTEM ] Retransmit List: 183
Sep 24 23:04:39 node1 corosync[4770]:   [TOTEM ] Retransmit List: 18c
Sep 24 23:04:39 node1 corosync[4770]:   [TOTEM ] Retransmit List: 18c
Sep 24 23:04:39 node1 corosync[4770]:   [TOTEM ] Retransmit List: 18c 18e
Sep 24 23:04:39 node1 corosync[4770]:   [TOTEM ] Retransmit List: 18e
Sep 24 23:07:20 node1 corosync[4770]:   [TOTEM ] Retransmit List: 23c
Sep 24 23:07:20 node1 corosync[4770]:   [TOTEM ] Retransmit List: 23c
Sep 24 23:07:20 node1 corosync[4770]:   [TOTEM ] Retransmit List: 23c
Sep 24 23:07:20 node1 corosync[4770]:   [TOTEM ] Retransmit List: 23e
Sep 24 23:07:20 node1 corosync[4770]:   [TOTEM ] Retransmit List: 23e
Sep 24 23:07:30 node1 corosync[4770]:   [TOTEM ] Retransmit List: 247
Sep 24 23:07:30 node1 corosync[4770]:   [TOTEM ] Retransmit List: 247
Sep 24 23:07:30 node1 corosync[4770]:   [TOTEM ] Retransmit List: 249
Sep 24 23:07:30 node1 corosync[4770]:   [TOTEM ] Retransmit List: 24b
Sep 24 23:07:30 node1 corosync[4770]:   [TOTEM ] Retransmit List: 24b
Sep 24 23:07:40 node1 corosync[4770]:   [TOTEM ] Retransmit List: 252
Sep 24 23:07:40 node1 corosync[4770]:   [TOTEM ] Retransmit List: 252
Sep 24 23:07:40 node1 corosync[4770]:   [TOTEM ] Retransmit List: 254
Sep 24 23:07:40 node1 corosync[4770]:   [TOTEM ] Retransmit List: 254
====

Certainly *looks* like a network problem, but I can't see what's
wrong... Any ideas?

Thanks!

-- 
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?




More information about the Users mailing list