[Pacemaker] ifstatus OCF RA
Dejan Muhamedagic
dejanmm at fastmail.fm
Tue Feb 22 10:03:13 EST 2011
Hi,
On Tue, Feb 22, 2011 at 03:54:20PM +0200, Vladislav Bogdanov wrote:
> Hi again,
> attached is a bit more polished revision.
A few general observations:
- the name (ifstatus) doesn't fit exactly (perhaps ifspeed?)
- since this is a stateless agent, you should use the existing
functions for that purpose:
http://www.linux-ha.org/doc/dev-guides/_pseudo_resources_literal_ha_pseudo_resource_literal.html
- almost all function names start with "ifstatus" which sometimes
makes it difficult to follow the code (e.g. ifstatus_iface_get_speed)
Seems like good work.
Cheers,
Dejan
> Best,
> Vladislav
>
> 22.02.2011 15:01, Vladislav Bogdanov wrote:
> > Hi Dejan,
> >
> > 22.02.2011 13:02, Dejan Muhamedagic wrote:
> >> Hi,
> >>
> >> Where can you get STP stuff from? How to interpret it? And then
> >
> > Please look at attached RA.
> >
> > I decided that today is a good time to finally brace myself to find 5
> > hours to write it, thanks Frederik ;) .
> > Tested, "works for me" (c) in that configuration I talked earlier - STP
> > bridge over 1x10Gbps eth + 2x1Gpbs bond.
> > (Hopefully) Supports any combination of bridges, bonds, vlans and
> > physical ethernet interfaces.
> > Tries to guess correct upstream bridge ports, can't test it more
> > thoroughly due to absence of more switch hardware (I currently have only
> > one c3570x stack per cluster).
> > May require some additional checks to be included.
> > Also it is linux-specific and requires bash because of my laziness and
> > fact that I mainly use Fedora which has nothing against bash yet.
> > Can be considered for inclusion in resource-agents (with common license,
> > GPLv2?).
> >
> >> how do you know it's something that won't change within next five
> >> minutes? Finally, every failover can incur downtime, is it worth
> >> the trouble because what you want is just more performance?
> >
> > This could be controlled by non-inf location score and f.e. time-based
> > stickiness.
> > Anyways, I'd better have 10 seconds lockup rather than 10Mb/s per-client
> > read for long time when second cluster node (32 disks in HW RAID10) is
> > able to easily give another 250-400Mb/s of aggregate throughput.
> >
> >> Perhaps you don't even need the extra performance at the time.
> >
> > This depends on what SLA I provide services with...
> >
> >> Other than that it sounds interesting :-)
> >
> > Then, please look at the implementation ;)
> >
> > Best,
> > Vladislav
> >
> >
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>
> #!/bin/bash
> #
> # OCF resource agent which monitors state of network interface and records it as a value in CIB
> # based on summ of speeds of its active underlying interfaces.
> #
> # Copyright (c) 2011 Vladislav Bogdanov <bubble at hoster-ok.com>
> # Partially based on 'ping' RA by Andrew Beekhof
> #
> # OCF instance parameters:
> # OCF_RESKEY_name: name of attribute to set in CIB
> # OCF_RESKEY_iface: network interface to monitor
> # OCF_RESKEY_bridge_ports: if not null and OCF_RESKEY_iface is a bridge, list of bridge ports to consider.
> # Default is all ports which have designated_bridge=root_id
> # OCF_RESKEY_weight_base: weight of each 10Mbps in interface speed (1Gbps = 100 * 10Mbps) in CIB score points
> #
> # Initialization:
>
> : ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/resource.d/heartbeat}
> . ${OCF_FUNCTIONS_DIR}/.ocf-shellfuncs
>
> # Defaults
> OCF_RESKEY_name_default="ifstatus"
> OCF_RESKEY_bridge_ports_default="detect"
> OCF_RESKEY_weight_base_default=10
> OCF_RESKEY_dampen_default=5
>
> : ${OCF_RESKEY_name=${OCF_RESKEY_name_default}}
> : ${OCF_RESKEY_bridge_ports=${OCF_RESKEY_bridge_ports_default}}
> : ${OCF_RESKEY_weight_base=${OCF_RESKEY_weight_base_default}}
> : ${OCF_RESKEY_dampen=${OCF_RESKEY_dampen_default}}
>
> meta_data() {
> cat <<END
> <?xml version="1.0"?>
> <!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd">
> <resource-agent name="ping">
> <version>1.0</version>
>
> <longdesc lang="en">
> Every time the monitor action is run, this resource agent records (in the CIB) speed of active network interfaces from a list.
> </longdesc>
> <shortdesc lang="en">Network interface status</shortdesc>
>
> <parameters>
>
> <parameter name="name" unique="1">
> <longdesc lang="en">
> The name of the attributes to set. This is the name to be used in the constraints.
> </longdesc>
> <shortdesc lang="en">Attribute name</shortdesc>
> <content type="string" default="${OCF_RESKEY_name_default}"/>
> </parameter>
>
> <parameter name="iface" unique="0" required="1">
> <longdesc lang="en">
> Network interface to monitor.
> </longdesc>
> <shortdesc lang="en">Network interface</shortdesc>
> <content type="string" default=""/>
> </parameter>
>
> <parameter name="bridge_ports" unique="0">
> <longdesc lang="en">
> If not null and OCF_RESKEY_iface is a bridge, list of bridge ports to consider.
> Default is all ports which have designated_bridge=root_id.
> </longdesc>
> <shortdesc lang="en">Bridge ports</shortdesc>
> <content type="string" default="${OCF_RESKEY_bridge_ports_default}"/>
> </parameter>
>
> <parameter name="weight_base" unique="0">
> <longdesc lang="en">
> Weight of each 10Mbps in interface speed (1Gbps = 100 * 10Mbps).
> With default value 1Gbps interface will be counted as 1000.
> </longdesc>
> <shortdesc lang="en">Weight of 10Mbps interface</shortdesc>
> <content type="integer" default="${OCF_RESKEY_weight_base_default}"/>
> </parameter>
>
> <parameter name="dampen" unique="0">
> <longdesc lang="en">
> The time to wait (dampening) further changes occur
> </longdesc>
> <shortdesc lang="en">Dampening interval</shortdesc>
> <content type="integer" default="${OCF_RESKEY_dampen_default}"/>
> </parameter>
>
> <parameter name="debug" unique="0">
> <longdesc lang="en">
> Enables to use default attrd_updater verbose logging on every call.
> </longdesc>
> <shortdesc lang="en">Verbose logging</shortdesc>
> <content type="string" default="false"/>
> </parameter>
>
> </parameters>
>
> <actions>
> <action name="start" timeout="30" />
> <action name="stop" timeout="30" />
> <action name="reload" timeout="30" />
> <action name="monitor" depth="0" timeout="30" interval="10"/>
> <action name="meta-data" timeout="5" />
> <action name="validate-all" timeout="30" />
> </actions>
> </resource-agent>
> END
> }
>
> ifstatus_usage() {
> cat <<END
> usage: $0 {start|stop|monitor|migrate_to|migrate_from|validate-all|meta-data}
>
> Expects to have a fully populated OCF RA-compliant environment set.
> END
> }
>
> ifstatus_start() {
> ifstatus_monitor
> if [ $? = $OCF_SUCCESS ]; then
> return $OCF_SUCCESS
> fi
> touch ${OCF_RESKEY_pidfile}
> ifstatus_update
> }
>
> ifstatus_stop() {
> rm -f ${OCF_RESKEY_pidfile}
> attrd_updater -D -n $OCF_RESKEY_name -d $OCF_RESKEY_dampen $attrd_options
> return $OCF_SUCCESS
> }
>
> ifstatus_monitor() {
> if [ -f ${OCF_RESKEY_pidfile} ]; then
> ifstatus_update
> return $OCF_SUCCESS
> fi
> return $OCF_NOT_RUNNING
> }
>
> ifstatus_validate() {
> # Is the state directory writable?
> state_dir=`dirname "$OCF_RESKEY_pidfile"`
> touch "$state_dir/$$"
> if [ $? != 0 ]; then
> ocf_log err "Invalid location for 'state': $state_dir is not writable"
> return $OCF_ERR_ARGS
> fi
> rm "$state_dir/$$"
>
> # Pidfile better be an absolute path
> case $OCF_RESKEY_pidfile in
> /*) ;;
> *) ocf_log warn "You should use an absolute path for pidfile not: $OCF_RESKEY_pidfile" ;;
> esac
>
> # Check the check interval
> if ocf_is_decimal "$OCF_RESKEY_CRM_meta_interval" && [ $OCF_RESKEY_CRM_meta_interval -gt 0 ]; then
> :
> else
> ocf_log err "Invalid check interval $OCF_RESKEY_interval. It should be positive integer!"
> exit $OCF_ERR_CONFIGURED
> fi
>
> # Check the intarfaces list
> if [ "x" = "x$OCF_RESKEY_iface" ]; then
> ocf_log err "Empty iface parameter. Please specify some network interface to check"
> exit $OCF_ERR_CONFIGURED
> fi
>
> return $OCF_SUCCESS
> }
>
> ifstatus_iface_get_speed() {
> local iface=$1
> local operstate
> local carrier
> local speed
>
> if [ ! -e "/sys/class/net/$iface" ] ; then
> echo 0
> elif ifstatus_iface_is_bridge $iface ; then
> ifstatus_bridge_get_speed $iface
> elif ifstatus_iface_is_bond $iface ; then
> ifstatus_bond_get_speed $iface
> elif ifstatus_iface_is_vlan $iface ; then
> ifstatus_iface_get_speed $( ifstatus_vlan_get_phy $iface )
> else
> read operstate < "/sys/class/net/$iface/operstate"
> read carrier < "/sys/class/net/$iface/carrier"
> if [ "$operstate" != "up" ] || [ "$carrier" != "1" ] ; then
> speed="0"
> else
> read speed < "/sys/class/net/$iface/speed"
> fi
> echo $speed
> fi
> }
>
> ifstatus_iface_is_vlan() {
> local iface=$1
> [ -e "/proc/net/vlan/$iface" ] && return 0 || return 1
> }
>
> ifstatus_iface_is_bridge() {
> local iface=$1
> [ -e "/sys/class/net/$iface/bridge" ] && return 0 || return 1
> }
>
> ifstatus_iface_is_bond() {
> local iface=$1
> [ -e "/sys/class/net/$iface/bonding" ] && return 0 || return 1
> }
>
> ifstatus_vlan_get_phy() {
> local iface=$1
> grep "^$iface " "/proc/net/vlan/config" | sed -r 's/.*\| +(.*)/\1/'
> }
>
> ifstatus_bridge_is_stp_enabled() {
> local iface=$1
> local stp
> read stp < "/sys/class/net/$iface/bridge/stp_state"
> [ "$stp" = "1" ] && return 0 || return 1
> }
>
> ifstatus_bridge_get_root_ports() {
> local bridge=$1
> local root_id
> local root_ports=""
> local bridge_id
>
> read root_id < "/sys/class/net/$bridge/bridge/root_id"
>
> for port in /sys/class/net/$bridge/brif/* ; do
> read bridge_id < "$port/designated_bridge"
> if [ "$bridge_id" = "$root_id" ] ; then
> root_ports="$root_ports ${port##*/}"
> fi
> done
> echo "${root_ports# }"
> }
>
> # From /inlude/linux/if_bridge.h:
> #define BR_STATE_DISABLED 0
> #define BR_STATE_LISTENING 1
> #define BR_STATE_LEARNING 2
> #define BR_STATE_FORWARDING 3
> #define BR_STATE_BLOCKING 4
>
> ifstatus_bridge_get_active_ports() {
> local bridge=$1
> shift 1
> local ports="$*"
> local active_ports=""
> local port_state
> local stp_state=ifstatus_bridge_is_stp_enabled $bridge
> local warn=0
>
> if [ -z "$ports" ] || [ "$ports" = "detect" ] ; then
> ports=$( ifstatus_bridge_get_root_ports $bridge )
> fi
>
> for port in $ports ; do
> if [ ! -e "/sys/class/net/$bridge/brif/$port" ] ; then
> ocf_log warning "Port $port doesn't belong to bridge $bridge"
> continue
> fi
> read port_state < "/sys/class/net/$bridge/brif/$port/state"
> if [ "$port_state" = "3" ] ; then
> if [ -n "$active_ports" ] && $stp_state ; then
> warn=1
> fi
> active_ports="$active_ports $port"
> fi
> done
> if [ $warn -eq 1 ] ; then
> ocf_log warning "More then one upstream port in bridge '$bridge' is in forwarding state while STP is enabled: $active_ports"
> fi
> echo "${active_ports# }"
> }
>
> ifstatus_bridge_get_speed() {
> local $iface=$1
>
> if ! ifstatus_iface_is_bridge $iface ; then
> echo 0
> return
> fi
>
> local ports=$( ifstatus_bridge_get_active_ports $iface ${OCF_RESKEY_bridge_ports} )
> for port in $ports ; do
> : $(( aggregate_speed += $( ifstatus_iface_get_speed $port ) ))
> done
> echo $aggregate_speed
> }
>
> ifstatus_bond_get_slaves() {
> local iface=$1
> local slaves
> read slaves < "/sys/class/net/$iface/bonding/slaves"
> echo $slaves
> }
>
> ifstatus_bond_get_active_iface() {
> local iface=$1
> local active
> read active < "/sys/class/net/$iface/bonding/active_slave"
> echo $active
> }
>
> ifstatus_bond_is_balancing() {
> local iface=$1
> read mode mode_index < "/sys/class/net/$iface/bonding/mode"
> case $mode in
> "balance-rr"|"balance-xor"|"802.3ad"|"balance-tlb"|"balance-alb")
> return 0
> ;;
> *)
> return 1
> ;;
> esac
> }
>
> ifstatus_bond_get_speed() {
> local iface=$1
> local aggregate_speed=0
>
> if ! ifstatus_iface_is_bond $iface ; then
> echo 0
> return
> fi
>
> local slaves=$( ifstatus_bond_get_slaves $iface )
> if ifstatus_bond_is_balancing $iface ; then
> for slave in $slaves ; do
> : $(( aggregate_speed += $( ifstatus_iface_get_speed $slave ) ))
> done
> # Bonding is unable to get speed*n
> : $(( aggregate_speed = aggregate_speed*8/10 ))
> else
> : $(( aggregate_speed = $( ifstatus_iface_get_speed $( ifstatus_bond_get_active_iface $iface ) ) ))
> fi
> echo $aggregate_speed
> }
>
> ifstatus_update() {
> local speed=$( ifstatus_iface_get_speed $OCF_RESKEY_iface)
>
> : $(( score = speed * $OCF_RESKEY_weight_base / 10 ))
> attrd_updater -n $OCF_RESKEY_name -v $score -d $OCF_RESKEY_dampen $attrd_options
> rc=$?
> case $rc in
> 0)
> ocf_is_true ${OCF_RESKEY_debug} && ocf_log debug "Updated $OCF_RESKEY_name = $score"
> ;;
> *)
> ocf_log warn "Could not update $OCF_RESKEY_name = $score: rc=$rc"
> ;;
> esac
> return $rc
> }
>
> if [ `uname` != "Linux" ] ; then
> ocf_log err "This RA works only on linux."
> exit $OCF_ERR_INSTALLED
> fi
>
> if ! ocf_is_true ${OCF_RESKEY_CRM_meta_globally_unique} ; then
> : ${OCF_RESKEY_pidfile:="$HA_VARRUN/ifstatus-${OCF_RESKEY_name}"}
> else
> : ${OCF_RESKEY_pidfile:="$HA_VARRUN/ifstatus-${OCF_RESOURCE_INSTANCE}"}
> fi
>
> attrd_options='-q'
> if ocf_is_true ${OCF_RESKEY_debug} ; then
> attrd_options=''
> fi
>
> case $__OCF_ACTION in
> meta-data)
> meta_data
> exit $OCF_SUCCESS
> ;;
> start)
> ifstatus_start
> ;;
> stop)
> ifstatus_stop
> ;;
> monitor)
> ifstatus_monitor
> ;;
> reload)
> ifstatus_start
> ;;
> validate-all)
> ifstatus_validate
> ;;
> usage|help)
> ifstatus_usage
> exit $OCF_SUCCESS
> ;;
> *)
> ifstatus_usage
> exit $OCF_ERR_UNIMPLEMENTED
> ;;
> esac
> exit $?
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
More information about the Pacemaker
mailing list