[ClusterLabs] [Linux-HA] fence_ec2 agent
東一彦
higashi.kazuhiko at lab.ntt.co.jp
Tue Mar 24 11:48:19 UTC 2015
Hi Markus,
Thank you for the comment.
> Would it be possible, to implement this idea as an additional configuration method to the fence_ec2 agent?
I think that your idea is good.
So, I tries to implement it.
I'm going to change the fence_ec2(ec2) the following points.
- the "tag" and the "port" options will be "not" required.
- if the "port" option is not set, the 2nd argument of ec2 will use as the "port".
- the 2nd argument of ec2 is "node to fence".
- the "stat" and "status" action will be same the "monitor" action.
(for do not use the "port" parameter in "stat" action.)
By the above modifications, If it is described uname in the Name tag,
the setting of the "tag" and "port" parameters are no longer necessary.
----
primitive prmStonith1-2 stonith:external/ec2 \
params \
pcmk_off_timeout="120s" \
op start interval="0s" timeout="60s" \
op monitor interval="3600s" timeout="60s" \
op stop interval="0s" timeout="60s"
----
You can use "tag" parameter like your "Clustername" tag.
If cluster nodes(instances) have "Cluster1" tag, and uname is described in that tag,
it works just like you to expect.
----
primitive prmStonith1-2 stonith:external/ec2 \
params \
pcmk_off_timeout="120s" \
tag="Cluster1" \
op start interval="0s" timeout="60s" \
op monitor interval="3600s" timeout="60s" \
op stop interval="0s" timeout="60s"
----
The 1st instance have "Cluster1=node01" tag-key.
The 2nd instance have "Cluster1=node02" tag-key.
The 3rd instance have "Cluster1=node03" tag-key.
...
The prmStonith1-2 can fence node01 , node02 and node03.
If you like above, I will implement that.
Regards,
Kazuhiko Higashi
On 2015/03/19 1:03, Markus Guertler wrote:
> Hi Kazuhiko, Dejan,
>
> the new resource agent is very good. Since there were a couple of days between my original question and the answer from
> Kazuhiko, I also have written a stonith agent proof of concept (attached to this email) in order to continue in my
> project. However, I think that your fence_ec2 agent is better from a development perspective and it doesn't make sense
> to have two different agents for the same use case.
>
> Nevertheless, I've implemented an idea, that is very useful in EC2 environments with clusters that have more than two
> nodes: All EC2 instances that belong to a cluster get a unique cluster name via an EC2 instance tag. The agent uses this
> tag to determine all cluster nodes that belong to his own cluster
>
> --- SNIP ---
> gethosts)
> # List of hostnames of this cluster
> init_agent
> ec2-describe-instances --filter "tag-key=Clustername" --filter "tag-value=$clustername" | grep "^TAG" |grep
> "Hostname" | awk '{ print $5 }' | sort -u
> --- SNIP ---
>
> The advantage of this method is, that you just need one configuration snippet for all nodes. This allows to dynamically
> add or remove EC2 instances / cluster nodes to/from a cluster without having to need to touch the cluster configuration.
> Dynamically adding or removing nodes (compute instances) is a very common scenario in a cloud.
>
> Would it be possible, to implement this idea as an additional configuration method to the fence_ec2 agent?
>
> Cheers,
> Markus
>
>>>> 東一彦 <higashi.kazuhiko at lab.ntt.co.jp> 3/12/2015 10:44 AM >>>
> Hi Dejan
>
> Thank you for add it and the fix some issues !
>
>
> > I was not able to test it, hope it works :)
> I confirmed that it works fine in my AWS environment :)
>
>
> Regards,
> Kazuhiko Higashi
>
> On 2015/03/11 21:27, Dejan Muhamedagic wrote:
>> Hi Kazuhiko-san,
>>
>> On Wed, Mar 11, 2015 at 02:36:43PM +0900, 東一彦 wrote:
>>> Hi, Dejan
>>>
>>> Thank you for the comment.
>>>
>>> I'd like to contribute it as glue stonith agents.
>>>
>>> So, I rename it to just "ec2".
>>>
>>> Would you please add it to glue repository (http://hg.linux-ha.org/glue/) ?
>>
>> I just added your stonith agent. There were this change in the
>> initial changeset:
>>
>> - replaced '-' which is not allowed in identifiers with '_' in
>> function getinfo_xml().
>>
>> There were other smaller changes. You can find them in the
>> repository.
>>
>> I was not able to test it, hope it works :)
>>
>> Many thanks for the contribution.
>>
>> Cheers,
>>
>> Dejan
>>
>>> Regards,
>>> Kazuhiko Higashi
>>>
>>> On 2015/03/06 2:38, Dejan Muhamedagic wrote:
>>>> Hi,
>>>>
>>>> On Tue, Mar 03, 2015 at 05:13:49PM +0900, 東一彦 wrote:
>>>>> Dear Markus,
>>>>>
>>>>> I was also thinking the same thing.
>>>>> So, Already I've created a new one.
>>>>
>>>> Perhaps you'd like to then contribute it upstream? Either to
>>>> glue stonith agents or RHT fencing agents. It appears that the
>>>> agent is using the stonith interface, but the name reflects the
>>>> fencing agents naming scheme.
>>>>
>>>> Cheers,
>>>>
>>>> Dejan
>>>>
>>>>> [ChangeSet]
>>>>> - An API to be used was changed from "Amazon EC2 CLI" to "AWS CLI".
>>>>> -- "AWS CLI" is based Python. So, CPU load might be reduced.
>>>>>
>>>>> - The "--private-key" and "--cert" options are deprecated in AWS CLI.
>>>>> So, I add a new option "--profile". Use a specific profile from that credential file.
>>>>> default is ""
>>>>>
>>>>>
>>>>> [How to use]
>>>>> - Plaese install the "AWS CLI".
>>>>> http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
>>>>>
>>>>> - Please copy the fence_ec2 in /usr/lib64/stonith/plugins/external/.
>>>>> And , Please set the permissions to 755.
>>>>>
>>>>> - Please set crm settings as in this example.
>>>>> - The instance that have been set as "node01" in the "Name" tag are fence.
>>>>> ------
>>>>> primitive prmStonith1-2 stonith:external/fence_ec2 \
>>>>> params \
>>>>> pcmk_off_timeout="300s" \
>>>>> port="node01" \
>>>>> tag="Name"
> \
>>>>> op start interval="0s" timeout="60s" \
>>>>> op monitor interval="3600s" timeout="60s" \
>>>>> op stop interval="0s" timeout="60s"
>>>>> ------
>>>>>
>>>>>
>>>>> Regards,
>>>>> Kazuhiko Higashi
>>>>>
>>>>> On 2015/02/25 7:22, Markus Guertler wrote:
>>>>>> Dear list,
>>>>>> I was just trying to configure the fence_ec2 stonith agent from 2012, written by Andrew Beekhof. It looks like,
> that this one not working anymore with newer stonith / cluster versions. Is there any other EC2 agent, that is still
> maintained?
>>>>>>
>>>>>> If not, I'll write one myself. However, I'd like to check all options first.
>>>>>>
>>>>>> Cheers,
>>>>>> Markus
>>>>>>
>>>>>> _______________________________________________
>>>>>> Linux-HA mailing list
>>>>>> Linux-HA at lists.linux-ha.org
>>>>>> http://lists.linux-ha.org/mailman/listinfo/linux-ha
>>>>>> See also: http://linux-ha.org/ReportingProblems
>>>>>>
>>>>>
>>>>>
>>
>>> #!/bin/bash
>>>
>>> description="
>>> fence_ec2 is an I/O Fencing agent which can be used with Amazon EC2 instances.
>>>
>>> API functions used by this agent:
>>> - aws ec2 describe-tags
>>> - aws ec2 describe-instances
>>> - aws ec2 stop-instances
>>> - aws ec2 start-instances
>>> - aws ec2 reboot-instances
>>>
>>> If the uname used by the cluster node is any of:
>>> - Public DNS name (or part there of),
>>> - Private DNS name (or part there of),
>>> - Instance ID (eg. i-4f15a839)
>>> - Contents of tag associated with the instance
>>> then the agent should be able to automatically discover the instances it can control.
>>>
>>> If the tag containing the uname is not [Name], then it will need to be specified using the [tag] option.
>>> "
>>>
>>> #
>>> # Copyright (c) 2011-2013 Andrew Beekhof
>>> # Copyright (c) 2014 NIPPON TELEGRAPH AND TELEPHONE CORPORATION
>>> # All Rights Reserved.
>>> #
>>> # This program is free software; you can redistribute it and/or modify
>>> # it under the terms of version 2 of the GNU General Public License as
>>> # published by the Free Software Foundation.
>>> #
>>> # This program is distributed in the hope that it would be useful, but
>>> # WITHOUT ANY WARRANTY; without even the implied warranty of
>>> # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>>> #
>>> # Further, this software is distributed without any warranty that it is
>>> # free of the rightful claim of any third person regarding infringement
>>> # or the like. Any license provided herein, whether implied or
>>> # otherwise, applies only to this software file. Patent licenses, if
>>> # any, provided herein do not apply to combinations of this program with
>>> # other software, or any other product whatsoever.
>>> #
>>> # You should have received a copy of the GNU General Public License
>>> # along with this program; if not, write the Free Software Foundation,
>>> # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA.
>>> #
>>> #######################################################################
>>>
>>> quiet=0
>>> port_default=""
>>>
>>> instance_not_found=0
>>> unknown_are_stopped=0
>>>
>>> action_default="reset" # Default fence action
>>> ec2_tag_default="Name" # EC2 Tag containing the instance's uname
>>>
>>> sleep_time="1"
>>>
>>> ec2_tag=${tag}
>>>
>>> : ${ec2_tag=${ec2_tag_default}}
>>> : ${port=${port_default}}
>>>
>>> function usage()
>>> {
>>> cat <<EOF
>>> `basename $0` - A fencing agent for Amazon EC2 instances
>>>
>>> $description
>>>
>>> Usage: `basename $0` -o|--action [-n|--port] [options]
>>> Options:
>>> -h, --help This text
>>> -V, --version Version information
>>> -q, --quiet Reduced output mode
>>>
>>> Commands:
>>> -o, --action Action to perform: on|off|reboot|status|monitor
>>> -n, --port The name of a machine/instance to control/check
>>>
>>> Additional Options:
>>> -p, --profile Use a specific profile from your credential file.
>>> -t, --tag Name of the tag containing the instance's uname
>>>
>>> Dangerous options:
>>> -U, --unknown-are-stopped Assume any unknown instance is safely stopped
>>>
>>> EOF
>>>
> exit 0;
>>> }
>>>
>>> function getinfo-xml()
>>> {
>>> cat <<EOF
>>> <parameters>
>>> <parameter name="port" unique="1" required="1">
>>> <content type="string" />
>>> <shortdesc lang="en">The name/id/tag of a instance to control/check</shortdesc>
>>> </parameter>
>>> <parameter name="profile" unique="0" required="0">
>>> <content type="string" default="default" />
>>> <shortdesc lang="en">Use a specific profile from your credential file.</shortdesc>
>>> </parameter>
>>> <parameter name="tag" unique="0" required="1">
>>> <content type="string" default="Name" />
>>> <shortdesc lang="en">Name of the tag containing the instances uname</shortdesc>
>>> </parameter>
>>> <parameter name="unknown_are_stopped" unique="0" required="0">
>>> <content type="string" default="false" />
>>> <shortdesc lang="en">DANGER: Assume any unknown instance is safely stopped</shortdesc>
>>> </parameter>
>>> </parameters>
>>> EOF
>>> exit 0;
>>> }
>>>
>>> function metadata()
>>> {
>>> cat <<EOF
>>> <?xml version="1.0" ?>
>>> <resource-agent name="fence_ec2" shortdesc="Fencing agent for Amazon EC2 instances" >
>>> <longdesc>
>>> $description
>>> </longdesc>
>>> <parameters>
>>> <parameter name="action" unique="0" required="1">
>>> <getopt mixed="-o, --action=[action]" />
>>> <content type="string" default="reboot" />
>>> <shortdesc lang="en">Fencing Action</shortdesc>
>>> </parameter>
>>> <parameter name="port" unique="1" required="1">
>>> <getopt mixed="-n, --port=[port]" />
>>> <content type="string" />
>>> <shortdesc lang="en">The name/id/tag of a instance to control/check</shortdesc>
>>> </parameter>
>>> <parameter name="profile" unique="0" required="0">
>>> <getopt mixed="-p, --profile=[profile]" />
>>> <content type="string" default="default" />
>>> <shortdesc lang="en">Use a specific profile from your credential file.</shortdesc>
>>> </parameter>
>>> <parameter name="tag" unique="0" required="1">
>>> <getopt mixed="-t, --tag=[tag]" />
>>> <content type="string" default="Name" />
>>> <shortdesc lang="en">Name of the tag containing the instances uname</shortdesc>
>>> </parameter>
>>> <parameter name="unknown-are-stopped" unique="0" required="0">
>>> <getopt mixed="-U, --unknown-are-stopped" />
>>> <content type="string" default="false" />
>>> <shortdesc lang="en">DANGER: Assume any unknown instance is safely stopped</shortdesc>
>>> </parameter>
>>> </parameters>
>>> <actions>
>>> <action name="on" />
>>> <action name="off" />
>>> <action name="reboot" />
>>> <action name="status" />
>>> <action name="list" />
>>> <action name="monitor" />
>>> <action name="metadata" />
>>> </actions>
>>> </resource-agent>
>>> EOF
>>> exit 0;
>>> }
>>>
>>> function instance_for_port()
>>> {
>>> local port=$1
>>> local instance=""
>>>
>>> # Look for port name -n in the INSTANCE data
>>> instance=`aws ec2 describe-instances $options | grep "^INSTANCES[[:space:]].*[[:space:]]$port[[:space:]]" | awk
> '{print $8}'`
>>> if [ -z $instance ]; then
>>> # Look for port name -n in the Name TAG
>>> instance=`aws ec2 describe-tags $options | grep
> "^TAGS[[:space:]]$ec2_tag[[:space:]].*[[:space:]]instance[[:space:]]$port$" | awk '{print $3}'`
>>> fi
>>>
>>> if [ -z $instance ]; then
>>> instance_not_found=1
>>> instance=$port
>>> fi
>>>
>>> echo $instance
>>> }
>>>
>>> function instance_on()
>>> {
>>> aws ec2 start-instances $options --instance-ids $instance
>>> }
>>>
>>> function instance_off()
>>> {
>>> if [ $unknown_are_stopped = 1 -a $instance_not_found ]; then
>>> : nothing to do
>>> ha_log.sh info "Assuming unknown instance $instance is already off"
>>> else
>>> aws ec2 stop-instances $options --instance-ids $instance --force
>>> fi
>>> }
>>>
>>> function instance_status()
>>> {
>>> local instance=$1
>>> local status="unknown"
>>> local rc=1
>>>
>>> # List of instances and their current status
>>> if [ $unknown_are_stopped = 1 -a $instance_not_found ]; then
>>> ha_log.sh info "$instance stopped (unknown)"
>>> else
>>> status=`aws ec2 describe-instance
> s $options --instance-ids $instance | awk '{
>>> if (/^STATE¥t/) { printf "%s", $3 }
>>> }'`
>>> rc=$?
>>> fi
>>> ha_log.sh info "status check for $instance is $status"
>>> echo $status
>>> return $rc
>>> }
>>>
>>>
>>> TEMP=`getopt -o qVho:e:p:n:t:U --long version,help,action:,port:,option:,profile:,tag:,quiet,unknown-are-stopped ¥
>>> -n 'fence_ec2' -- "$@"`
>>>
>>> if [ $? != 0 ];then
>>> usage
>>> exit 1
>>> fi
>>>
>>> # Note the quotes around `$TEMP': they are essential!
>>> eval set -- "$TEMP"
>>>
>>> if [ -z $1 ]; then
>>> # If there are no command line args, look for options from stdin
>>> while read line; do
>>> case $line in
>>> option=*|action=*) action=`echo $line | sed s/.*=//`;;
>>> port=*) port=`echo $line | sed s/.*=//`;;
>>> profile=*) ec2_profile=`echo $line | sed s/.*=//`;;
>>> tag=*) ec2_tag=`echo $line | sed s/.*=//`;;
>>> quiet*) quiet=1;;
>>> unknown-are-stopped*) unknown_are_stopped=1;;
>>> --);;
>>> *) ha_log.sh err "Invalid command: $line";;
>>> esac
>>> done
>>> fi
>>>
>>> while true ; do
>>> case "$1" in
>>> -o|--action|--option) action=$2; shift; shift;;
>>> -n|--port) port=$2; shift; shift;;
>>> -p|--profile) ec2_profile=$2; shift; shift;;
>>> -t|--tag) ec2_tag=$2; shift; shift;;
>>> -U|--unknown-are-stopped) unknown_are_stopped=1; shift;;
>>> -q|--quiet) quiet=1; shift;;
>>> -V|--version) echo "1.0.0"; exit 0;;
>>> --help|-h)
>>> usage;
>>> exit 0;;
>>> --) shift ; break ;;
>>> *) ha_log.sh err "Unknown option: $1. See --help for details."; exit 1;;
>>> esac
>>> done
>>>
>>> [ -n "$1" ] && action=$1
>>>
>>> if [ -z "$ec2_profile"]; then
>>> options="--output text --profile default"
>>> else
>>> options="--output text --profile $ec2_profile "
>>> fi
>>>
>>> action=`echo $action | tr 'A-Z' 'a-z'`
>>>
>>> case $action in
>>> metadata)
>>> metadata
>>> ;;
>>> getinfo-xml)
>>> getinfo-xml
>>> ;;
>>> getconfignames)
>>> for i in profile port tag
>>> do
>>> echo $i
>>> done
>>> exit 0
>>> ;;
>>> getinfo-devid)
>>> echo "EC2 STONITH device"
>>> exit 0
>>> ;;
>>> getinfo-devname)
>>> echo "EC2 STONITH external device"
>>> exit 0
>>> ;;
>>> getinfo-devdescr)
>>> echo "fence_ec2 is an I/O Fencing agent which can be used with Amazon EC2 instances."
>>> exit 0
>>> ;;
>>> getinfo-devurl)
>>> echo ""
>>> exit 0
>>> ;;
>>> esac
>>>
>>> # get my instance id
>>> myinstance=`curl http://169.254.169.254/latest/meta-data/instance-id`
>>>
>>> # check my status.
>>> # When the EC2 instance be stopped by the "aws ec2 stop-instances" , the stop processing of the OS is executed.
>>> # While the OS stop processing, Pacemaker can execute the STONITH processing.
>>> # So, If my status is not "running", it determined that I was already fenced. And to prevent fencing each other
>>> # in split-brain, I don't fence other node.
>>> if [ -z "$myinstance" ]; then
>>> ha_log.sh err "Failed to get My Instance ID. so can not check my status."
>>> exit 1
>>> fi
>>> mystatus=`instance_status $myinstance`
>>> if [ "$mystatus" != "running" ]; then #do not fence
>>> ha_log.sh warn "I was already fenced (My instance status=$mystatus). I don't fence other node."
>>> exit 1
>>> fi
>>>
>>> # get target's instance id
>>> instance=""
>>> if [ ! -z "$port" ]; then
>>> instance=`instance_for_port $port $options`
>>> fi
>>>
>>> case $action in
>>> reboot|reset)
>>> status=`instance_status $instance`
>>> if [ "$status" != "stopped" ]; then
>>> instance_off
>>> fi
>>> while true;
>>> do
>>> status=`instance_status $instance`
>>> if [ "$status" = "stopped" ]; then
>>> break
>>> fi
>>> sleep $sleep_time
>>> done
>>> instance_on
>>> while true;
>>> do
>>> status=`instance_status $instance`
>>> if [ "$status" = "running" ]; then
>>> break
>>> fi
>>> sleep $sleep_time
>>> done
>>> ;;
>>> poweron|on)
>>> instance_on
>>> while true;
>>> do
>>> status=`instance_status $instance`
>>> if [ "$
> status" = "running" ]; then
>>> break
>>> fi
>>> done
>>> ;;
>>> poweroff|off)
>>> instance_off
>>> while true;
>>> do
>>> status=`instance_status $instance`
>>> if [ "$status" = "stopped" ]; then
>>> break
>>> fi
>>> sleep $sleep_time
>>> done
>>> ;;
>>> monitor)
>>> # Is the device ok?
>>> aws ec2 describe-instances $options | grep INSTANCES &> /dev/null
>>> ;;
>>> gethosts|hostlist|list)
>>> # List of names we know about
>>> a=`aws ec2 describe-instances $options | awk -v tag_pat="^TAGS¥t$ec2_tag¥t" -F '¥t' '{
>>> if (/^INSTANCES/) { printf "%s¥n", $8 }
>>> else if ( $1"¥t"$2"¥t" ‾ tag_pat ) { printf "%s¥n", $3 }
>>> }' | sort -u`
>>> echo $a
>>> ;;
>>> stat|status)
>>> instance_status $instance > /dev/null
>>> ;;
>>> *) ha_log.sh err "Unknown action: $action"; exit 1;;
>>> esac
>>>
>>> status=$?
>>>
>>> if [ $quiet -eq 1 ]; then
>>> : nothing
>>> elif [ $status -eq 0 ]; then
>>> ha_log.sh info "Operation $action passed"
>>> else
>>> ha_log.sh err "Operation $action failed: $status"
>>> fi
>>> exit $status
>>
>>> _______________________________________________
>>> Linux-HA mailing list
>>> Linux-HA at lists.linux-ha.org
>>> http://lists.linux-ha.org/mailman/listinfo/linux-ha
>>> See also: http://linux-ha.org/ReportingProblems
>>
>> _______________________________________________
>> Linux-HA mailing list
>> Linux-HA at lists.linux-ha.org
>> http://lists.linux-ha.org/mailman/listinfo/linux-ha
>> See also: http://linux-ha.org/ReportingProblems
>>
>>
>
>
>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
--
----------------------------------------------------
東 一彦
NTT OSSセンタ 基盤技術ユニット 高信頼担当
(SV総研 ソフトウェアイノベーションセンタ OSS推進PJ)
Mail:higashi.kazuhiko at lab.ntt.co.jp
Tel :03-5860-5135 (直通：5111)
〒108-8019 東京都港区港南1-9-1 NTT品川TWINSビル11階
----------------------------------------------------
More information about the Users
mailing list