[Pacemaker] custom resource-agent

Wed Oct 2 04:41:04 EDT 2013

I guess the question is if this is just observed performance, or if this is 
something that the API spec guarantees. If it's just observed performance, I 
would be a bit leery about depending on it being instantanious, as it may not be 
if Amazon is under load.

David Lang

On Wed, 2 Oct 2013, Peter Romfeld wrote:

> the changes take instant effect, i let "watch
> /usr/lib64/nagios/plugins/check_tcp -H my.vip.add.r -p 3306" on a 3rd
> instance running, and the second i issued on the new master node the aws
> command it changed from CRITICAL to OK
>
>
> On Wed, Oct 2, 2013 at 4:13 PM, Peter Romfeld <peter.romfeld.hk at gmail.com>wrote:
>
>> In you VPC the change take effect after you got "true", you can check it
>> with describe-eip or in console :)
>> right now i just try to add the command with hardcoded variables to
>> IPaddr2 start, just to get it running for now
>>
>> i put my attempt to create a resource agent on github, maybe someone can
>> help me find why params are not working?
>> wget https://raw.github.com/peterromfeldhk/pacemaker/master/AWSFIP
>>
>> sorry im just started with scripting with nagios_nrpe last project :)
>> i have 2 problems, 1s2 the OCF_RESKEY_ params dont work as i thought, if i
>> hardcode the variables the commands are working at least in testrun, the
>> second big issue i have is the monitoring
>>
>>
>> On Wed, Oct 2, 2013 at 2:34 PM, David Lang <david at lang.hm> wrote:
>>
>>> Unless something has changed in the AWS API in the last few months, when
>>> the aws command exits successfully, that doesn't mean the change has taken
>>> effect, just that the aws system has accepted the change and it will take
>>> effect 'soon'
>>>
>>>
>>> David Lang
>>>
>>> On Wed, 2 Oct 2013, Peter Romfeld wrote:
>>>
>>>  yes i need to use the aws command, i am using a VPC, after issueing the
>>>> command i get a "true" statement and its done
>>>>
>>>> so i only want pacemaker to issue the one-shot command at failover. Here
>>>> is
>>>> what i have atm: (i know its still dirty, just lerning pacemaker)
>>>>
>>>> primitive drbd_mysql ocf:linbit:drbd \
>>>>        params drbd_resource="mydata" \
>>>>        op monitor interval="15s"
>>>> primitive fs_mysql ocf:heartbeat:Filesystem \
>>>>        params device="/dev/drbdx" directory="/mountpint" fstype="ext4"
>>>> options="relatime,barrier=1" \
>>>>        op start interval="0" timeout="60" \
>>>>        op stop interval="0" timeout="60" \
>>>>        op monitor interval="10s" timeout="60s" OCF_CHECK_LEVEL="20" \
>>>>        meta target-role="started"
>>>> primitive fvip ocf:heartbeat:AWSFIP \
>>>>        params fvip="192.168.2.10" region="ap-southeast-1"
>>>> primitive ip_mysql ocf:heartbeat:IPaddr2 \
>>>>        params ip="192.168.2.10" cidr_netmask="20" \
>>>>        op monitor interval="10" \
>>>>        meta target-role="started"
>>>> primitive mysqld lsb:mysql
>>>> group mysql fs_mysql ip_mysql mysqld
>>>> ms ms_drbd_mysql drbd_mysql \
>>>>        meta master-max="1" master-node-max="1" clone-max="2"
>>>> clone-node-max="1" notify="true"
>>>> colocation mysql_on_drbd inf: fvip mysql ms_drbd_mysql:Master
>>>> order mysql_after_drbd_and_fvip inf: ms_drbd_mysql:promote fvip:start
>>>> mysql:start
>>>>
>>>> my AWSFIP(adjusted Dummy :%s/dummy/awsfip/g|%s/Dummy/**AWSFIP/g):
>>>>
>>>> <parameter name="fvip" unique="1" required="1">
>>>> <longdesc lang="en">
>>>> The IPv4 address to be configured in dotted quad notation, for example
>>>> "192.168.1.1".
>>>> </longdesc>
>>>> <shortdesc lang="en">IPv4 address</shortdesc>
>>>> <content type="string" default="" />
>>>> </parameter>
>>>>
>>>> <parameter name="region" unique="1" required="1">
>>>> <longdesc lang="en">
>>>> The name of the AWS region
>>>> </longdesc>
>>>> <shortdesc lang="en">AWS region</shortdesc>
>>>> <content type="string"/>
>>>> </parameter>
>>>>
>>>> awsfip_start() {
>>>>    awsfip_monitor
>>>>    Instance_ID=`/usr/bin/curl --silent
>>>> http://169.254.169.254/latest/**meta-data/instance-id`<http://169.254.169.254/latest/meta-data/instance-id>
>>>>    ENI_ID=`aws ec2 describe-instances --instance-id $Instance_ID --region
>>>> $OCF_RESKEY_region | grep NetworkInterfaceId | cut -d '"' -f 4`
>>>>    if [ $? =  $OCF_SUCCESS ]; then
>>>>        return $OCF_SUCCESS
>>>>    fi
>>>>         aws ec2 assign-private-ip-addresses --network-interface-id
>>>> $ENI_ID
>>>> --private-ip-addresses $OCF_RESKEY_fvip --allow-reassignment --region
>>>> $OCF_RESKEY_region
>>>>         sleep 4
>>>>         aws ec2 assign-private-ip-addresses --network-interface-id
>>>> $ENI_ID
>>>> --private-ip-addresses $OCF_RESKEY_fvip --allow-reassignment --region
>>>> $OCF_RESKEY_region
>>>>         /etc/init.d/networking restart
>>>>    touch ${OCF_RESKEY_state}
>>>> }
>>>>
>>>> I couldn't get it to work yet, and i don't want to run a external script
>>>> for it. I can't be so hard to let pacemaker execute an additional
>>>> one-shot
>>>> command at failover (in the correct order..)
>>>>
>>>> Thanks for your help!
>>>>
>>>>
>>>> On Wednesday, October 02, 2013 07:33 AM, David Lang wrote:
>>>>
>>>> the aws command is making the call to inform aws, if you were to bring up
>>>> the address without making the aws command, would it work? If you are on
>>>> a
>>>> Virtual Private Cloud (VPC), it may, but I didn't think it would.
>>>>
>>>> If you can make it work without the aws command, then you can just use
>>>> the
>>>> standard pacemaker VIP configuration. I know that this doesn't work if
>>>> you
>>>> have an external IP that you are moving (you must use an aws call to tell
>>>> Amazon to move the IP), but it's possible that you don't have to for an
>>>> internal IP, but I would be surprised.
>>>>
>>>> David Lang
>>>>
>>>>
>>>> On Wed, 2 Oct 2013, Peter Romfeld wrote:
>>>>
>>>> Hey,
>>>> when i change the secondary IP per hand or with external script on a
>>>> Ubuntu
>>>> Instance I just need:
>>>> /etc/network/interfaces
>>>> auto eth0
>>>> iface eth0 inet dhcp
>>>>   address 192.168.32.12
>>>>   netmask 255.255.240.0
>>>>   gateway 192.168.32.1
>>>>   up ip addr add 192.168.32.11/20 dev eth0
>>>>
>>>> and then run the script which basically just does:
>>>> #!/bin/sh
>>>>
>>>> VIP=172.32.32.11
>>>> REGION=ap-southeast-1
>>>>
>>>> Instance_ID=`/usr/bin/curl --silent
>>>> http://169.254.169.254/latest/**meta-data/instance-id`<http://169.254.169.254/latest/meta-data/instance-id>
>>>> ENI_ID=`aws ec2 describe-instances --instance-id $Instance_ID --region
>>>> $REGION | grep NetworkInterfaceId | cut -d '"' -f 4`
>>>>
>>>> aws ec2 assign-private-ip-addresses --network-interface-id $ENI_ID
>>>> --private-ip-addresses $VIP --allow-reassignment --region $REGION
>>>>
>>>>
>>>> I dont need to inform AWS or restart network, only the correct network
>>>> config and the one command, when i tested it with pinging from a 3rd
>>>> instance during IP change i didnt got any interupts. I dont know about
>>>> monitoring it
>>>>
>>>>
>>>> On Wed, Oct 2, 2013 at 1:38 AM, David Lang <david at lang.hm>
>>>> <david at lang.hm>wrote:
>>>>
>>>> On Tue, 1 Oct 2013, Dejan Muhamedagic wrote:
>>>>
>>>> On Tue, Oct 01, 2013 at 10:07:12AM -0700, David Lang wrote:
>>>>
>>>>
>>>> On Tue, 1 Oct 2013, Dejan Muhamedagic wrote:
>>>>
>>>> On Tue, Oct 01, 2013 at 07:22:20AM -0700, David Lang wrote:
>>>>
>>>>
>>>> On Tue, 1 Oct 2013, Dejan Muhamedagic wrote:
>>>>
>>>> Hi David,
>>>>
>>>>
>>>> On Mon, Sep 30, 2013 at 12:41:23PM -0700, David Lang wrote:
>>>>
>>>> On Mon, 30 Sep 2013, David Lang wrote:
>>>>
>>>> On Mon, 30 Sep 2013, Michael Schwartzkopff wrote:
>>>>
>>>>
>>>> Am Montag, 30. September 2013, 21:12:56 schrieb Peter Romfeld:
>>>>
>>>>
>>>> I am working in AWS i cant just use a VIP i need to use a floating
>>>> secondary IP which i reassign through script, i want to let
>>>> pacemaker
>>>> handle the reassignment...
>>>>
>>>>
>>>> Please explain the difference of a VIP and a "secondary IP" in
>>>> your opinion.
>>>>
>>>>
>>>> with AWS you need to inform amazon of the change, not just change
>>>> the IP on the local box, that requires much more work than a
>>>> simple local VIP
>>>>
>>>>
>>>> being more detailed, instead of just
>>>> ifconfig eth0:0 $vip
>>>> you have to do something like
>>>>
>>>> /opt/aws/bin/ec2-assign-****private-ip-addresses -n $ENI_ID
>>>> --secondary-private-ip-address $VIP --allow-reassignment --region $REGION
>>>>
>>>>
>>>> We may consider adding such an option to IPaddr2. Has anybody
>>>> ever tried that?
>>>>
>>>> pingresult=`ping -c 1 -W 1 $VIP | grep time= | wc -l`
>>>>
>>>> if [ "$pingresult" == "0" ]; then
>>>> echo `date` "-- Restarting network"
>>>> /sbin/service network restart > /dev/null 2>&1
>>>>
>>>>
>>>> That may break the cluster communication, which may lead to split
>>>> brain, etc. Is that really the only way?
>>>>
>>>>
>>>> It's not the only way, but you do have the problem that the call to
>>>> aws management interface is asynchronous, you don't know when it's
>>>> going to complete, and until it does, the IP doesn't actually work.
>>>>
>>>>
>>>> Wouldn't it be then safer to wait until it starts working, i.e.
>>>> to monitor in a loop?
>>>>
>>>>
>>>> that's exactly what the snippet of code above is for, to detect when
>>>> the other box no longer has the address.
>>>>
>>>>
>>>> Hmm, perhaps I'm missing something, but I couldn't notice a loop
>>>> in that code. What I meant was something like this:
>>>>
>>>> while ! ping -c 1 -W 1 $VIP | grep -qs time=; do
>>>>        :
>>>> done
>>>>
>>>> Then network restart wouldn't be necessary, right? Sorry, I don't
>>>> know much about aws.
>>>>
>>>>
>>>> I haven't used this exact script before, but I have seen the problem that
>>>> this script is designed to address. I am not saying that I agree with
>>>> this
>>>> script, but it's what Amazon is suggesting, so it's probably a reasonable
>>>> start.
>>>>
>>>>
>>>> this was a cut-n-paste from the URL provided earlier
>>>> http://aws.amazon.com/****articles/2127188135977316<http://aws.amazon.com/**articles/2127188135977316>
>>>> <http://aws.amazon.com/**articles/2127188135977316<http://aws.amazon.com/articles/2127188135977316>
>>>>> <htt**p://aws.amazon.com/articles/**2127188135977316<http://aws.amazon.com/articles/2127188135977316>
>>>>>
>>>>
>>>>
>>>>
>>>> #!/bin/sh
>>>> # This script will monitor another HA node and take over a Virtual IP
>>>> (VIP)
>>>> # if communication with the other node fails
>>>>
>>>> # High Availability IP variables
>>>> # Other node's IP to ping and VIP to swap if other node goes down
>>>> HA_Node_IP=10.0.0.11
>>>> VIP=10.0.0.10
>>>>
>>>> # Specify the EC2 region that this will be running in
>>>> REGION=us-west-2
>>>>
>>>> # Run aws-apitools-common.sh to set up default environment variables and
>>>> to
>>>> # leverage AWS security credentials provided by EC2 roles
>>>> . /etc/profile.d/aws-apitools-****common.sh
>>>>
>>>> # Determine the instance and ENI IDs so we can reassign the VIP to the
>>>> # correct ENI. Requires EC2 describe-instances and
>>>> assign-private-ip-address
>>>> # permissions. The following example EC2 roles policy will authorize
>>>> these
>>>> # commands:
>>>> # {
>>>> # "Statement": [
>>>> # {
>>>> # "Action": [
>>>> # "ec2:AssignPrivateIpAddresses"****,
>>>> # "ec2:DescribeInstances"
>>>> # ],
>>>> # "Effect": "Allow",
>>>> # "Resource": "*"
>>>> # }
>>>> # ]
>>>> # }
>>>>
>>>> Instance_ID=`/usr/bin/curl --silent http://169.254.169.254/latest/****<http://169.254.169.254/latest/**>
>>>> meta-data/instance-id`ENI_ID=`****/opt/aws/bin/ec2-describe-****
>>>> instances
>>>> <http://169.254.169.254/**latest/meta-data/instance-**
>>>> idENI_ID=/opt/aws/bin/ec2-**describe-instances<http://169.254.169.254/latest/meta-data/instance-idENI_ID=/opt/aws/bin/ec2-describe-instances>
>>>>> <http://**169.254.169.254/latest/meta-**data/instance-idENI_ID=/opt/**
>>>> aws/bin/ec2-describe-instances<http://169.254.169.254/latest/meta-data/instance-idENI_ID=/opt/aws/bin/ec2-describe-instances>
>>>> **>$Instance_ID
>>>>
>>>> --region $REGION | grep eni -m 1 | awk '{print $2;}'`
>>>>
>>>> echo `date` "-- Starting HA monitor"
>>>> while [ . ]; do
>>>> pingresult=`ping -c 3 -W 1 $HA_Node_IP | grep time= | wc -l`
>>>>
>>>> if [ "$pingresult" == "0" ]; then
>>>> echo `date` "-- HA heartbeat failed, taking over VIP"
>>>>
>>>> /opt/aws/bin/ec2-assign-****private-ip-addresses -n $ENI_ID
>>>> --secondary-private-ip-address $VIP --allow-reassignment --region $REGION
>>>> pingresult=`ping -c 1 -W 1 $VIP | grep time= | wc -l`
>>>> if [ "$pingresult" == "0" ]; then
>>>> echo `date` "-- Restarting network"
>>>> /sbin/service network restart > /dev/null 2>&1
>>>> fi
>>>> sleep 60
>>>> fi
>>>> sleep 2
>>>> done
>>>>
>>>>
>>>>
>>>> David Lang
>>>>
>>>> ______________________________****_________________
>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>> http://oss.clusterlabs.org/****mailman/listinfo/pacemaker<http://oss.clusterlabs.org/**mailman/listinfo/pacemaker>
>>>> <http://oss.clusterlabs.org/**mailman/listinfo/pacemaker<http://oss.clusterlabs.org/mailman/listinfo/pacemaker>
>>>>> <ht**tp://oss.clusterlabs.org/**mailman/listinfo/pacemaker<http://oss.clusterlabs.org/mailman/listinfo/pacemaker>
>>>>>
>>>>
>>>>
>>>> Project Home: http://www.clusterlabs.org
>>>> Getting started: http://www.clusterlabs.org/****
>>>> doc/Cluster_from_Scratch.pdf<http://www.clusterlabs.org/**doc/Cluster_from_Scratch.pdf>
>>>> <http://www.clusterlabs.org/**doc/Cluster_from_Scratch.pdf<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>>>> <**http://www.clusterlabs.org/**doc/Cluster_from_Scratch.pdf<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>>>>
>>>>
>>>> Bugs: http://bugs.clusterlabs.org
>>>>
>>>>
>>>>
>>>>
>>>> ______________________________**_________________
>>>>
>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>> http://oss.clusterlabs.org/**mailman/listinfo/pacemaker<http://oss.clusterlabs.org/mailman/listinfo/pacemaker>
>>>>
>>>>
>>>>
>>>> Project Home: http://www.clusterlabs.org
>>>>
>>>> Getting started: http://www.clusterlabs.org/**
>>>> doc/Cluster_from_Scratch.pdf<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>>>
>>>> Bugs: http://bugs.clusterlabs.org
>>>>
>>>>
>>>>
>>>> ______________________________**_________________
>>>> Pacemaker mailing list:
>>>> Pacemaker at oss.clusterlabs.**orghttp://oss.clusterlabs.org/**
>>>> mailman/listinfo/pacemaker<http://oss.clusterlabs.org/mailman/listinfo/pacemaker>
>>>>
>>>>
>>>> Project Home: http://www.clusterlabs.org
>>>> Getting started: http://www.clusterlabs.org/**
>>>> doc/Cluster_from_Scratch.pdf<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>>> Bugs: http://bugs.clusterlabs.org
>>>>
>>>
>>> _______________________________________________
>>>
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>>
>>>
>>> Project Home: http://www.clusterlabs.org
>>>
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>
>>> Bugs: http://bugs.clusterlabs.org
>>>
>>>
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>>
>>>
>>
>
-------------- next part --------------
_______________________________________________

Pacemaker mailing list: Pacemaker at oss.clusterlabs.org

http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org

Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf

Bugs: http://bugs.clusterlabs.org