[ClusterLabs] issue with awscli profile for AWS resource agents

[kgaillot at redhat.com]

On Thu, 2021-07-08 at 13:18 +0000, Aaron Kennedy wrote:
>  
> Hello,
>  
> I am trying to use AWS resource agents such as ‘awsvip’ and ‘awseip’
> but my awscli profile “could not be found”
>  
> [ec2-user at ip-172-31-43-116 ~]$ sudo pcs resource debug-start --full
> privip                                              
>                                                                 
>  warning: unpack_rsc_op_failure:        Processing failed start of
> privip on ip-172-31-43-84.eu-west-1.compute.internal: not running |
> rc=7
> warning: unpack_rsc_op_failure:        Processing failed start of
> privip on ip-172-31-43-116.eu-west-1.compute.internal: not running |
> rc=7
> warning: unpack_rsc_op_failure:        Processing failed start of
> privip on ip-172-31-45-167.eu-west-1.compute.internal: not running |
> rc=7
> Operation start for privip (ocf:heartbeat:awsvip) returned: 'not
> running' (7)
> >  stderr: + 12:47:21: 45: OCF_RESKEY_awscli_default=/usr/bin/aws
> >  stderr: + 12:47:21: 46: OCF_RESKEY_profile_default=default
> >  stderr: + 12:47:21: 47: OCF_RESKEY_api_delay_default=3
> >  stderr: + 12:47:21: 49: : /usr/bin/aws
> >  stderr: + 12:47:21: 50: : default
> >  stderr: + 12:47:21: 51: : 3
> >  stderr: + 12:47:21: 196: case $__OCF_ACTION in
> >  stderr: + 12:47:21: 203: AWSCLI=/usr/bin/aws
> >  stderr: + 12:47:21: 204: SECONDARY_PRIVATE_IP=172.31.45.5
> >  stderr: ++ 12:47:21: 205: curl -s 
> http://169.254.169.254/latest/meta-data/instance-id
> >  stderr: + 12:47:21: 205: INSTANCE_ID=i-0ea1bfc7adcc6197e
> >  stderr: ++ 12:47:21: 206: /usr/bin/aws --profile default --output
> json ec2 describe-instances --instance-id i-0ea1bfc7adcc6197e
> >  stderr: ++ 12:47:21: 206: awk '-F"' '{print$4}'
> >  stderr: ++ 12:47:21: 206: grep -m 1 eni
> >  stderr:
> >  stderr: The config profile (default) could not be found
> >  stderr: + 12:47:21: 206: NETWORK_ID=
> >  stderr: + 12:47:21: 208: case $__OCF_ACTION in
> >  stderr: + 12:47:21: 210: awsvip_validate
> >  stderr: + 12:47:21: awsvip_validate:181: check_binary /usr/bin/aws
> >  stderr: + 12:47:21: check_binary:56: have_binary /usr/bin/aws
> >  stderr: + 12:47:21: have_binary:68: '[' '' = 1 ']'
> >  stderr: ++ 12:47:21: have_binary:71: echo /usr/bin/aws
> >  stderr: ++ 12:47:21: have_binary:71: sed -e 's/ -.*//'
> >  stderr: + 12:47:21: have_binary:71: local bin=/usr/bin/aws
> >  stderr: ++ 12:47:21: have_binary:72: which /usr/bin/aws
> >  stderr: + 12:47:21: have_binary:72: test -x /usr/bin/aws
> >  stderr: + 12:47:21: awsvip_validate:183: '[' -z default ']'
> >  stderr: + 12:47:21: awsvip_validate:188: '[' -z i-
> 0ea1bfc7adcc6197e ']'
> >  stderr: + 12:47:21: awsvip_validate:193: return 0
> >  stderr: + 12:47:21: 211: awsvip_start
> >  stderr: + 12:47:21: awsvip_start:132: awsvip_monitor
> >  stderr: + 12:47:21: awsvip_monitor:171: /usr/bin/aws --profile
> default ec2 describe-instances --instance-id i-0ea1bfc7adcc6197e
> >  stderr: + 12:47:21: awsvip_monitor:171: grep -q 172.31.45.5
> >  stderr:
> >  stderr: The config profile (default) could not be found
> <truncated>
>  
> But if I simply run the binary with the same options there is no
> issue:
>  
> [ec2-user at ip-172-31-43-116 ~]$ /usr/bin/aws --profile default --
> output json ec2 describe-instances --instance-id i-0ea1bfc7adcc6197e
> {
>     "Reservations": [
>         {
>             "Instances": [
>                 {
>                     "Monitoring": {
>                         "State": "disabled"
>                     },
>                     "PublicDnsName": "ec2-34-243-96-249.eu-west-
> 1.compute.amazonaws.com",
>                     "State": {
>                         "Code": 16,
>                         "Name": "running"
>                     },
>                     "EbsOptimized": true,
>                     "LaunchTime": "2021-07-07T11:10:57.000Z",
>                     "PublicIpAddress": "34.243.96.249",
>                     "PrivateIpAddress": "172.31.43.116",
>                     "ProductCodes": [],
>                     "VpcId": "vpc-58799c3f",
>                    "CpuOptions": {
>                         "CoreCount": 1,
>                         "ThreadsPerCore": 2
>                     },
> <truncated>

If something runs from the command line but not from the cluster, the
first thing I check is SELinux denials. Another possibility is a
missing ordering (in the boot system or the cluster) such that some
required dependency is not yet available when the agent is run.
 
> I’ve tried recreating the awscli profile with a named option (instead
> of default) and create the resource with the same named option
> (instead of default) but I get the same result, eg “The config
> profile (aaron) could not be found.” I’m not sure how to troubleshoot
> this further since the RA is leveraging /usr/bin/aws which is using a
> properly configured profile.
>  
> Thanks for any helpful suggestions!
>  
> Best regards,
> Aaron
-- 
Ken Gaillot <kgaillot at redhat.com>