Home / server / Questions / 614890
Accepted
binaryorganic
Asked: 2014-07-24 07:51:59 +0800 CST

Can't run AWS CLI from CRON (credentials)

  • 772

Trying to run a simple AWS CLI backup script. It loops through lines in an include file, backs those paths up to S3, and dumps output to a log file. When I run this command directly, it runs without any error. When I run it through CRON I get an "Unable to locate credentials" error in my output log.

The shell script:

AWS_CONFIG_FILE="~/.aws/config"

while read p; do
 /usr/local/bin/aws s3 cp $p s3://PATH/TO/BUCKET --recursive >> /PATH/TO/LOG 2>&1
done </PATH/TO/INCLUDE/include.txt

I only added the line to the config file after I started seeing the error, thinking this might fix it (even though I'm pretty sure that's where AWS looks by default).

Shell script is running as root. I can see the AWS config file at the specified location. And it all looks good to me (like I said, it runs fine outside of CRON).

bash

14 Answers

  1. When you run a job from crontab, your $HOME environment variable is /

    The Amazon client looks for either

    ~/.aws/config

    or

    ~/.aws/credentials

    If $HOME = /, then the client won't find those files

    To make it work, update your script so that it exports an actual home directory for $HOME

    export HOME=/root

    and then put a config or credentials files in 

    /root/.aws/
    • 45
  2. Best Answer

    If it works when you run it directly but not from cron there is probably something different in the environment. You can save your environment interactively by doing

    set | sort > env.interactive

    And do the same thing in your script

    set | sort > /tmp/env.cron

    And then diff /tmp/env.cron env.interactive and see what matters. Things like PATH are the most likely culprits.

    • 24

  3. I was able to solve this issue through the following:

    export AWS_CONFIG_FILE="/root/.aws/config"
export AWS_ACCESS_KEY_ID=XXXX
export AWS_SECRET_ACCESS_KEY=YYYY
    • 6

  4. Put this code before your command line to be executed into crontab -e

    SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    • 3

  5. The aws cli tool's binaries are installed under /usr/local/bin/aws.

    The error I had is that the cron user could not access /usr/local/bin/aws while running; it can only access /usr/bin/

    What I did was to create a link in /usr/bin for aws with the below command.

    root@gateway:~# ln -s /usr/local/bin/aws /usr/bin/aws

    I also added some changes in my script; here is a sample function:

    starter () {
    echo "
    ==================================================

    Starting Instance

    ==================================================
    "

    /usr/bin/aws ec2 start-instances --instance-ids $instance --region us-east-1

    sleep 30

    echo "Assigning IP Address "

    /usr/bin/aws ec2 associate-address --instance-id $instance  --region us-east-1 --public-ip XX.XX.XX.XX

}

    And the cron entry:

    30 5 * * * sh /usr/local/cron/magentocron.sh

    This method worked for me.

    • 1

  6. This line in the default .bashrc file for the user will prevent non-interactive shells from getting the full user environment (including the PATH variable):

    # If not running interactively, don't do anything
[ -z "$PS1" ] && return

    Comment the line out to allow $HOME/.bashrc to be executed from a non-interactive context.

    I also had to add an explicit source command to my shell script to get the environment set up correctly:

    #!/bin/bash
source $HOME/.bashrc

    See this answer for additional info.

    • 1

  7. We all know that environment path variable $PATH has location of binaries. $PATH of Crontab might not have location awscli.

    What you can do is, find the path of awscli binary.

    # which aws
/usr/local/bin/aws

    and add the path in $PATH of crontab by adding below line in the beginning of your script(after shebang).

    PATH=$PATH:/usr/local/bin/

    This worked for me!!!

    • 1

  8. I know it's not the perfect solution but that worked for me:

    export HOME=/home/user
export AWS_CONFIG_FILE="/home/user/.aws/config"
export AWS_ACCESS_KEY_ID=XXX
export AWS_SECRET_ACCESS_KEY=XXX
    • 0

  9. Just to add some value added i was having issue with new bash version while using awscli tool installed via PIP i found that nothing will work with this tool with new bash versions.

    I was able to resolve by installing aws-apitools-ec2 this can be install by 

    yum install -y aws-apitools-ec2

    I am attaching its guide for more reference.

    http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ec2-clt.pdf

    • 0

  10. I had the same issue, but after removing the stderr redirect from my cron entry (2>@1), I saw aws: command not found in the log.

    This is because the AWS cli was installed in the user's home folder and I had added a line to my user's .bash_profile to add the AWS cli path to the $PATH. Oddly, this is in fact the way that the AWS cli install documentation tells you to install it. But the user's .bash_profile doesn't get used when the user's crontab is executed (at least not in my environment anyway).

    So all I did to fix this was ensure my crontab script also had the aws cli in its path. So below the shebang of my script, I now have PATH=~/.local/bin:$PATH.

    • 0