Running bash script with AWS CLI on Lambda

AWS Lambda can easily run bash scripts, yet how do I run it with AWS CLI?

The answer is to create a package that has all of the required modules. First run python virtualenv, then install virtualenv and AWS CLI into the virtual environment. Then create a tarball of the virtual environment with AWS CLI. The tarball will be included in package.

When the python script calls our bash script it creates a virtual environment in /tmp and activates it.

For details how to create the virtual environment see v.sh below.

I have a simple bash script that performs aws ec2 describe-instances then then places the output into a S3 bucket:

|16:22:21|ec2-user@ip-172-31-10-191:[task]> pwd
/var/task
|16:22:24|ec2-user@ip-172-31-10-191:[task]> ls -l
total 19840
-rwxr-xr-x 1 ec2-user ec2-user     583 Jul 21 16:14 bash_function.py
-rw-rw-r-- 1 ec2-user ec2-user 9938086 Jul 21 16:18 bash_function.zip
-rw-rw-r-- 1 ec2-user ec2-user 9844717 Jul 21 11:21 env.tgz
-rwxr-xr-x 1 ec2-user ec2-user  497799 Jul 20 12:21 jq
-rw-rw-r-- 1 ec2-user ec2-user      97 Jul 21 13:21 lambdaupdate
-rw-rw-r-- 1 ec2-user ec2-user    5815 Jul 21 15:20 linuxinfo.sh
-rw-rw-r-- 1 ec2-user ec2-user    1341 Jul 21 16:18 simple-awscli.sh
-rwxr-xr-x 1 ec2-user ec2-user     167 Jul 21 11:20 v.sh
|16:22:26|ec2-user@ip-172-31-10-191:[task]> cat v.sh
cd /tmp
virtualenv env
source env/bin/activate
pip install virtualenv
pip install awscli
tar cPzf /var/task/env.tgz /tmp/env

cd /var/task
zip -r9 bash_function.zip *
|16:22:30|ec2-user@ip-172-31-10-191:[task]> cat bash_function.py
from __future__ import print_function

import json
import os
import subprocess
import commands

print('Loading function')

def lambda_handler(event, context):
    #os.system("bash ec2.sh; uname -a;id;w;free -m;vmstat 2 10;df -h;netstat -in;cat /etc/issue; netstat -nr; echo $PATH;type aws; aws ec2 describe-instances; jq 2>&1; ls -l")
    print (commands.getstatusoutput('echo;bash ./simple-awscli.sh;echo'))

    return 'Hello from Lambda'

|16:22:38|ec2-user@ip-172-31-10-191:[task]> cat simple-awscli.sh
pwd
tar xPzf ./env.tgz
source /tmp/env/bin/activate
aws ec2 describe-instances | ./jq -r  '.Reservations[].Instances[] | (.Tags | map(.value=.Value | .key=.Key) | from_entries) as $tags | "\(.InstanceId)#\(.InstanceType)#\(.PrivateIpAddress)#\(.PublicIpAddress)#\(.PublicDnsName)#\(.State.Name)#\(.Platform)#\(.LaunchTime)#\(.VpcId)#\(.SubnetId)#\(.Placement.AvailabilityZone)#\(.KeyName)#\(.ImageId)#\(.VirtualizationType)#\(.Monitoring.State)#\(.BlockDeviceMappings[0].Ebs.VolumeId)#\(.BlockDeviceMappings[1].Ebs.VolumeId)#\(.BlockDeviceMappings[2].Ebs.VolumeId)#\(.BlockDeviceMappings[3].Ebs.VolumeId)#\(.BlockDeviceMappings[4].Ebs.VolumeId)#\(.BlockDeviceMappings[5].Ebs.VolumeId)#\(.BlockDeviceMappings[6].Ebs.VolumeId)#\(.BlockDeviceMappings[7].Ebs.VolumeId)#\(.BlockDeviceMappings[8].Ebs.VolumeId)#\(.BlockDeviceMappings[9].Ebs.VolumeId)#\(.SecurityGroups[0].GroupId)#\(.SecurityGroups[1].GroupId)#\(.SecurityGroups[2].GroupId)#\(.SecurityGroups[3].GroupId)#\(.SecurityGroups[4].GroupId)#\($tags.Name)#\($tags.Product)#\($tags.Bu)#\($tags.Environment)#\($tags.Owner)#\($tags.Cc)#\($tags.Capacity)#\($tags["aws:cloudformation:stack-name"])"'  2>/dev/null |  tr ',' ';' | tr '#' ',' | cut -d',' -f1-15 > /tmp/instances-`date +%d%m%Y`.csv
aws s3 cp /tmp/instances-`date +%d%m%Y`.csv s3://lambdatestbucket/instances-`date +%d%m%Y`.csv

Lambda output:

START RequestId: fe19f570-4f0a-11e6-b416-c51cea3fb147 Version: $LATEST
(0, '
/var/task
upload: ../../tmp/instances-21072016.csv to s3://lambdatestbucket123/instances-21072016.csv
')
END RequestId: fe19f570-4f0a-11e6-b416-c51cea3fb147
REPORT RequestId: fe19f570-4f0a-11e6-b416-c51cea3fb147 Duration: 3725.94 ms Billed Duration: 3800 ms  Memory Size: 1024 MB Max Memory Used: 106 MB 

S3 bucket object:

i-0ed25bd1,t2.medium,172.27.2.94,null,,stopped,null,2016-07-13T05:59:24.000Z,vpc-b48c48d1,subnet-3897455d,ap-southeast-2a,cloudformation-ap-southeast-2,ami-e17f37db,hvm,disabled
i-02ee982f,t2.small,172.27.3.149,<REDACTED>,ec2-52-62-118-12.ap-southeast-2.compute.amazonaws.com,stopped,null,2016-07-19T23:52:30.000Z,vpc-b48c48d1,subnet-3f97455a,ap-southeast-2a,<REDACTED>,ami-ba3e14d9,hvm,disabled
i-5ac68d84,r3.large,172.27.2.52,null,,stopped,null,2016-07-12T01:58:38.000Z,vpc-b48c48d1,subnet-bb3dc5cc,ap-southeast-2b,cloudformation-ap-southeast-2,ami-1d713927,hvm,disabled
i-8cf2c752,t2.medium,172.27.3.221,<REDACTED>,ec2-52-62-138-11.ap-southeast-2.compute.amazonaws.com,running,null,2015-12-23T00:23:18.000Z,vpc-b48c48d1,subnet-a53dc5d2,ap-southeast-2b,<REDACTED>,ami-63f79559,hvm,disabled
i-41628d90,t2.medium,192.168.133.221,<REDACTED>,ec2-52-63-107-9.ap-southeast-2.compute.amazonaws.com,running,null,2016-04-22T06:18:09.000Z,vpc-d8723cbd,subnet-37391c40,ap-southeast-2b,<REDACTED>,ami-63f79559,hvm,disabled
i-6d9398b3,t2.medium,172.27.151.88,<REDACTED>,ec2-52-62-233-205.ap-southeast-2.compute.amazonaws.com,running,null,2016-02-15T06:10:02.000Z,vpc-e894fa8d,subnet-fb88f18c,ap-southeast-2b,<REDACTED>,ami-63f79559,hvm,disabled
i-1c171dc3,t2.medium,192.168.133.198,<REDACTED>,ec2-52-63-252-137.ap-southeast-2.compute.amazonaws.com,running,null,2016-04-22T06:17:15.000Z,vpc-d8723cbd,subnet-9d655af8,ap-southeast-2a,<REDACTED>,ami-63f79559,hvm,disabled
i-33e36aec,r3.large,172.27.2.79,null,,stopped,null,2016-07-12T01:56:46.000Z,vpc-b48c48d1,subnet-3897455d,ap-southeast-2a,cloudformation-ap-southeast-2,ami-df7f37e5,hvm,disabled
i-8e326d50,t2.medium,172.27.2.46,null,,stopped,null,2016-05-23T11:43:49.000Z,vpc-b48c48d1,subnet-bb3dc5cc,ap-southeast-2b,cloudformation-ap-southeast-2,ami-48d38c2b,hvm,disabled
i-a58ef27a,t2.medium,172.27.3.198,<REDACTED>,ec2-52-62-141-27.ap-southeast-2.compute.amazonaws.com,running,null,2015-12-23T00:22:08.000Z,vpc-b48c48d1,subnet-3a97455f,ap-southeast-2a,<REDACTED>,ami-63f79559,hvm,disabled

i-a5ce857b,c4.xlarge,172.27.2.6,null,,stopped,null,2016-07-06T00:13:42.000Z,vpc-b48c48d1,subnet-bb3dc5cc,ap-southeast-2b,cloudformation-ap-southeast-2,ami-af723a95,hvm,disabled

Update 29/7/2016 - Easy to add cloudwatch logging:
update v.sh to include: pip install awscli-cwlogs
update simple-awscli.sh to include: 
export AWS_CONFIG_FILE=/tmp/.aws/config
aws configure set plugins.cwlogs cwlogs
cat /tmp/instances-`date +%d%m%Y`.csv | aws logs push --log-group-name simpletest --log-stream-name instances

Can also send emails using SNS:

aws sns publish --topic-arn arn:aws:sns:ap-southeast-2:43152512345:EmailTopic --subject "Message from Lambda simple-awscli.sh" --message "`cat simple-awscli.sh`"