Useful JMESPath in AWS CLI

  Posted On September 25, 2017  |    8 minutes

Thought I’d take the time to consolidate some of the learnings I’ve made along the way when making queries against an AWS account using the command line interface. Originally I came from a development background and did most of my automation scripts as a combination of Python 2.7, Boto3 and Fabric. While I still see these as having their place when managing complex objects or performing heavier string manipulation I’ve found you can get a long way with a combination of Bash scripting, AWS CLI and Cloudformation templates.

The reasoning behind this is if you are following a truly immutable architecture approach you will push a lot of your infrastructure into stack-like layers of Cloudformation templates which work as a place:

  • to make reference to important resources in other stacks;
  • for logical grouping of modular functionality;
  • to update and use as a reflection of state;
  • to be a source of truth for architecture, kept in a VCS such as Git.

With Cloudformation stacks keeping track of the state of your resources all you need to do is deploy one with the aws cloudformation create-stack CLI command, passing environment variables as argument. This can be done simply through a Bash script that encapsulates the dependencies and environment tracking. This is not the only use of the AWS CLI, it can be used to make many queries on the state of an account in such a way that can be chained to yield complex automations. Couple this with the fact that many continuous integration tools support Bash scripting, you can write libraries of automation scripts and keep your CI/CD tools decoupled from execution logic, achieving high portability and reusability.

So with this it follows that you often need to pull values from the AWS API to inform the creation of infrastructure resources. The CLI tool utilises the JMESPath query language for value extraction.

JMESPath

JMESPath is the query language of the AWS CLI, since the CLI will return data in the form of JSON which has a regular structure it can be queried and be expected to return a regular value. It’s worth checking out the following resources to get a feel for JMESPath:

Dependencies

Being familiar with the standard environment variables that are referenced by the CLI tool takes a lot of the pain out of making multiple calls in an automation. Defining these once at the beginning of your CI scripts will save a lot of --region ap-southeast-2, --output text and --profile some-profile being littered throughout. These can be defined like

#!/bin/bash -ex
export AWS_REGION=your-region-here
export AWS_DEFAULT_PROFILE=your-cli-access-profile-here
export AWS_DEFAULT_OUTPUT=text

making sure export is present. Export assures the variables are passed recursively to child processes of the script (and to the childrens children etc.), the CLI command being one of the children. As an aside -ex specified against the bash header is useful as -e exits immediately if a failure is flagged in one of your commands and -x echos to screen the command executed, important when you want to know where your automation is up to.

These environment variables can also be set in ini files managed by the AWS CLI. The two following file templates describe profiles that can be referenced to be able to operate against multiple AWS accounts via different aliases.

~/.aws/credentials:

[profile-name]
aws_access_key_id = <KEYID>
aws_secret_access_key = <SECRETKEYID>

~/.aws/config:

[profile profile-name]
mfa_serial = <MFAARN>
output = text
region = ap-southeast-2
role_arn = <ROLE_ARN>
s3 =
    signature_version = s3v4
source_profile = <CREDSPROFILE>

These profiles can then be referenced with --profile profile-name in the CLI command or with the export AWS_PROFILE=profile-name environment variable.

Given this, here are a few CLI examples that utilise JMESPath queries to manipulate AWS data and allow you to work with it in the Bash environment.

CLI Examples

1. User ARN

It is very straight forward to get the ARN of the current user. This can be great for debugging who the CLI is executing as remotely and which AWS account ID it is in as it is part of the ARN.

aws iam get-user --query 'User.Arn'

Without the query argument the User key contains a map of key value pairs. This is selected by using User.Arn.

2. Get Keypairs

To get a sorted list of keypairs for a region the following command can be executed.

aws ec2 describe-key-pairs \
  --query 'KeyPairs[*].KeyName | sort(@)'

The asterisk paired with the dot notation indicates to return all instances in KeyPair but only the attribute KeyName. The pipe | allows the evaluation of the left hand query, taking the result and processing it with the query on the right. In this scenario the list of keypairs are given to the sort function and sorted alphabetically. The at @ symbol is a placeholder for the evaluated left hand input to the pipe.

3. Account Role ARNs

The following can be used to get a list of available roles in the given region. These might not all be assumable but can be determined by describing the individual roles.

# List account role ARNs
aws iam list-roles --query 'Roles[*].Arn'

Here the empty list operator is used to access each role ARN.

4. List instances by name regex

Here’s an example of the nested access of a JSON document. A substring of the instance name can be used to determine the instance IDs.

aws ec2 describe-instances \
  --filter 'Name=tag:Name,Values=instance-name-here' \
  --query 'Reservations[*].Instances[*].InstanceId'

Reservation has nested lists of documents containing Instances which each have their own instance ID. The resulting transformation is returned as a list.

5. Create a volume and get the ID

JMESPath queries can also apply to CLI commands that create objects. AWS CLI returns a JSON object on volume creation that can be queried for specific information as shown.

aws ec2 create-volume \
  --snapshot-id snap-id \
  --encrypted true \
  --availability-zone az \
  --query VolumeId

6. Get a specific output value from Cloudformation

Other than specifically accessing values exported in Cloudformation it is possible to query both the parameters and output values from a Cloudformation stack.

aws cloudformation describe-stacks --stack-name cfn_stack \
  --query "Stacks[0].Outputs[?OutputKey=='key'].OutputValue"

This query picks a key in the list of stack outputs and returns the resulting value.

7. Get available AMIs with a substring in their name

Given a label substring, return the AMI IDs in the state available that share the substring. Useful if you are searching for AMIs that fit a pattern.

aws ec2 describe-images \
  --filters Name=name,Values=*-name-contains-* Name=state,Values=available \
  --query 'Images[*].[ImageId,Name] | sort(@)'

As can be seen in this query it is possible to extract two attributes from a list of maps, by bracketing the attributes ([ImageId,Name]).

8. Get completed Cloudformation stacks by age

This is a great command that lets you search for successfully created Cloudformation stacks and return their creation time and stack name, sorted by creation date.

aws cloudformation list-stacks \
  --query 'StackSummaries[?StackStatus=="CREATE_COMPLETE"].[CreationTime,StackName] | sort(@)'

This demonstrates the combination of map queries and multiple attribute selection from 6 and 7.

9. Get the private IP of a set of instances based on a shared tag

Here is more nested queries, we can see the selection of an attribute paired with nested index selection.

aws ec2 describe-instances \
  --filter Name=tag:Name,Values=tag \
  --query 'Reservations[*].Instances[].[NetworkInterfaces[0].PrivateIpAddress]'

10. Availability zone subnets

It’s possible to list out subnets and their associated VPC ID and availability zones, good for getting an eye on general architecture in a region/account.

aws ec2 describe-subnets \
  --query 'Subnets[*].[VpcId,SubnetId,AvailabilityZone]'

11. Get an instances volume ID

Given an instances ID and the mount point of a volume, return the associated volume Id.

aws ec2 describe-volumes \
  --filters Name=attachment.instance-id,Values=instance-id \
  --query 'Volumes[*].Attachments[?Device==`/dev/sda`].VolumeId'

12. Get Instance Userdata

Seeing what is being executed on an instance at startup without getting into the instance logs can be very useful. The following command returns the instance user data.

aws ec2 describe-instance-attribute \
  --attribue userData \
  --instance-id instance-id \
  --query 'Userdata.Value' | base64 --decode

Once returned the userdata is base64 encoded and must be decoded with a separate command line tool available.

13. Get Account Security Groups as JSON

This one is showing how it is possible to influence the transformed structure from a JMESPath query.

aws ec2 describe-security-groups \
  --query SecurityGroups[*].{ID: GroupId}

Curly brackets are used here to indicate that the resulting output should be a list of maps, where the key of the map is ID.

14. Get the First Security Group ID alphabetically

The above query can be transformed to retrieve the first security group ID with the lowest value.

aws ec2 describe-security-groups \
  --query 'SecurityGroups[*].GroupId | sort(@) | [0]'

IDs are sorted and the first index is returned to the command line.

Conclusion

This concludes a demonstration of some of the queries that can be made against the AWS CLI. I hope you find this useful, the idea behind this post was to communicate this pattern to the world and hopefully get some feedback or improvements. Also it’s good to have something to reference back to when describing this method of scripting automation to other developers.


AWS  bash 
comments powered by Disqus