AWS Config and Resource Tags

  Posted On September 18, 2016  |    6 minutes

An infrastructure piece I’ve been working on over the last fortnight is enforcing tags on resources in the AWS account environment. If you’ve worked in an Amazon account that hosts multiple environments with different resource types and jobs you will know it can quickly become difficult to tell if some resource is necessary or not, it may have been created by another team member and no one remembers if it’s used anymore.

Amazon provides the tagging functionality for most of the available resources; this makes it easier to search in an environment and get some context of what it’s used for, or if you’re sharing the account with others what resources are billable to who. Keeping track of tag presence on resources requires rigor on the part of the account tenants, it’s better to automate this to remove the human element. To assist this Amazon have released Config Rules and to my luck have recently made them available in Sydney. Config Rules allow you to track when changes to the configuration of resources have been made and perform automated responses. The Config service also has a dashboard for you to easily see which are compliant and which aren’t.

In this article I will describe how to create a simple config rule name tag format checker that can tell you if the instances in your environment are labelled correctly. The config rule will be backed using a Lambda function to perform a regular expression check. I’ll also use the Boto3 module of Python to notify the user of non-compliance on a resource update. Each necessary resource will be outlined using Cloudformation template resource YAML.

There are a few AWS resources that the config rule is reliant on. Before we jump to it I’ll describe what you need and the gotchas I came across along the way.

Permissions, Permissions, Permissions

The Config rule and it’s dependencies need permission to be able to operate. I’ve added the template resources that need to be referenced to run the backing Lambda function and for the rule to be able to invoke the function.

Here is the managed policy where I’ve defined the actions allowed to the config rule to undertake via the Lambda function. To operate Lambda needs to be able to send logs to CloudWatch logs. These logs will help you to catch any issues with the logic of your Lambda function. To be able to work with Config and make notifications through SNS the Lambda function also needs permissions to put Config changes and SNS notifications.

Managed Policy Resource:

TagComplianceLambdaManagedPolicy:
  Type: AWS::IAM::ManagedPolicy
  Properties:
    Description: Provides config service and log access to lambda
    PolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: Allow
          Action: logs:CreateLogGroup
          Resource: !Sub |
            arn:aws:logs:${AWS::Region}:${AWS::AccountId}:*
        - Effect: Allow
          Action:
          - logs:CreateLogStream
          - logs:PutLogEvents
          Resource: !Sub |
            arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/tag-compliance:*
        - Effect: Allow
          Action:
            - s3:GetObject
          Resource: arn:aws:s3:::*/AWSLogs/*/Config/*
        - Effect: Allow
          Action:
            - sns:Publish
          Resource:
            Ref: NotificationSNSTopicARN
        - Effect: Allow
          Action:
            - config:Put*
            - config:Get*
            - config:List*
            - config:Describe*
          Resource: !Sub |
            arn:aws:config:${AWS::Region}:${AWS::AccountId}:config-rule/*

Actions allowable by Lambda are defined by the IAM role. Here I reference the previously created managed policy and add the statement that the role is able to be assumed by the Lambda service.

IAM Role Resource:

LambdaRole:
  Type: "AWS::IAM::Role"
  Properties:
    ManagedPolicyArns:
      - Ref: TagComplianceLambdaManagedPolicy
    AssumeRolePolicyDocument:
      Statement:
        - Action: "sts:AssumeRole"
          Effect: Allow
          Principal:
            Service: lambda.amazonaws.com

For the config rule to use the Lambda function it must be assigned permission. The following template resource snippet will allow the rule to invoke the created function.

Lambda Permission Resource:

ConfigPermissionToCallLambda:
  Type: AWS::Lambda::Permission
  DependsOn: LambdaFunctionTagCompliance
  Properties:
    FunctionName: tag-compliance
    Action: lambda:InvokeFunction
    Principal: config.amazonaws.com

Working with Lambda

To respond to the Config service alerting of changes to resource configuration it is backed by a Lambda function that takes as input the configuration of the resource. To check the name tag of modified EC2 instances I’ve added the following Python code. As you can see the main parts involve taking in the configuration_item and interpreting it for compliance and then constructing an evaluation response to send to the config service through put_evaluation. configuration_item contains all configuration for the resource Config has flagged as changed. This is where you can extract the instance tags. This code needs to be zipped and put into a bucket accessible by the Lambda function.

tag_compliance.py:

import json
import re

import boto3


ROLES = ('web', 'app', 'jumpbox', 'proxy', 'firewall', )
ENVIRONMENTS = ('shared', 'build', 'nonprod', 'prod', )


def handler(event, context):
    config_service = boto3.client('config')
    sns_service = boto3.client('sns')
    ec2_regex = re.compile('^[a-z]+-[a-z0-9]+-[a-z-0-9]+[-a-z]*')
    config_item = json.loads(event['invokingEvent'])['configurationItem']
    topic_arn = json.loads(event['ruleParameters'])['notification_topic_arn']
    resource_type = config_item['resourceType']

    if config_item['configurationItemStatus'] == 'ResourceDeleted' or \
       resource_type != 'AWS::EC2::Instance':
        return

    # the resource evaluation map, tells config the state of the resource
    # at the current point in time
    evaluation = {
        'ComplianceResourceType': config_item['resourceType'],
        'ComplianceResourceId': config_item['resourceId'],
        'ComplianceType': 'NON_COMPLIANT',
        'OrderingTimestamp': config_item['configurationItemCaptureTime']
    }

    resource_id = config_item['resourceId']
    if 'Name' not in config_item['tags']:
        sns_message = 'This is a notification that the resource with ID' + \
            resource_id + 'has no Name tag.' + \
            'Please review and update the resource.\n'
    else:
        resource_name = config_item['tags']['Name']
        sns_message = 'ID: %s\nName Tag: %s\nRequired Format: %s\n' % \
            (resource_id, resource_name, ec2_regex) + \
            'The above resource does not follow the correct tag format,' + \
            'please review and update the resource.'
        # Check the name meets the regex rules for that resource
        match_found = bool(ec2_regex.match(resource_name))
        resource_sections = resource_name.split('-')
        if match_found and len(resource_sections) > 2 and \
           resource_sections[0] in ROLES and \
           resource_sections[2] in ENVIRONMENTS:
            evaluation['ComplianceType'] = 'COMPLIANT'

    if evaluation['ComplianceType'] == 'NON_COMPLIANT':
        resource_id = config_item['resourceId']
        sns_subject = 'Notification of non-compliant resource,' + \
            'Type: %s, ID: %s' % (resource_type.split('::')[-1], resource_id)
        sns_service.publish(
            TopicArn=topic_arn, Message=sns_message, Subject=sns_subject
        )

    config_service.put_evaluations(
        Evaluations=[evaluation], ResultToken=event['resultToken']
    )
    return evaluation['ComplianceType']

This uploaded code package can now be referenced in the Cloudformation template.

Lambda Function

Having defined the Lambda handler I now can create a function that references it. In the template will be the name of the file, plus .handler, in our example tag_compliance.handler. Be sure the zip tool does not also zip any of the underlying folders, otherwise the handler will not be found. Also referenced in this resource is the IAM role previously created. To finish reference the S3 bucket and key created earlier.

Lambda Function Resource:

LambdaFunctionTagCompliance:
  DependsOn: LambdaRole
  Type: AWS::Lambda::Function
  Properties:
    FunctionName: tag-compliance
    Handler: tag_compliance.handler
    Role:
      Fn::GetAtt:
        - LambdaRole
        - Arn
    Code:
      S3Bucket:
        Ref: S3Bucket
      S3Key: tag_compliance.py.zip
    Runtime: python2.7

Now that the Lambda function is taken care of we can now begin on the Config rule.

Config Rule

Finally the Config rule can be created by referencing the Lambda permission created earlier. The template resource needs to be given:

  • The name of the config rule,
  • The scope, i.e. what resources the rule applies to, in this case EC2 instances,
  • The source of configuration data. We pass the CUSTOM_LAMBDA as the owned to indicate the rule being backed by it and pass the ARN of the Lambda function as the SourceIdentifier and indicate in the source detail that the configuration passed is from ConfigurationItemChangeNotification.
  • The ARN of the notification SNS Topic as an input parameter passed into the handler function.

Config Rule Resource:

ConfigRuleTagCompliance:
  Type: AWS::Config::ConfigRule
  DependsOn: ConfigPermissionToCallLambda
  Properties:
    ConfigRuleName: ConfigRuleTagCompliance
    Scope:
      ComplianceResourceTypes:
        - AWS::EC2::Instance
    Source:
      Owner: CUSTOM_LAMBDA
      SourceIdentifier:
        Fn::GetAtt:
          - LambdaFunctionTagCompliance
          - Arn
      SourceDetails:
        - EventSource: aws.config
          MessageType: ConfigurationItemChangeNotification
    InputParameters:
      notification_topic_arn:
        Ref: NotificationSNSTopicARN

Conclusion

To summarise this is a great way to keep track of your AWS account resources and put some context to what has been allocated. There are a few pitfalls along the way but hopefully I’ve illustrated the main points. If your new to Lambda this project is also a great way to familiarise yourself with the service.

Benefits

  • A native solution for tag compliance in AWS,
  • Fully automated,
  • Only needs to run when a change is made.

Limitations

  • Limited selection of AWS resources at the moment (I’m hanging out for AMIs and Snapshots)

If you’re interested in trying this the above example files can be accessed from my projects github repository.

Update (28/02/2017): This article was originally in JSON, updated it for YAML as it is clearer to see what is happening, also applied PEP8 formatting to the Python Lambda handler.

Note

As with anything AWS related your account is attached to a credit card; if you are madly thrashing instances in and out and making configuration changes there will be a cost to using the services.

AWS  cloudformation  config  lambda  python 
comments powered by Disqus