KMS Encryption of Autoscaled Instance Volumes -

A lot of effort is put into securing the file system of your servers, and security can be brought into question if volumes are not being encrypted. This can be a challenge with autoscaled instance groups and encrypted AMIs. Whenever I created an autoscaled application, I would apply the necessary permissions to the launch configuration through a role¹, only to have instances go into a starting stage and then shut down.

This was part of a Cloudformation template which timed out in creation and eventually failed to which I then had to diagnose the cause. In this article I will explain how to design an autoscaling stack that can work with encrypted volume AMIs and provide Cloudformation resource snippets to give an idea of the implementation.

Context

This works for autoscaled instances in an autoscaling group with a launch configuration referencing an encrypted AMI. The AMI was encrypted using a non-default customer master key (CMK) using the key material provided by KMS. This was done with the copy-image CLI command, specifying the ID of the CMK. On this CMK I created resource whitelists so that agents can perform actions with it through key policies. This facilitates visibility of permissions against the key, improving security.

Solution

Firstly, create a role that can be associated against the entity creating the stack that has encrypted volumes. In this case I have created a principal of ec2.amazonaws.com so that the privileges can be assumed by an EC2 instance.

InstanceRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - ec2.amazonaws.com
          Action:
            - sts:AssumeRole
    Path: '/'

Figure 1: Role resource

Next I create a policy that is mapped against the role that has some base abilities the entity creating the stack with encrypted volumes will need, in this case I have added full access to S3 and to CloudFormation. You can also reduce the scope of the resource to specific ARNs to improve security and limit the instances resource access.

RolePolicies:
  Type: AWS::IAM::Policy
  Properties:
    PolicyName: Instance Abilities
    PolicyDocument:
      Statement:
        - Action:
            - s3:*
            - cloudformation:*
            ...
          Effect: Allow
          Resource:
            - '*'
    Roles:
      - !Ref InstanceRole

Figure 2: Instance policy

Finally, map the role against an instance profile such that it can be attached to an EC2 instance. With permissions handled create the Customer Master Key.

InstanceProfile:
  Type: AWS::IAM::InstanceProfile
  Properties:
    Path: '/'
    Roles:
      - !Ref InstanceRole

Figure 3: Instance profile

Here is where the meat of the solution is. Create the CMK for volume encryption with a key policy that allows the EC2 instance access to the key. Access is allowed by referencing InstanceRoleArn as the policy principle and then defining all required KMS actions.

Next, you need to allow the account root access to the key and relevant actions to decrypt/encrypt an encrypted volume. This is a very open policy so it is necessary to further constrain it by adding the conditions kms:ViaService to limit access to the EC2 service and aws:userid to limit user access to only the root user and no other account user². This will be what allows autoscaling actions to operate.

KeyAlias:
  Type: AWS::KMS::Alias
  Properties:
    AliasName: !Sub 'alias/${KeyAliasName}'
    TargetKeyId: !Ref KMSKey
KMSKey:
  Type: AWS::KMS::Key
  Properties:
    EnableKeyRotation: true
    KeyPolicy:
      Version: 2012-10-17
      Id: kms-key
      Statement:
        - Sid: Allow use of the key by the instance
          Effect: Allow
          Principal:
            AWS: !ImportValue InstanceRoleArn
          Action:
            - kms:CreateGrant
            ...
          Resource: '*'
        - Sid: Allow use of the key by the root user for autoscaling purposes
          Effect: Allow
          Principal:
            AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
          Action:
            - kms:CreateGrant
            - kms:Decrypt
            - kms:DescribeKey
            - kms:Encrypt
            - kms:GenerateDataKey*
            ...
          Resource: '*'
          Condition:
            StringEquals:
              'kms:ViaService': !Sub 'ec2.${AWS::Region}.amazonaws.com'
              'aws:userid': !Ref AWS::AccountId

Figure 4: Customer Master Key

Attaching a key policy to a CMK is a double-edged sword, in that you can essentially whitelist which entities can interact with the key, improving confidence in security. The downside of key policies is that it can be difficult to debug services which are trying to work with resources that are encrypted using the referenced key. It is up to the architect to determine the balance between ease of use and security.

Later you will be able to reference this created key when encrypting the AMI underlying the autoscaling group. This can be done with the copy image CLI command:

aws ec2 copy-image --source-region <region-here> \
    --name "Encrypted AMI name here" \
    --source-image-id <base-ami-here> \
    --encrypted --kms-key-id <keyid>

With a newly encrypted AMI, the ID may be referenced in the launch configuration resource for your autoscaling group; this can be done in the following manner:

...
InstanceLaunchConfig:
  Type: AWS::AutoScaling::LaunchConfiguration
  Properties:
    IamInstanceProfile:
      Fn::ImportValue:
        !Sub '${RoleStack}-ProfileArn'
    ImageId: !Ref InstanceAMI
    InstanceType: !Ref InstanceType
...

Figure 5: Autoscaled instance launch configuration

Conclusion

That’s it, so many applications developed in AWS are built in self-contained stacks with autoscaling capabilities and security is also very frequently important. It’s good to not have to compromise on these, and to be able to use keys that the owner controls.

Context

Solution

Conclusion

See Also