CloudFormation Master Class

What is CloudFormation

Having >50 services, CloudFormation was brought in to help develops scaffold out the requires AWS stack.

Eg. I want a security group, two EC2 machines with it, two elastic IPs, an S3 bucket + a load balancer in front.

CloudFormation will create all of this in the right order with the exact config.

Benefits

CloudFormation vs Ansible / Terraform

First template

---
Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      AvailabilityZone: us-east-1a
      ImageId: ami-a4c7edb2
      InstanceType: t2.micro
copy

The stack instance can be created, updated or destroyed.

You cannot edit the stack itself later, you need to just re-update the stack by uploading a new file.

The stack itself can clean up instances after itself too.

YAML Intro

You can use YAML or JSON for writing it - but JSON is tough for it

Array support:

product:
    - test  : 1
      quantity: 2
    - test  : 2
      quantity: 4
copy

Creating a S3 Bucket

Googling for the type, you will get the in depth docs from AWS.

---
Resources: # always the start
    MyS3Bucket: # template name
        Type: "AWS::S3::Bucket"
        Properties:
            AccessControl: PublicRead
            BucketName: "www.site.com"
copy

Types of updates

Properties

On the properties under the docs, you can see info about the properties.

Deleting the stack

Just right click on the CloudFormation and delete the resources.

CloudFormation template options

You have a few template options:

These (if you manually do it) all show up on the "create stack" part of CloudFormation.

The template review also gives you an opportunity to estimate cost.

CloudFormation Designer

A visual aid to help build the CF Stack. Ensure the template is also well written.

You can drag and drop basically everything. Dropping it will give you options to selecting documentation etc.

It's great for dragging and dropping templates and giving information on that template as well.

Building Blocks

There are a number of building blocks for each template:

Template helpers

CloudFormation Parameters

Overview

What are they? The way to provide inputs to your AWS CloudFormation template.

They're important to know about it:

The major benefit: you won't have to re-upload a template to change its content.

Theory and hands on

Parameters can be controlled by all these settings:

This can be found in the 0-parameters-hands-on.yaml.

Again - check the docs.

To reference a parameter, you then go with Key: !Ref Reference.

If you have !Select for a CommaDelimitedList, you need to go Key: !Select [ArrayNumber, !Ref Reference].

How to reference a parameter

Resources

Resources are the core of your CloudFormation template. They represent the different AWS Components that will be created and configured.

They are declared and can be references by eachother. AWS figures out creation, updates, deletes etc.

There are over 224 types of resources.

They are identified using the form AWS::aws-product-name::data-type-name.

Reading the docs for an ES2 Instance

If you look at the docs, if comes up with both JSON and YAML docs.

Optional Attributes for Resources

FAQ for resources

Mappings

What are mappings? Fixed ariables within your CF Template. Great for dev vs prod, regions, AMI types etc.

Every mapping has top, middle and bottom.

Great to use when you know in advance:

They allow safer control over the template. Use parameters when the values are really user specific.

Fn::FindInMap - Accessing Mapping Values

Use Fn::FindInMap to return a named value from a specific key.

Example:

AWSTemplateFormatVersion: '2010-09-09'
Mappings:
  RegionMap:
    us-east-1:
      '32': 'ami-6411e20d'
      '64': 'ami-7a11e213'
Resources:
  myEC2Instance:
    Type: 'AWS::EC2::Instance'
    Properties:
      ImageId: !FindInMap [RegionMap, !Ref 'AWS::Region', 32]
      InstanceType: m1.small
copy

Mappings in practise

Parameters:
  EnvironmentName:
    Description: Environment Name
    Type: String
    AllowedValues: [development, production]
    ConstraintDescription: must be development or production

Mappings:
  AWSRegionArch2AMI:
    us-east-1:
      HVM64: ami-6869aa05
  EnvironmentToInstantType:
    development:
      instanceType: t2.micro
    production:
      instanceType: t2.small

Resources:
  EC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType:
        !FindInMap [
          EnvironmentToInstanceType,
          !Ref 'EnvironmentName',
          instanceType,
        ]
      ImageId: !FindInMap [AWSRegionArch2AMI, !Ref 'AWS::Region', HVM64]
copy

Concept: Pseudo Parameters

CloudFormation Outputs

What are they? They are optional values that we can import into other stacks.

You can also view the outputs in the AWS Console or in using the AWS CLI.

They're very useful for example if you define a network CloudFormation, and output the variables such as VPC ID and your Subnet IDs.

It's the best way to perform some collaboration cross stack. Let the expert handle their part and you handle yours.

Outputs Hands-On

Creating a SSH Security Group as part of one template. We can create an output that references that security group.

Outputs:
  <Logical ID>:
    Description: Information about the value
    Value: Value to return
    Export:
      Name: Value to export
copy

In 0-create-ssh-security-group.yaml

Resources:
  # here we define a SSH security group that will be used in the entire company
  MyCompanyWideSSHSecurityGroup:
    # http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-security-group.html
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable SSH access via port 22
      SecurityGroupIngress:
        # we have a lot of rules because it's a perfect security group
        # finance team network
        - CidrIp: 10.0.48.0/24
          FromPort: 22
          IpProtocol: tcp
          ToPort: 22
          # marketing team network
        - CidrIp: 10.0.112.0/24
          FromPort: 22
          IpProtocol: tcp
          ToPort: 22
          # application team support network
        - CidrIp: 10.0.176.0/24
          FromPort: 22
          IpProtocol: tcp
          ToPort: 22

Outputs:
  StackSSHSecurityGroup:
    Description: The SSH Security Group for our Company
    Value: !Ref MyCompanyWideSSHSecurityGroup
    Export:
      Name: SSHSecurityGroup
copy

It is important to note that for an output to be used anywhere, you need to define an export value.

Cross Stack Reference

We use Fn::ImportValue in a simple block:

Resources:
  MySecureInstance:
    # http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html
    Type: AWS::EC2::Instance
    Properties:
      AvailabilityZone: us-east-1a
      ImageId: ami-a4c7edb2
      InstanceType: t2.micro
      SecurityGroups:
        # we reference the output here, using the Fn::ImportValue function
        - !ImportValue SSHSecurityGroup
copy

Conditions

Conditionals are used to control the creation of resources or outputs based on a condition.

Conditions can be whatever you want them to be, but common ones are:

Each condition can reference another condition, parameter value or mapping.

How to define a condition?

Conditions:
  [Logical ID]: [Intrinsic function]
copy

Logical ID is for you to choose. It's how you name the condition.

The intrinsic function (logical) can be any of the following: - Fn::And - Fn::Equals - Fn::If - Fn::Not - Fn::Or

Conditional Hands On

AWSTemplateFormatVersion: '2010-09-09'
Mappings:
  RegionMap:
    us-east-1:
      AMI: 'ami-a4c7edb2'
      TestAz: 'us-east-1a'
    us-west-1:
      AMI: 'ami-6df1e514'
      TestAz: 'us-west-1a'
    us-west-2:
      AMI: 'ami-327f5352'
      TestAz: 'us-west-2a'
    eu-west-1:
      AMI: 'ami-d7b9a2b1'
      TestAz: 'eu-west-1a'
    sa-east-1:
      AMI: 'ami-87dab1eb'
      TestAz: 'sa-east-1a'
    ap-southeast-1:
      AMI: 'ami-77af2014'
      TestAz: 'ap-southeast-1a'
    ap-southeast-2:
      AMI: 'ami-10918173'
      TestAz: 'ap-southeast-2a'
    ap-northeast-1:
      AMI: 'ami-e21cc38c'
      TestAz: 'ap-northeast-1a'
Parameters:
  EnvType:
    Description: Environment type.
    Default: test
    Type: String
    AllowedValues:
      - prod
      - test
    ConstraintDescription: must specify prod or test.

Conditions:
  CreateProdResources: !Equals [!Ref EnvType, prod]

Resources:
  EC2Instance:
    Type: 'AWS::EC2::Instance'
    Properties:
      ImageId: !FindInMap [RegionMap, !Ref 'AWS::Region', AMI]
      InstanceType: t2.micro
      AvailabilityZone: !FindInMap [RegionMap, !Ref 'AWS::Region', TestAz]

  MountPoint:
    Type: 'AWS::EC2::VolumeAttachment'
    Condition: CreateProdResources
    Properties:
      InstanceId: !Ref EC2Instance
      VolumeId: !Ref NewVolume
      Device: /dev/sdh

  NewVolume:
    Type: 'AWS::EC2::Volume'
    Condition: CreateProdResources
    Properties:
      Size: 100
      AvailabilityZone: !GetAtt EC2Instance.AvailabilityZone

Outputs:
  VolumeId:
    Condition: CreateProdResources
    Value: !Ref NewVolume
copy

Note that conditions can not be applied to parameters.

Fn::GetAtt

Get an attribute attached to any resource that exists. To know the attributes, check the docs.

CF Metadata

This is any optional metadata section to include arbitrary YAML that provide details about the template or resource.

There are 3 metadata keys that have special meaning:

Describes how the resources are laid out in your template. This is automatically added by the AWS Designer. This helps the UI (x and y)

Define grouping and ordering of input parameters when they are displayed in the AWS Console.

Define configuration tasks for cfn-init. It's the most powerful usage of the metadata. This is very important and a lot to learn about it below.

AWS::CloudFormation::Designer hands on

This is automatically added for you but worth deleting for online sharing and usage. When dragging and dropping each resource you will see a huge set of metadata left there. The metadata can also be added to each resource.

AWS::CloudFormation::Interface hands on

Define grouping and ordering of input parameteres when they are displayed in the AWS Console. This is meant when users must input params manually.

You provide them with grouping, or sorting, that allow them to input parameters efficiently.

Example: Group all the EC2 related params together.

---
Parameters:
  KeyName:
    Description: Name of an existing EC2 key pair for SSH access to the EC2 instance.
    Type: AWS::EC2::KeyPair::KeyName
  InstanceType:
    Description: EC2 instance type.
    Type: String
    Default: t2.micro
    AllowedValues:
      - t2.micro
      - t2.small
      - t2.medium
      - m3.medium
      - m3.large
      - m3.xlarge
      - m3.2xlarge
  SSHLocation:
    Description: The IP address range that can SSH to the EC2 instance.
    Type: String
    MinLength: '9'
    MaxLength: '18'
    Default: 0.0.0.0/0
    AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
    ConstraintDescription: Must be a valid IP CIDR range of the form x.x.x.x/x.
  VPCID:
    Description: VPC to operate in
    Type: AWS::EC2::VPC::Id
  SubnetID:
    Description: Subnet ID
    Type: AWS::EC2::Subnet::Id
  SecurityGroupID:
    Description: Security Group
    Type: AWS::EC2::SecurityGroup::Id

Resources:
  MyEC2Instance:
    Type: 'AWS::EC2::Instance'
    Properties:
      AvailabilityZone: us-east-1a
      ImageId: ami-a4c7edb2
      InstanceType: !Ref InstanceType
      SecurityGroups:
        - !Ref SecurityGroupID
      SubnetID: !Ref SubnetID

Metadata:
  # This is the important part
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default: 'Network Configuration'
        Parameters:
          - VPCID
          - SubnetID
          - SecurityGroupID
      - Label:
          default: 'Amazon EC2 Configuration'
        Parameters:
          - InstanceType
          - KeyName
    ParameterLabels:
      VPCID:
        default: 'Which VPC should this be deployed to?'
copy

If you deploy a new stack using the above, you will see that the Parameters block will then drop you to questions about what configuration you are looking for.

CFN Init and EC2 User Data

EC2 User Data Overview

Many CF templates will be about provisioning computer resources in your AWS Cloud eg. EC2 instances, autoscaling.

Usually, you want to the instances to be self configured so that they can perform the job they are supposed to perform.

You can fully automate the EC2 fleet with CF init.

Example: an EC2 instance that has php and mysql installed on it.

We want a user-data script to get this up and going. From the EC2 management console, you can basically use the advanced section to add a /bin/bash section. This is already started to become more tedious than what we want.

How can we do this in CloufFormation?

The following script can use UserData to add the script:

Parameters:
  KeyName:
    Description: Name of an existing EC2 key pair for SSH access to the EC2 instance.
    Type: AWS::EC2::KeyPair::KeyName
  SSHLocation:
    Description: The IP address range that can be used to SSH to the EC2 instances
    Type: String
    MinLength: '9'
    MaxLength: '18'
    Default: 0.0.0.0/0
    AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
    ConstraintDescription: must be a valid IP CIDR range of the form x.x.x.x/x.

Resources:
  WebServer:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ami-a4c7edb2
      InstanceType: t2.micro
      KeyName: !Ref KeyName
      SecurityGroups:
        - !Ref WebServerSecurityGroup
      UserData:
        Fn::Base64: | # everything after will be kept as is
          #!/bin/bash
          yum update -y
          yum install -y httpd24 php56 mysql55-server php56-mysqlnd
          service httpd start
          chkconfig httpd on
          groupadd www
          usermod -a -G www ec2-user
          chown -R root:www /var/www
          chmod 2775 /var/www
          find /var/www -type d -exec chmod 2775 {} +
          find /var/www -type f -exec chmod 0664 {} +
          echo "<?php phpinfo(); ?>" > /var/www/html/phpinfo.php

  WebServerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: 'Enable HTTP access via port 80 + SSH access'
      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0
          FromPort: '80'
          IpProtocol: tcp
          ToPort: '80'
        - CidrIp: !Ref SSHLocation
          FromPort: '22'
          IpProtocol: tcp
          ToPort: '22'
copy

Now that we see the power of this, let's have a look at CF Init.

CloudFormation Init

What is the problem with EC2 user data? Well, what happens if we have a large configuration? What if we want to evolve the state without terminating it? How do we make it readable? How do we know or signal that our EC2 user-data script actually completed successfully?

Amazon creating CF helper scripts.

There are 4 python scripts that come directly with Amazon Linux AMI or can be installed using yum on non-Amazon Linux. They are:

The usual flow? cfn-init, then cfn-signal, then optionally cfn-hup.

AWS::CloudFormation::Init

A config contains the following and is executed in that order:

You can also have multiple configs and you can run them sequentially etc.

Packages

You can install packages from the following repositories:

Packages are processed in the following order: rpm, yum/apt, and then rubygems and python.

You can also specify a version if you want.

AWS::CloudFormation::Init:
  config:
    packages:
      rpm:
        epel: 'http://download....'
      yum:
        httpd: [] # means latest
        php: []
        wordpress: []
      rubygems:
        chef:
          - '0.10.2' # get this version
copy

Groups and Users

If you want to have multiple users and groups (with optional gid) in your ec2 instance, you can add groups and users to CF and metadata.

AWS::CloudFormation::Init:
  config:
    groups:
      groupeOne: {}
      groupTwo:
        gid: '45' #gid = group ID
    users:
      myUser:
        groups:
          - 'groupOne'
          - 'groupTwo'
        uid: '50'
        homeDir: '/tmp'
copy

In the larger example...

AWS::CloudFormation::Init:
  config:
    groups:
      apache: {} # assign any group ID
    users:
      'apache':
        groups:
          - 'apache' # user apache belongs to apache
copy

Sources

These are conveninence for a compressed archieve.

AWS::CloudFormation::Init:
  config:
    # where to unpack and from where
    sources:
      '/home/ec2-user/aws-cli': 'https://github.com/aws/aws-cli/tarball/master'
copy

Files

Files can be the most used section. Almost all the full power. It can be a specific URL or written inline for what you are doing.

Base example:

AWS::CloudFormation::Init:
  config:
    # where to unpack and from where
    files:
      /tmp/setup.mysql:
        content: !Sub |
          CREATE DATABASE ${DBName};
          CREATE USER '${DBUsername}'@'localhost' IDENTIFIED BY '${DBPassword}';
          GRANT ALL ON ${DBName}.* TO '${DBUsername}'@'localhost';
          FLUSH PRIVILEDGES;
        mode: '000644'
        owner: 'root'
        group: 'root'
copy

Full example:

Note: !Sub is a function used for making substitution. ie where you see ${AWS::StackName}.

AWS::CloudFormation::Init:
  config:
    # where to unpack and from where
    files:
      '/tmp/cwlogs/apacheaccess.conf':
        content: !Sub |
          [general]
          state_file= /var/awslogs/agent-state
          [/var/log/httpd/access_log]
          file = /var/log/httpd/access_log
          log_group_name = ${AWS::StackName}
          log_stream_name = {instance_id}/apache.log
          datetime_format = %d/%b/%Y:%H:%M:%S
        mode: '000400'
        owner: apache
        group: apache
      '/var/www/html/index.php':
        content: !Sub |
          <?php
          echo '<h1>AWS CloudFormation sample PHP application for ${AWS::StackName}</h1>';
          ?>
        mode: '000644'
        owner: apache
        group: apache
      '/etc/cfn/cfn-hup.conf':
        content: !Sub |
          [main]
          stack=${AWS::StackId}
          region=${AWS::Region}
        mode: '000400'
        owner: 'root'
        group: 'root'
      '/etc/cfn/hooks.d/cfn-auto-reloader.conf':
        content: !Sub |
          [cfn-auto-reloader-hook]
          triggers=post.update
          path=Resources.WebServerHost.Metadata.AWS::CloudFormation::Init
          action=/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource WebServerHost --region ${AWS::Region}
        mode: '000400'
        owner: 'root'
        group: 'root'
copy

Fn::Sub

(Or as !Sub) is used to substitute variables from a text. It's a very handy function that will allow you to fully customize your templates.

For example, you can combine !Sub with References or AWS Pseudo variables.

Must be in the form ${VarName}.

Forms:

# You can do this
!Sub
  - String
  - { Var1Name: Var1Value, Var2Name: Var2Value }

# or (more complicated and rarely seen)
!Sub String
copy

Commands

You can run commands one at a time in the alphabetical order.

You can set a directory from which that command is run, environment variables etc.

You can also provide a test to control whether the command is executed or not.

This should be a last resort. You can execute any of the scripts from the above files in this section.

Example: call the echo command only if the file doesn't exist

commands:
  test:
    command: 'echo "$MAGIC" > test.txt'
    env:
      MAGIC: 'I come from the environment!'
    cwd: '~'
    test: 'test ! -e ~/test.txt' # check file exists
    ignoreErrors: 'false' # fail if is doesn't work
copy

Services

AWS::CloudFormation::Init:
  config:
    services:
      sysvinit:
        httpd:
          enabled: 'true'
          ensureRunning: 'true'
        sendmail:
          enabled: 'false'
          ensureRunning: 'false'
copy

CFN Init and Signal

First, we use cfn-init to launch the config.

Then we use cfn-signal to tell when the config is complete, which will let CF know that the resource creation has been successful.

This has to be used in conjuction with a CreationPolicy.

This example means waiting a max of 5 minutes for the instance to come online and be self configured. If we don't hear back by cfn-signal by the, CF will fail and rollback.

CreationPolicy:
  ResourceSignal:
    Timeout: PT5M
copy

This is useful in case of a bad update.

cfn-hup

Example from the "files" declation:

'/etc/cfn/cfn-hup.conf':
  content: !Sub |
    [main]
    stack=${AWS::StackId}
    region=${AWS::Region}
  mode: '000400'
  owner: 'root'
  group: 'root'
'/etc/cfn/hooks.d/cfn-auto-reloader.conf':
  content: !Sub |
    [cfn-auto-reloader-hook]
    triggers=post.update
    path=Resources.WebServerHost.Metadata.AWS::CloudFormation::Init
    action=/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource WebServerHost --region ${AWS::Region}
  mode: '000400'
  owner: 'root'
copy

CFN Init Hands-On

After loading the CFN Init yaml file into CF, it will go through a series of different events.

You can under "status reason" if there is a success message sent back.

You need to practise your !Init skilles. It will be extremely handy for creating EC2 Instances or AutoScaling groups.

Remember logs for ec2-user data are in /var/log/cloud-init-output.log and logs for cfn-init are in /var/log/cfn-init.log, which is really helpful if commands don't complete like you want them to.

Advanced CF Concepts

Review of current standing:

The example with WordPress is what is shown in the course.

Using the AWS CLI

We can use the AWS CLI to create, update or delete CF templates.

Super conventient for when you start automating your deployments.

Once you've downloaded the AWS config, use aws configure --profile <profile_name> to configure a profile with the ID and Secret Access key.

To run a CF command, you can use something like the following aws cloudformation create-stack --stack-name example-cli-stack --template-body file://0-sample-template.yaml --parameters file://0-parameters.json --profile cf-course --region us-east-1

We can use the parameters.json file to set ParameterKey and ParameterValue for all the keys and values we are looking to share.

After running the command, what you get back is the StackId.

Using Troposphere (Python) to generate CloudFormation templates

Troposhere allows you to leverage Python write the templates.

This means you can start using types in your templates for safety.

You will also have valid CF and can dynamically generate CloudFormation.

This means you can also have very complex conditions.

The disadvantage is that the Python needs to generate the JSON for it to be.

DeletionPolicy

This policy can prevent resources from being deleted, or in some cases, back them up before the deletion. This will help prevent doing something really, really bad.

Deletion Policy can take up the following values:

Resources:
  myS3Bucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
copy

In the above example, it will create the S3 Bucket, you will see the bucket created.

Now if we delete that stack and the deletion policy is retain, you will still have that bucket there.

Custom Resources with AWS Lambda

Custom resources enable you to write custom provisioning logic in templates that AWS CloudFormation runs anytime you create, update (if you changed the custom resource) or delete stacks.

For example, you might want to include resources that aren't available as AWS CloudFormation resource types.

Check online for a walkthrough of custom resources.

Best practises to organize your CloudFormation templates

Cost estimate for templates

You can estimate the cost of a stack very easily.

For this, just upload the stack onto the AWS console, enter the params and click "cost".