Run scripts on CloudFormation stack creation with Custom Resources.

You may find this short article interesting if you know the basics of CloudFormation — such as the definition of resources and parameters — and are keen to explore further capabilities of this powerful AWS service.

In particular, I will talk about using Custom Resources in order to apply custom programming logic during stack creation. For example, you may want to send notifications or run commands against the newly created resources. This can be achieved easily with Custom Resources.

Abbreviations

• CR — CustomResource
• CFN — CloudFormation

What and why of custom resources

As per the official AWS documentation, CustomResource is a resource type that can be used to write custom provisioning logic. An example — they allow you to run scripts or issue events during CFN template creation.

Some CFN experts may prefer CommandRunner instead. I tend to use CustomResources as they allow me to write logic in my language of choice and not just bash. Besides, CustomResource works out of the box and does not require us to install any plugins.

How do custom resources work?

CustomResource is defined in CFN templates just as any other AWS resource. One peculiarity is its type — you are free to choose one of the two options:

Two ways to define type for CustomResource

Custom Resource refers to either SNS topic or a Lambda function as handler for the custom logic. Handler is identified by service token, which is nothing more than an Arn of an existing AWS resource. So far only SNS and Lambda are supported as handlers.

When CFN creates or updates a Custom Resource —it will send a request to the handler and wait for response. This is the most precarious part, as if you forget to send back the response — the stack creation will be left in CREATE_IN_PROGRESS until it timeouts. Default timeout for custom resources is 1 hour, so better be sure to respond back to CFN.

The body of the request CFN sends to the handler includes a presigned URL. The handler will issue a put request to this URL when it is done running its logic. That is how CFN will know that the custom resource is created and it can move on.

It is your task as the developer of the custom resource to ensure that by the end of your custom function you will put the response object using that URL.

This design pattern is quite often encountered across AWS services.

If you’ve ever integrated Lambda into a CodePipeline — this concept may sound familiar to you. A lambda running as a build step would also need to signal either success or failure back to CodePipeline.

Same applies when you run cfn-init scripts on EC2 instances created with CFN and wait for cfn-signal to proceed with template creation.

Example using Custom Resource to run a script on stack creation

Let us go through a little hands-on lab.

We have an S3 bucket with an object in it. This object is a bash script. Its content is irrelevant. Its key is demo/script.sh. The bucket and its objects are encrypted using a KMS key managed by AWS.

We want to update our infrastructure as follows:

1. Create a new S3 bucket.
2. After the bucket is created — copy the script from the existing bucket to the new one.

We will accomplish this task by means of a single CFN template.

Let’s first define the new bucket. It will be very basic — no lifecycle management, no specific encryption — nada.

We will use Lambda in order to perform the copy operation. Its runtime will be python 3.6, but could be any language and version you like, as long as it is supported by Lambda.

Let us first define the lambda execution role which will allow it to leverage all the AWS resources it needs.

That’s quite some code. Remember that our function will need permissions to do the following:

1. Access CW logs.
2. Read and list objects in the existing bucket.
3. Decrypt objects in the existing bucket using KMS key.
4. Write into the new bucket.

Finally, the definition of the Lambda itself. Since we have little code — I will just write it inline, however ugly it looks.

You may be wondering what this cfnresponse library is all about. It’s but a tiny python util provided for us by the AWS team.

Under the hood it sends a PUT request to the S3 bucket using a pre-signed URL it got from CloudFormation — just like I’ve told you previously.

The body of the request includes the status of your custom resource — either SUCCESS or FAILURE, as well as any data you provide plus some meta information.

You can have a look at the source of this module in the public AWS documentation.

See also the request and response objects reference for CFN custom resources.

Another important moment is how we get the destination_bucket_name. Hold on. We will get to it in a moment.

The final part in our CFN design is the custom resource itself. Its configuration is fairly simple. We need to specify the ARN of the handler — Lambda in our case — as service token. At the same level we put all the necessary arguments, such as destination_bucket_name. Those arguments can be accessed inside the Lambda handler via event[‘ResourceProperties’], as you may see above.

Below is the assembled template once again.

Now that our template is ready, let’s go on and upload it to CFN in our account.

I am logged in as Admin user, so CFN will use my role to deploy resources. You may need to specify a different role in the next screen if your case is different.

Let us launch the stack creation and observe the events tab.

At some point the custom resource gets created — first the execution role, then lambda function, then finally — the custom resource itself. The green letters and complete status look promising, so let’s go to the new bucket and see if the script was successfully copied there.

Indeed, so it is!

Thanks for your attention! A little remark in the end. As you may know, CFN does not update resources unless any of their properties changed values since the last update.

If you want to repeat the copy operation on each and every stack update — you may simply include some random redundant property in your custom resource and change it before every stack deployment. You can use CFN parameters to change its value.

You don’t have to use this property in your lambda code, but it will ensure the custom resource is updated on every stack update.

I hope you’ve found this small lab useful. If you followed hands-on and actually created the stack— don’t forget to delete it in order to prevent any negative impact on your wallet. Enjoy your day and happy coding!