Deploying your first AWS CloudFormation stack
As a machine learning engineer, deploying automated infrastructure-as-code is basically 80% of the job.
We are responsible for building scalable, maintainable, and reliable systems and for that we need to have our infrastructure defined in code.
This makes our system easy to deploy and more importantly easy to re-deploy if something goes wrong.
In this blog post I want to run you through a simple deployment of a single s3 bucket.
Although deploying a single s3 bucket is fairly trivial, getting comfortable with cloudformation is super important if you want to grow in your career.
It’s a bit like wax-on-wax off in the karate kid, you just have to do it a bunch of times to get good at it.
Please note that following along for this tutorial requires you to have a valid AWS profile with the right credentials, which I won’t get into
Setup
Install the aws-cli
pip install aws-cli
Verify correct installation
$ aws --version
aws-cli/2.2.26
Adding a main stack
Now we are going to add our main stack and in that stack we will add a substack.
Make a new file at the following location infra/main.yml
# infra/main.yml
AWSTemplateFormatVersion: '2010-09-09'
Description: A basic cloudformation template
Parameters:
Environment:
Type: String
Resources:
ModelData:
Type: AWS::CloudFormation::Stack
Properties:
Parameters:
ServiceName: !Ref AWS::StackName
Environment: !Ref Environment
TemplateURL: model-data.substack.yml
This is our main stack and this is the one we will deploy. We see the Parameters which are parameters that we can pass and we see Resources which are the AWS resources we want to deploy.
In this case the resource that we want to deploy is another AWS::CloudFormation::Stack, a substack.
Note that the TemplateURL points to model-data.substack.yml, this is the file we are going to make now.
Make a new file model-data.substack.yml
# infra/model-data.substack.yml
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
ServiceName:
Type: String
Environment:
Type: String
Resources:
Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub "${ServiceName}.${AWS::AccountId}.${AWS::Region}.${Environment}.org"
OwnershipControls:
Rules:
- ObjectOwnership: ObjectWriter
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
Again, we have some Parameters and Resources and for now you can Ignore the settings on the s3 bucket.
Now we are ready to deploy.
If we did not have any substacks we could deploy directly.
$ aws cloudformation deploy ...
But because we have substacks in our main stack, we first need to package this deployment.
The reason for this is that when we try to deploy the main stack, it starts looking for model-data.substack.yml, but this doesn’t exist on AWS.
We first need to turn all local references into references that are also understandable in the cloud, this is what the packaging step does.
Let’s do that.
aws cloudformation package
--template-file infra/main.yml
--s3-bucket tmp-bucket
--s3-prefix tmp-prefix
--output-template-file packaged-template.json
--use-json
--force-upload
--profile=profile
--region=eu-west-1
Uploading to tmp/115d6216e2ba888dfb9ea95c58b7e420.template 685 / 685.0 (100.00%)
Successfully packaged artifacts and wrote output template to file packaged-template.json.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file C:\Users\JanMeppe\Documents\example-cf-deploy\packaged-template.json --stack-name <YOUR STACK NAME>
Taking a look in the template we see this
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Example cf deploy",
"Parameters": {
"Environment": {
"Type": "String"
}
},
"Resources": {
"ModelData": {
"Type": "AWS::CloudFormation::Stack",
"Properties": {
"Parameters": {
"ServiceName": {
"Ref": "AWS::StackName"
},
"Environment": {
"Ref": "Environment"
}
},
"TemplateURL": "https://s3.eu-west-1.amazonaws.com/tmp-bucket/tmp-prefix/115d6216e2ba888dfb9ea95c58b7e420.template"
}
}
}
}
See how this TemplateURL no longer references to a file in our local storage, but to some file on s3?
Now we can deploy this in the cloud and TemplateURL will be read from that s3 location.
aws cloudformation deploy
--template-file packaged-template.json
--stack-name tmp-jan-foo
--parameter-overrides Environment=master
--region=eu-west-1
--profile=profile
Resulting in
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - tmp-jan-foo
Inspect the stack
aws cloudformation describe-stack-events
--stack-name tmp-jan-foo
--region=eu-west-1
--profile=profile
Results in
{
"StackEvents": [
{
"StackId": "arn:aws:cloudformation:eu-west-1:12345:stack/tmp-jan-foo/...",
"EventId": "...",
"StackName": "tmp-jan-foo",
"LogicalResourceId": "tmp-jan-foo",
"PhysicalResourceId": "arn:aws:cloudformation:eu-west-1:12345:stack/tmp-jan-foo/...",
"ResourceType": "AWS::CloudFormation::Stack",
"Timestamp": "2024-05-10T11:45:49.820000+00:00",
"ResourceStatus": "CREATE_COMPLETE"
},
...
That’s it! Everything works.
But we can also see it in the cloud console!
So cool! Your first cloudformation stack!
And that’s it! That’s your first deployment! You are officially now a machine learning engineer.
Now, don’t forget to delete your stack when you are done with it!
Usually, in production systems we have certain protections in place that forbid us to manually remove these stacks, so that we can never accidentally delete production.
aws cloudformation delete-stack
--stack-name tmp-jan-foo
--region=eu-west-1
--profile=profile
The fact that we need to upload our other templates to s3 first also means that we need a deployment bucket where we store our deployment artifacts.
Comments