We’ve been enjoying the use of AWS CloudFormation. While the templates can be a bit of a bear, the end result is always consistent. (That said, I think that Terraform has some real promise).
One thing we do is to lock our templates to specific AMIs, like this:
"AWSRegion2UbuntuAMI" : {
"us-east-1" : { "id" : "ami-7fe7fe16" },
"us-west-1" : { "id" : "ami-584d751d" },
"us-west-2" : { "id" : "ami-ecc9a3dc" },
"eu-west-1" : { "id" : "ami-aa56a1dd" },
"sa-east-1" : { "id" : "ami-d55bfbc8" },
"ap-southeast-1" : { "id" : "ami-bc7325ee" },
"ap-southeast-2" : { "id" : "ami-e577e9df" },
"ap-northeast-1" : { "id" : "ami-f72e45f6" }
}
That’s great, because we always get the exact same build based on that image and we don’t introduce unexpected changes. For those of you who know their AMI IDs very well, you will realize that this is actually for an older version of Ubuntu.
Sometimes, however, it makes sense to bring the AMIs up to a new version and that means having to find all of the new AMI IDs.
Here is a potential approach using the . I’m going to assume you either have it installed already or run on one of the platforms there the installation instructions work. (Side note: if you are on an Ubuntu box I recommend installed the version via pip since it works as advertised, while the version in the Ubuntu repo has some odd issues).
Using the awscli it’s possible to list the images. Since I’m interested in Ubuntu images I search for Canonical’s ID or 099720109477 and also apply some filters to show me only the 64 bit machines with an ebs root device:
aws ec2 describe-images --owners 099720109477 \
--filters Name=architecture,Values=x86_64 \
Name=root-device-type,Values=ebs
That produces a very long dump of JSON (which I truncated):
{
"Images": [
{
"VirtualizationType": "paravirtual",
"Name": "ubuntu/images-testing/ebs-ssd/ubuntu-trusty-daily-amd64-server-20141007",
"Hypervisor": "xen",
"ImageId": "ami-001fad68",
"RootDeviceType": "ebs",
"State": "available",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"Ebs": {
"DeleteOnTermination": true,
"SnapshotId": "snap-bde4611a",
"VolumeSize": 8,
"VolumeType": "gp2",
"Encrypted": false
}
},
{
"DeviceName": "/dev/sdb",
"VirtualName": "ephemeral0"
}
],
"Architecture": "x86_64",
"ImageLocation": "099720109477/ubuntu/images-testing/ebs-ssd/ubuntu-trusty-daily-amd64-server-20141007",
"KernelId": "aki-919dcaf8",
"OwnerId": "099720109477",
"RootDeviceName": "/dev/sda1",
"Public": true,
"ImageType": "machine"
},
......
{
"VirtualizationType": "hvm",
"Name": "ubuntu/images/hvm/ubuntu-quantal-12.10-amd64-server-20140302",
"Hypervisor": "xen",
"ImageId": "ami-ff4e4396",
"State": "available",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"Ebs": {
"DeleteOnTermination": true,
"SnapshotId": "snap-8dbadf4a",
"VolumeSize": 8,
"VolumeType": "standard",
"Encrypted": false
}
},
{
"DeviceName": "/dev/sdb",
"VirtualName": "ephemeral0"
},
{
"DeviceName": "/dev/sdc",
"VirtualName": "ephemeral1"
}
],
"Architecture": "x86_64",
"ImageLocation": "099720109477/ubuntu/images/hvm/ubuntu-quantal-12.10-amd64-server-20140302",
"RootDeviceType": "ebs",
"OwnerId": "099720109477",
"RootDeviceName": "/dev/sda1",
"Public": true,
"ImageType": "machine"
}
]
}
That output is pretty thorough and good for digging through things, but for my purposes it’s too much and lists lots of things I don’t need.
To drill in on the salient input a little more I use the excellent jq command-line JSON processor and pull out the things I want and also grep for the specific release:
aws ec2 describe-images --owners 099720109477 \
--filters Name=architecture,Values=x86_64 \
Name=root-device-type,Values=ebs \
| jq -r '.Images[] | .Name + " " + .ImageId' \
| grep 'trusty-14.04'
The result is something I can understand a little better:
ubuntu/images/ebs-io1/ubuntu-trusty-14.04-amd64-server-20140829 ami-00389d68 ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20140926 ami-0070c468 ubuntu/images/ebs/ubuntu-trusty-14.04-amd64-server-20140416.1 ami-018c9568 ... ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20140923 ami-80fb51e8 ubuntu/images/ebs-io1/ubuntu-trusty-14.04-amd64-server-20140927 ami-84aa1cec ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20140607.1 ami-864d84ee ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20140724 ami-8827efe0 ubuntu/images/hvm/ubuntu-trusty-14.04-amd64-server-20140923 ami-8afb51e2 ubuntu/images/ebs/ubuntu-trusty-14.04-amd64-server-20140927 ami-8caa1ce4 ubuntu/images/hvm-io1/ubuntu-trusty-14.04-amd64-server-20140923 ami-8efb51e6 ubuntu/images/ebs-ssd/ubuntu-trusty-14.04-amd64-server-20140927 ami-98aa1cf0 ubuntu/images/hvm/ubuntu-trusty-14.04-amd64-server-20140927 ami-9aaa1cf2 ubuntu/images/hvm-io1/ubuntu-trusty-14.04-amd64-server-20140927 ami-9caa1cf4 ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20140927 ami-9eaa1cf6 ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20140816 ami-a0ff23c8 ubuntu/images/hvm-io1/ubuntu-trusty-14.04-amd64-server-20140607.1 ami-a28346ca ubuntu/images/ebs/ubuntu-trusty-14.04-amd64-server-20140724 ami-a427efcc ... ubuntu/images/ebs/ubuntu-trusty-14.04-amd64-server-20140813 ami-fc4d9f94 ubuntu/images/hvm-io1/ubuntu-trusty-14.04-amd64-server-20140924 ami-fe338696
After a little more investigation I see that the latest version can be identified based on the datastamp, in this case 20140927. I’ve seen some other ways things are named, but in this case the datastamp works well enough and I can look for ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20140927 in each region for the AMI IDs.
for x in us-east-1 us-west-2 us-west-1 eu-west-1 ap-southeast-1 ap-southeast-2 ap-northeast-1 sa-east-1; do
echo -n "$x "
aws --region ${x} ec2 describe-images --owners 099720109477 --filters Name=architecture,Values=x86_64 \
Name=root-device-type,Values=ebs \
Name=name,Values='ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20140927' \
| jq -r '.Images[] | .Name + " " + .ImageId'
done
The result is a nice tidy list with the AMI ID for each region:
us-east-1 ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20140927 ami-9eaa1cf6 us-west-2 ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20140927 ami-3d50120d us-west-1 ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20140927 ami-076e6542 eu-west-1 ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20140927 ami-f0b11187 ap-southeast-1 ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20140927 ami-d6e7c084 ap-southeast-2 ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20140927 ami-1711732d ap-northeast-1 ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20140927 ami-e74b60e6 sa-east-1 ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20140927 ami-69d26774
Now, to make this pastable into the the CloudFormation template I run that output through some more shell processing:
cut -f1,3 -d' ' | sed 's/^\(.*\) \(.*\)$/"\1": { "id": "\2" },/'
and end up with
"us-east-1": { "id": "ami-9eaa1cf6" },
"us-west-2": { "id": "ami-3d50120d" },
"us-west-1": { "id": "ami-076e6542" },
"eu-west-1": { "id": "ami-f0b11187" },
"ap-southeast-1": { "id": "ami-d6e7c084" },
"ap-southeast-2": { "id": "ami-1711732d" },
"ap-northeast-1": { "id": "ami-e74b60e6" },
"sa-east-1": { "id": "ami-69d26774" },
I can now paste that into the template and remove the final comma.
Voilà, the new stack will now run with the latest AMIs and can be subjected to testing.
\@matthias
