AWS IAM Roles in Windows Containers

Windows containers are one of the most exciting features in Windows Server 2016. All new applications should be designed with containers in mind. However, these are still early days and you might run into problems that are not yet solved with a quick Google search. We ran into a problem using instance credentials in Windows containers and wanted to share how we solved it.

Your servers should be using instance IAM roles rather than storing credentials on your instances. You will find, however, that Windows containers on AWS cannot contact the EC2 Metadata Service by default. To fix this, you will need to add a route to the NAT network’s gateway for the 169.254.169.254 address.

Everyone should be using IAM roles to provide credentials to applications running on servers. IAM roles allow you to provide credentials to your applications automatically rather than having to generate and save keys in your environment. Instance keys are rotated automatically, so you don’t have to change them yourself. Also, AWS tools will automatically use IAM profile credentials to connect to AWS services.

There is a hoop you have to jump through on Windows containers, though.  EC2 provides IAM role credentials to the client by way of the EC2 Metadata Service. It runs over HTTP at 169.254.169.254. That address is non routeable and by default this address is not available to Windows containers.

Your calls will fail when you try to use AWS APIs, AWS Powershell commandlets, or the AWS CLI inside of a Windows container using instance credentials.

Windows containers fail to connect to the EC2 metadata service by default

Windows containers fail to connect to the EC2 metadata service by default

If you start your container and view the routes, you will notice that you inherit the routes of your host operating system. On AWS instances, this means that the 169.254.169.254 address is routed to your local subnet router (i.e. 10.10.1.1 in the example below). Windows defaults to using a NAT network for containers, and we need a valid route for it.

EC2 Metadata service route points to the wrong gateway

EC2 Metadata service route points to the wrong gateway

The solution to that is pretty simple:

You must add a route on your Windows docker container to route 169.254.169.254 traffic to the container’s NAT router address.

Connection works after adding a route to the NAT gateway

Connection works after adding a route to the NAT gateway

In our case, we have a NAT network with a 172.17.0.1 gateway. We want to route all EC2 metadata calls to the host machine. We can do that by adding the route using the following command:

route -p add 169.254.169.254 172.17.0.1

You may have a different gateway in your case, so check first with ipconfig.

Add this command to your Dockerfile when building your application to allow your containers to use the AWS APIs with instance credentials.