Build time secrets with docker containers

Check out first part of this blog post if you are looking to provide secrets during container runtime.
Part 1: Runtime Secrets with Docker Containers

So far we have talked about run time secrets. But you also need secrets during the build time of docker images. For ex: During your build process, you need:

• To hit some private repo to pull dependency.
• Need private keys for remote SSH connection to pull stuff.
• Build environment is behind PROXY than deploy envt.
• Tons of other use cases.

There isn’t a single solution that addresses all the use-cases I listed above. For some there are features available in the latest docker version (1.9), for some there are third party solutions and for the others, there are custom hacks. I will list all of them depending upon their use-cases.

Solution 1: Dockerfile ENV variable

You can use ENV directive in Dockerfile to define variables and use it with other primitives in Dockerfile. According to me, this just helps to make your Dockerfile pretty and it shouldn’t be use for passing secrets or any sensitive information during the build process. Here’s why:

Issues:

• Env variables that are passed are preserved in the final image as well as all the intermediate layers of the image. Anyone can do a “docker inspect” and can see the values of these variables. So, if you use this approach to pass secrets for your buildtime dependencies, you fear exposing them to anyone who pulls down the image.
• Your Dockerfile is static with this approach. You cannot override ENV. values if you are building the image from various hosts that require different proxies.
• Be mindful that the variables(secrets) are written to the disk at every intermediate and final layer of the image.

Solution 2: Docker build time variables

To overcome the issue about the variable persisting in the intermediate and final image, Docker introduced -build-arg variable in their 1.9 release. This flag allows you to pass the build-time variables that are accessed like regular environment variables in the RUN directive. Also, these values don’t persist in the intermediate or final images like ENV values do.

A good example is http_proxy or source versions for pulling intermediate files. The ARG instruction lets Dockerfile authors define values that users can set at build-time using the –build-arg flag:

Example: docker build –build-arg HTTP_PROXY=http://10.20.30.2:1234 .

Issues:

mapuri mentions it shouldn’t be used when the image caching is turned on.

Solution 3: Flattening Images

For the fear that your variables are visible in the intermediate layers of the image, people have started flatting their images. This also helps in reducing the overall size of the image and can save on your image upload/deploy time.

There are quite a few projects that lets you do that:

• docker-squash from jwilder
• Or you can do it yourself with exporting and importing the image:
  ID=$(docker run -d image-name /bin/bash)
  docker export $ID | docker import – flat-image-name

Issues:

• You have to add flatteing(hacking) in your build and deployment workflow.
• It will remove the secrets from the intermediate layers, but it may still end up in the build cache.
• It unnecessary adds complexity when you are automatically building images using CI/CD pipeline.
• Be mindful that the variables(secrets) are written to the disk at every intermediate and flattened final layer of the image.

Solution 4: Hosting secrets on a server

To serve secrets or ssh keys to the build process, there are various tools available like vault from dockito, that runs in its own container to serve the key over the HTTP.During the build, it’s invoked from the Dockerfile, using a special RUN directive, where you just pass it the command that requires the secret. It will fetch the key from the server and executes the command.

docker-ssh-exec is a bit more advanced that vault from dockito. In that, it fetches the key over the network from the server container, writes it to disk, executes the desired command, and then removes the key. Since all this occurs within the scope of a single RUN directive, the key data is never written into the resulting filesystem layer.

Conclusion

There isn’t a single solution that satisfies every use case for the build time secrets. People have gotten restless about the lack of features from docker. So, they have went ahead and added their own custom hacks.

As mentioned by thaJeztah, there are lots of PR(Pull Request) at Docker still pending:

• Add private files support #5836
• Add secret store #6075
• Continuation of the docker secret storage feature #6697
• The Docker Vault” #10310
• Provide roadmap / design for officially handling secrets. Make injecting secrets pluggable, so that they use existing offerings in this area, for example: Vault, Keywhiz, Sneaker