We, at YP are using docker containers for quite some time now. Onboarding onto docker wasn’t always that easy. There are lots of things to account for before running a docker container in production. One of the thing to address is how to deal with secrets during runtime.
We have done significant work on that front. I will be discussing in multiple blog posts about the problem and the potential solution with regards to injecting secrets to the docker container. In this post, I will be talking about how do people use secrets with docker containers and their issues.
Why are secrets important?
Secrets are important for every application. Some of the application secrets that you may need are:
- database credentials
- api tokens
- ssh keys
- TLS certificates
- GPG keys etc.
Traditionally, we have been storing these secrets under some packages that are encrypted or storing it in “secrets store” or just putting it as a part of the source code. Well that was all ok and good. But we cannot use the similar solutions with docker images. Then how do we use it with the docker containers?
Solution 1: Baking it in the image
Well this is straight forward: you will just put it as part of the image. This is the first thing you will do when you are onboarding your app onto docker. Maybe you will put under some dot file, chown it to root and think that everything is fine. This is the most prevalent anti-pattern in security.
Issues:
- When it is published to any registry, anyone can pull that image and the secrets would be at their disposal.
- None of Github or Dockerhub or your repository is designed to securely store or distribute secrets.
- Updating secrets is a tedious job. Updating all the images.
- This could still be ok if you have few number of images, but consider you tie in CI/CI pipeline to your image build process. Now you are managing tons of images.
- Accounting for certificate expiration becomes difficult.
- Old, EOL/EOS or decommissioned hardware can cause secrets leak.
Solution 2: Put it under ENV variables
This is the most common way to pass secrets to the applications (more than 90% of people do it). It is widely used because 12 factor app guidelines recommend apps to be delivered and consumed as a service.
Example: docker run –it –e “DBUSER=dbuser” –e “DBPASSWD=dbpasswd” myimage /bin/bash
Issues:
thaJeztah and diogomonica have captured in detail about the best practices about using secrets. However, I am just summarizing the issues with this solution here:
- Kept in intermediate layers of image and can be easily viewed using “docker inspect”.
- Accessible by all the processes in the image. Can be easily leaked.
- Shared with any linked container.
- Incredibly common having the app grab the whole envt., print it out or even send it part of error report or pager duty.
- Env. variables are passed down to child processes. Imagine that you call third party tool to perform some action, all of a sudden that third party has access to your environment.
- Very common for the apps that crashes to store env. variables in log files for debugging.
Solution 3: Volume Mounts
This is again as straight-forward as passing ENV variables. You put your secrets in some directory structure on docker hosts. That directory structure can be on local file system, NFS or DFS like CEPH. You then mount the right directory inside the container for that particular app.
Example: docker run –i –t –v /mnt/app1/secrets:/secrets myimage /bin/bash
Issues:
- Bad design putting all the secrets for all the images on a single machine.
- Secrets are unencrypted, in plain text.
Solution 4: Secrets encryption
Some people are paranoid about keeping their secrets in plain text. And they are even more paranoid about putting image with plaintext secrets to some private/public docker registries. So, they encrypt the secrets using public key and elliptic curve cryptography using tools like “ejson” from Shopify and others. To decrypt, private keys are hosted on the docker hosts and those production machines are locked down. At least, with this way your image is safe from snooping.
Issues:
- To update secrets, you need to create new images.
- Solution is fairly static.
- You can still see which private keys are used to decrypt using “docker inspect”.
Solution 5: Secrets store
There are secrets management and distribution services like: HashiCorp’s Vault, Square’s Keywhiz and Sneaker (for AWS). They help you generate and distribute secrets for services. Main benefit of this approach is that secrets are centrally managed in a secure manner. And there is also an auditability with the secret access. Almost all these solutions are API based and are mostly reliable.
There is already an integration of Keywhiz secrets store with docker as a volume-driver plugin. This solution is the robust of all and already integrated with docker. However, if docker (or docker swarm) is the only way you manage and run your containers. This plugin doesn’t extend well if you are using orchestration tools like Mesos or Kubernetes to manage/run your containers.
If you orchestrate containers through Mesos or Kubernetes, watch out my next series of post regarding the solution.