Application Configuration Management with Mesos

Configuration Management has always been a big ticket item. It is much bigger now that we are running applications at scale with orchestration systems like Mesos. Everyone has their own opinion and they have their different ways to specify and retrieve configurations for the application. Some people use key value store, some use home grown solution whereas some have tied it with their version control repositories. We want a way to systematically specify, control and account for changes to the application running through an orchestration system. We want to maintain software integrity and traceability throughout the lifecycle of that application.

With all the effort invested around creating multiple frameworks for Mesos, adding new features to the Mesos kernel, and creating a nice frontend like DCOS from Mesosphere, a very little effort has been spent around the configuration management of an application running through Mesos. As things stand right now, if you run containerized workload, you bake the configuration of your application in the image itself. Maybe you can make it more dynamic by passing ENV values through the framework to the containers. But then again, how many ENV variables will you expose if there are multiple configurations for different environments ? That is the first part of the issue.

Second part is how would you store those configurations externally and how would they interact with Mesos’s orchestration of containers. Lets say you are able to retrofit your existing configuration management system with Mesos, but how would you built in access control mechanism since the containers are ephemeral and they have a unique name.

That brings us to our third problem. What if you want your applications to automatically adapt to the changes happening in the infrastructure or one of your endpoint changes. How would the application pick up those changes, reconfigure themselves and automatically adapt to it without having to relaunch your application through the frameworks (for ex: Marathon).

These are real burning questions that one needs to address before running heterogeneous workload on Mesos. Things would have been simple if you are running simple workload and their configuration is pretty static. With Mesos, you can scale those applications to millions of copies (provided you have resources). But things can be tremendously different when you have a heterogeneous workload and they have myriad set of configurations.

We, at YP, are one of the big Mesos shop. We have spend significant amount of time and energy to run heterogeneous workload on Mesos and have written tools and technologies around it. Since we are leveraging open source technology for our benefit, we feel we should share our findings and solutions so that we can make a healthy Mesos ecosystem.

I would love to talk about this during the #MesosCon2016 if they accept this proposal. Register for the conference if you haven’t.