Month: February 2016

Application Configuration Management with Mesos

Imran

Configuration Management has always been a big ticket item. It is much bigger now that we are running applications at scale with orchestration systems like Mesos. Everyone has their own opinion and they have their different ways to specify and retrieve configurations for the application. Some people use key value store, some use home grown solution whereas some have tied it with their version control repositories. We want a way to systematically specify, control and account for changes to the application running through an orchestration system. We want to maintain software integrity and traceability throughout the lifecycle of that application.

With all the effort invested around creating multiple frameworks for Mesos, adding new features to the Mesos kernel, and creating a nice frontend like DCOS from Mesosphere, a very little effort has been spent around the configuration management of an application running through Mesos. As things stand right now, if you run containerized workload, you bake the configuration of your application in the image itself. Maybe you can make it more dynamic by passing ENV values through the framework to the containers. But then again, how many ENV variables will you expose if there are multiple configurations for different environments ? That is the first part of the issue.

Second part is how would you store those configurations externally and how would they interact with Mesos’s orchestration of containers. Lets say you are able to retrofit your existing configuration management system with Mesos, but how would you built in access control mechanism since the containers are ephemeral and they have a unique name.

That brings us to our third problem. What if you want your applications to automatically adapt to the changes happening in the infrastructure or one of your endpoint changes. How would the application pick up those changes, reconfigure themselves and automatically adapt to it without having to relaunch your application through the frameworks (for ex: Marathon).

These are real burning questions that one needs to address before running heterogeneous workload on Mesos. Things would have been simple if you are running simple workload and their configuration is pretty static. With Mesos, you can scale those applications to millions of copies (provided you have resources). But things can be tremendously different when you have a heterogeneous workload and they have myriad set of configurations.

We, at YP, are one of the big Mesos shop. We have spend significant amount of time and energy to run heterogeneous workload on Mesos and have written tools and technologies around it. Since we are leveraging open source technology for our benefit, we feel we should share our findings and solutions so that we can make a healthy Mesos ecosystem.

I would love to talk about this during the #MesosCon2016 if they accept this proposal. Register for the conference if you haven’t.

IT shouldn’t be a cost-center with Docker

Imran

IT is critical for every big and small organization. It touches all the business processes and data in every company. Every year organization allocates budget to keep the “Lights On”. The #1 priority of IT is to make sure that the business is running as usual. That means that it reinforces the view of IT as a cost-center. Every aspect in the IT is focused on minimizing downtime rather than providing value to the organization. And that is the how things are and that is how every company is being run. All the “C-Suite” bosses are happy to cut the fat checks and are content with it. There is nothing wrong with that. Except that you can’t improvise on that and can’t convert that cost-center into a profit-center.

If you are running a data center of your own or outsourcing it to cloud services or running a hybrid cloud, then you need a whole ensemble of a team to manage your systems, networks, databases, infrastructures etc. They will babysit your infrastructure, perform changes and do maintenances. Any change to the state of production would require long maintenance window involving all the stakeholders, tons of overtime pay, increased frustration and no family life. It is not only the people and processes, but you will also accumulate computing power along the way and keep on piling your racks in the data centers. You will tons of wasted resources and would hardly be using even 20% of the computing power that you have available. Unfortunately, this is the sorry state of affair for most of IT departments in majority of the companies.

Alas! IT Gods have finally smiled upon us. Things doesn’t have to be same like before with the advent of technologies like docker. Docker and its ecosystem have unleashed myriad range of technologies that lets you put your systems and infrastructure on an auto-pilot. Docker ecosystem provides you with the tools and technologies that allows your IT to add value to your business. I am not claiming we will be able to solve all the issues with old-school IT but there are already tools at our disposal which when combine with docker can help us solve real complex IT solutions that we have been dreading all these years. Much has been talked about docker being a developer’s friend, but it can truly help alleviate IT Ops problem and can help us run a real lean DevOps IT organization.

Using docker successfully to manage and run part of our IT operations at YellowPages, we have leveraged its capabilities with technologies like push based metrics, service discovery in ephemeral environment, orchestration tools, build pipeline and message queues to solve real world IT problems. There were lots of new challenges that we overcame doing that. Also, there is a lot of work that needs to be done for docker to become a IT department’s best friend. In short, things are looking up and we can do quite a few things with docker which will help us evolve the eco-system to make IT a truly profit center.