Pre-Mesos we had to prepare the necessary execution environments from testing to production and everything in between. Using Docker alongside Mesos allows us to encapsulate execution environments inside the container. That frees us from the effort of provisioning the infrastructure of every new microservice we want to release.
Our release process initially consisted of pushing to a service’s git release branch, which automatically triggered the continuous integration process. Marathon serves as deployment manager. In coordination with Mesos it allocates a suitable machine for the service, and deploys it by pulling it from our private Docker hub onto the allocated machine.
However, for external software like ElasticSearch we have no need for continuous integration and we release them directly from our local dev environment. To handle this use case we developed Shovel, which we plan to open-source shortly. It automates the process from building the Docker image containing the microservice to finally releasing them to the public. To release a microservice today we only have to prepare a Dockerfile and provide basic configuration. The basic configuration includes settings like public URL endpoint or amount of required CPU and memory resources. The rest of the release process is then completely handled by Shovel. To further simplify bootstrapping, we have a service template that contains commonly used components and allows us release a new microservice in minutes.
Mesos is still in its early days, probably best exemplified by the very sub-1.0 version numbers. New Mesos releases often include important bug fixes but upgrading has been a pain point for us due to the number of moving parts that led to catch-22 situations.
As an example, we experienced memory leaks with Docker 1.6 but were not able to upgrade for some time even though the bug got fixed in Docker 1.8. Upgrading Docker would have required upgrading to a Mesos version (0.23) that was untested with Marathon version 0.10.