For decades now, computer scientists have debated on how to coordinate groups of heterogeneous compute resources to solve a set of domain specific problems. “Scheduling” was the catch all moniker that was used to describe this space. This category of problems is old, therefore the scheduling universe is vast, and expansive.
The most recent generation of schedulers that have emerged, strive to address the problem of coordinating several distributed applications across a data-center. The reason why I find this interesting, is that many distributed applications reinvent aspects of a “scheduler”, often without realizing the depth and breadth of the domain they just stepped into. There is a great ACM article that highlights this point “There’s Just No Getting around It: You’re Building a Distributed System”.
Now-a-days, we’re seeing a Cambrian explosion of software stacks, each reinventing pieces of the scheduling wheel. That’s all well and good, but there is a much easier way. Enter Apache Mesos, a cluster manager that provides efficient resource isolation and sharing across distributed applications.
At its core, Mesos is a focused meta-scheduler that provides primitives to express a wide variety of scheduling patterns and use cases. Solutions are written atop of Mesos, and are targeted for a particular use case. By remaining focused at its core, Mesos is not architecturally encumbered by domain specific problems that often exist within other monolithic schedulers.
- Introduction to Mesos
- John Wilkes #MesosCon Keynote
- Mesos Architecture
- Building and Running Distributed Systems using Apache Mesos
- Framework Development Guide
- Twitter University
I often find software first impressions are usually pretty important, and if I have to spend hours setting up an application, then that usually colors my perspective about the technology.
Therefore, in this post I will run through how simple it is for you to get started with Mesos on Fedora 21 and CentOS 7. I’ll leave HA deployments for another post, because I want to outline just how simple it is to setup for “trying it out”.
Currently dependent packages have not been fully pulled into CentOS 7, or epel channels, but I’ve enabled the mesos.spec to build a bundled distribution for those who want to run on CentOS 7. For convenience, rpms can be found here
But for those who want to rebuild it for themselves, you can download the srpm and run:
$ mock --clean --init -r epel-7-x86_64 --rebuild mesos-0.20.0-2.f421ffd.fc21.src.rpm
You will also need to update your docker installation to 1.X.
- If you setting up a multi-node cluster, it is recommended that you have DNS setup.
- You may also want to alter your firewall settings.
$ sudo yum install mesos python-mesos mesos-devel mesos-java
NOTE: If you are seeing missing dependencies such as (protobuf-python) on CentOS 7, then you do not have the epel repositories installed correctly.
Make sure docker is running:
$ systemctl start docker
Out of the box, the package is configured to run mesos on a single host machine. Which is convenient for developers who just want to install and test their applications locally before submitting to their cluster.
If you want to setup a multi-node cluster there is simply one parameter you need to set on your worker nodes /etc/mesos/mesos-slave-env.sh file:
$ systemctl start mesos-master
and open a browser to localhost:5050
$ systemctl start mesos-slave
Now check the “Slaves” tab on the browser window to verify.
$ mesos execute --command="/bin/sleep 10" --master="yourmaster.yourdomain.com:5050" --name="whizbang"
Verify that it ran under the “Frameworks” tab.
Distributed systems can be a complicated and thorny road, but setting up and deploying them doesn’t have to be, and it’s a breeze with Mesos.