Roger Ignazio - Mesos in Action

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact

2016 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps.

Recognizing the importance of preserving what has been written, it is Mannings policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine.

Manning PublicationsCo.20 Baldwin RoadPO Box 761Shelter Island, NY 11964

ISBN: 9781617292927

Printed in the United States of America

1 2 3 4 5 6 7 8 9 10 EBM 21 20 19 18 17 16

For Sarah

If you ever want to see a mans head explode, walk up to somebody whose job it is to manually configure and provision scores of servers inside a datacenter, and say the following: Wow! It must be so easy, and so much fun, to keep track of whats running on which machines.

Or find a person who has spent years carrying a pager and responding to server outages and say, Sounds like a stress-free job. At least it guarantees you get a good nights sleep.

The truth, of course, is that managing servers and other datacenter infrastructure has historically been difficult, tedious, and full of sleepless nights for the poor men and women charged with configuring all these machines and responding to failures whenever they strike. As companies started relying more heavily on information technology during the past 20 years, often housing one application per server (or, in recent years, virtual machine), the practice grew increasingly difficult. A few servers grew to dozens, dozens grew to hundreds, and hundreds sometimes grew to thousands.

Then the web blew up, fueled by incredibly popular services such as Google, Facebook, and Twitter. The mobile web, fueled by billions of smartphones, tablets, and other devices, followed shortly thereafter. Old ways of computing could no longer cut it in a world where millions of users might be on a site or app at any given time.

Inside datacenters, single-server databases (single-server everything, really) were rapidly replaced by distributed systems that could handle previously unimaginable volumes of data and traffic. Often, complex monolithic applications were replaced by microservicescollections of single-purpose services managed separately, then connected via API to construct end-user applications. Scalability demands increased, but so did the learning curve for building these systems and the complexity of managing them.

Google famously solved this problem inside its own datacenters with a system called Borg, which ostensibly let most employeessystems administrators and developers alikemanage hundreds of thousands of servers like one big computer. Several years after Borg simplified operations at Google, the open source Apache Mesos project hit the scene and changed the lives of its users in similar ways. All of a sudden, the process of deploying, running, and managing complex distributed systems became much simpler; everything shares the same set of machines, and Mesos handles the legwork of matching workload requirements with available resources.

I experienced this shift firsthand as a software engineer at Twitter, where Mesos helped conquer the infamous fail whale and helped Twitter reach new heights of scalability and reliability. When I moved on to Airbnb in 2012, then a fast-growing startup just four years old, Mesos once again helped us grow our infrastructurebut not the complexity of italong with our user base. I was so impressed with Mesos and its promise that I decided to start a company, Mesosphere, dedicated to making Mesos usable by mainstream enterprises.

As Mesos grew in popularity and Mesosphere grew in size, we set our sights on hiring the best Mesos engineers and practitioners around. When we saw the work Roger Ignazio had done building out a Mesos-based continuous integration platform at Puppet Labs, we knew we had to have him. Running scalable production systems at established companies is always valuable experience, and since joining Mesosphere, Rogers experience has been a boon to the quality of our Mesos-based Datacenter Operating System technology and to our customers experiences with it.

With Mesos in Action, Roger lets anyone who is interested in Mesos and its ecosystem of technologies take advantage of his experience. The book is a great guide to getting a Mesos cluster running and installing your first frameworks, but also delves into more-advanced topics such as mastering powerful Mesos frameworks (including Marathon for container orchestration and Spark for big data analytics) and even building your own framework.

Whether youre preparing to deploy Mesos or already have it running and want to improve your knowledge, you would be hard-pressed to find a better guide than Roger and a better book than Mesos in Action.

F LORIAN L EIBERT

C OFOUNDER AND CEO , M ESOSPHERE

Apache Mesos began in 2009 as a research project at the University of California at Berkeley, led by Benjamin Hindman. Ben and his team wanted to improve datacenter efficiency by allowing multiple applications to share a single computing cluster, just like multiple applications can share the processor, memory, and hard drive in your laptop or workstation. But they wanted to do this across the many servers that make up a modern datacenter. After an initial implementation of 10,000 lines of C++ code, they published the paper Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center in 2010.

Not long after, Ben joined Twitter and used Mesos to better scale its infrastructure, largely bringing an end to the era of the fail whale that became infamous as Twitter was rising in popularity and its servers couldnt handle the demand from users. Although Twitter doesnt publicly disclose the number of servers in its expansive infrastructure, online sources and firsthand knowledge from presentations put this somewhere in the ballpark of 10,000 Mesos nodes per cluster.

In December 2010, the Mesos project entered the Apache Incubator, an arm of the Apache Software Foundation that provides a means for projects to gain the full support of the ASFs efforts. The Apache Mesos project graduated from the incubator in June 2013 and is now a top-level project.