James Turnbull - The Art of Monitoring

This book is for engineers, developers, sysadmins, operations staff, and those with an interest in monitoring and DevOps. It provides a simple, hands-on introduction to the art of modern application and infrastructure monitoring.

There is an expectation that the reader has basic Unix/Linux skills and is familiar with the command line, editing files, installing packages, managing services, and basic networking.

• Ruth Brown, who continues to be the most amazing person in my life.
• Kyle Kingsbury, for writing Riemann and being an excellent resource when I had dumb questions.
• Pierre-Yves Ritschard, for providing Riemann and Clojure help.
• Ben Linsay, who provided feedback on the introductory Clojure material.
• Baron Schwartz, Dean Wilson, Brice Figureau and Marc Fournier, who provided valuable feedback on the book.
• Jeff Danzinger, for kindly letting me use his cartoon about averages.
• Simone Bottecchia, Katherine Daniels, Laurie Denness, Ryan Franz, Kelvin Jasperson, Marc Fournier, Pierre-Yves Ritschard, Javier Uruen Val, Avleen Vig, and John Vincent, for answering monitoring questions.
• The folks at PagerDuty, for giving me free access to their platform.
• Bimlendu Mishra, for his scripted Grafana dashboard example.
• Michael Jakl, for his RESTful Clojure example application.

Caitie McCaffrey is a Backend Brat and Distributed Systems Diva at Twitter, where she is the Tech Lead of the Observability Team. Prior to that she spent the majority of her career building large-scale services and systems that power the entertainment industry at 343 Industries, Microsoft Game Studios, and HBO. Caitie has a degree in Computer Science from Cornell University, and has worked on several video games, including Gears of War 2, Gears of War 3, Halo 4, and Halo 5. She maintains a blog at CaitieM.com and frequently discusses technology on Twitter @Caitie.

Paul Stack is an infrastructure coder and is passionate about continuous integration, continuous delivery, and good operational proceduresand how they should be part of what developers and system administrators do on a day-to-day basis. He believes that reliably delivering software is as important as its development.

Jamie is a Site Reliability Engineer in Google's Storage Infrastructure team. He began in Linux systems administration in 1999, while earning a Bachelors in Computer Science, so knows just enough theory of computation to be dangerous in his field. He contributed a chapter on monitoring to the Google SRE Book. He lives with his family in Sydney, Australia.

Sid Orlando is a writer and editor (among other things), currently word-nerding out as Managing Editor at Kickstarter. Since starting work on more tech-focused projects, she may or may not be having recurring dreams about organizing her closet with dreamscape Docker containers.

James is an author and open-source geek. His most recent books were The Docker Book about the open-source container virtualization technology and The LogStash Book about the popular open-source logging tool. James also authored two books about Puppet (Pro Puppet and the earlier book about Puppet). He is the author of three other books, including Pro Linux System Administration, Pro Nagios 2.0, and Hardening Linux.

For a real job, James is CTO at Kickstarter. He was formerly at Docker as VP of Services and Support, Venmo as VP of Engineering, and Puppet as VP of Technical Operations. He likes food, wine, books, photography, and cats. He is not overly keen on long walks on the beach or holding hands.

This is an inline code statement.

This is a code block:

Long code strings are broken.

The code and example configurations contained in the book are available on GitHub at:

https://github.com/jamtur01/aom-code

This book was written in Markdown with a large dollop of LaTeX. It was then converted to PDF and other formats using PanDoc (with some help from scripts written by the excellent folks who wrote Backbone.js on Rails).

Please email any errata you find to .

This book is presented solely for educational purposes. The author is not offering it as legal, accounting, or other professional services advice. While best efforts have been used in preparing this book, the author makes no representations or warranties of any kind and assume no liabilities of any kind with respect to the accuracy or completeness of the contents and specifically disclaims any implied warranties of merchantability or fitness of use for a particular purpose. The author shall not be held liable or responsible to any person or entity with respect to any loss or incidental or consequential damages caused, or alleged to have been caused, directly or indirectly, by the information or programs contained herein. Every company is different and the advice and strategies contained herein may not be suitable for your situation. You should seek the services of a competent professional before beginning any monitoring program.

This is version v1.0.0 (b30718d) of The Art of Monitoring.

Let's begin with an origin story for a company called Example.com. Once upon a time(-series), Example.com had a sysadmin. She managed infrastructure that lived in data centers. Every time a new host was added to that environment she installed a monitoring agent and set up some monitoring checks. Every now and again one of those hosts would break and a check would trigger. A notification would be sent, and she would wake up and run rm -fr /var/log/*.log to fix it.

For many years this approach worked just fine. Of course, there was some drama. Occasionally something would go wrong for which there wasn't a check, or there just wasn't time to act on a notification, or some applications and services on top of the hosts weren't monitored. But largely, monitoring was fine.

Then the Information Technology (IT) industry started to change. Virtualization and Cloud computing were introduced, and the number of hosts that needed to be monitored increased by one or more orders of magnitude. Some of those hosts were run by people who weren't sysadmins, or the hosts were outsourced to third parties. Some of the hosts in her data center were moved into the Cloud, or even replaced with Software-as-a-Service applications.

Most importantly, IT became a core channel for businesses to communicate with and sell to their customers. Applications and services that had previously been seen as just technology now became critical to customer satisfaction and providing high-quality customer service. IT was no longer a cost center, but something a company's revenue relied on.

As a result, aspects of monitoring began to break down. It became hard to keep track of hosts (there were a lot more of them), applications and infrastructure became more complex, and expectations around availability and quality became more aggressive.

It got harder and harder to check for all the possible things that could go wrong using the current system. Notifications piled up. More hosts and services meant more demand on monitoring systemsmost of which were only able to scale by adding bigger, more powerful hosts, and could not be easily distributed. Under these loads, detecting and locating faults and outages grew ever slower and more challenging.