Monitoring with Graphite
Jason Dixon
Beijing Boston Farnham Sebastopol Tokyo
Preface
This book is a work in progress new chapters will be added as they are written.We welcome feedback if you spot any errors or would like to suggest improvements, please email the author at .
Twenty years ago it was considered bleeding-edge to monitor your Internet router with SNMP and a Perl script. This gave network administrators a new level of insight into their operations and user activity. As the popularity of the Internet soared, so too did the number of successful online businesses and services. This, predictably, led to better monitoring and trending software. On the one hand, businesses had to defend their investment against service and network outages; on the other, they needed improved methods for capacity planning as their systems scaled to meet user demand.
Graphite is one of the most powerful monitoring tools available today. Its popularity is driven not only by its Open Source availability, but its ease of use, rapid graph prototyping abilities, and a friendly rendering API that allows anyone to embed Graphite charts in their own applications or websites. The Graphite user community is huge and routinely feeding enhancements back into the core project, such as new statistical and transformative functions, output formats like JSON for client-side chart rendering, and even pluggable storage backends to leverage the growing ecosystem of distributed database systems.
Who Should Read This Book
Monitoring with Graphite is for anyone who wants to learn more about monitoring systems, services or applications. The book makes few assumptions about your background or experiences other than that you have access to a computer for setting up and interacting with your own Graphite instance.
Systems Administrators and Users who wish to gain advanced skills related to scaling Graphite should be comfortable with a Linux or UNIX-style command line environment. You wont be dropped in the morass and expected to fend for yourself, but youll get more out of this experience if you already know how to navigate a UNIX filesystem, interact with package managers, and what to do when disk fills up due to an excess of Whisper files.
Everyone should be prepared to have fun learning about monitoring best practices, time-series data storage and retrieval, and constructing charts with a toolkit full of rendering functions and statistical primitives. You will impress your friends and family with the skills you take away from this volume. I almost guarantee it.
Last but certainly not least, understanding how to use and administer Graphite is a valuable skill. Search any online technical job board and youre sure to find a number of opportunities for individuals that know how to use Graphite effectively or better yet, install and administer it.
Why I Wrote This Book
In spite of Graphites popularity, advanced knowledge around the maintenance and scaling out of its software components tends to be concentrated among a handful of experienced users and developers. As a core developer and maintainer of the Graphite project, Ive seen the effects of this knowledge gap first-hand as businesses and individuals approach me routinely for scaling advice and demonstrations. Unfortunately, I dont scale nearly as well as the project I love. Therefore, it only makes sense for me to relent and communicate my experiences in this book.
Some of my friends refer to me as the Graphite Whisperer. As ridiculous as the title may sound, I admit that I feel a certain pride arranging an elaborate sequence of statistical functions into a graph that accurately represents the source data. I hope that Im able to imbue you, the reader, with as much Graphite mastery as you can endure. Understandably, not everyone gets as excited talking about time-series line charts as I do. Ill do my best to keep the forthcoming content instructional yet entertaining, relevant and applicable to the most common use cases.
A Word on Monitoring Today
Three years ago I was writing about how #monitoringsucks and we need to reconsider our approach to monitoring architectures. Traditional monitoring systems were monolithic and unwieldy. Nagios already had a largely negative connotation but there were few alternatives on the horizon.
Either by luck or instinct, I correctly predicted that the monitoring ecosystem would evolve towards a composable model of well-defined services and compatible interfaces. Look at the Open Source monitoring community and this is precisely what youll find today.
In particular, the emergence of NoSQL databases has resulted in enormous momentum and competition among the various time-series databases. Users and businesses have embraced Etsys Measure Anything, Measure Everything mantra, which has driven the need for larger and faster data stores. A hugely successful conference series arose out of the desire for more collaboration and discussion around our monitoring toolchain. People are more open and transparent about their processes and tooling and these shared experiences benefit everyone.
In other words, software moves fast and Graphite is no exception. Fortunately, Graphite has reached such a high level of adoption among users and businesses that youll be hard-pressed to find tools and services that dont integrate with it directly through its robust APIs or among any number of third-party bridge utilities. In many regards, Graphite has become a time-series specification that other software projects leverage for themselves.
Navigating This Book
This book is organized roughly as follows:
- Chapters 1 and 2 provide a basic introduction to monitoring and trending concepts and terminologies. Theres a large vocabulary of terms shared among Graphite users; it helps to speak the same language.
- Chapter 3 discusses various approaches to instrumentation and gathering Telemetry data. Engineers in particular should ideally take a lot of new concepts away from reading this chapter.
- Chapters 4 and 5 introduce the components that make up Graphite and their respective features and functionality. You will learn how to install Graphite and configure it for a basic setup.
- Chapter 6 and 7 cover the typical user workflow for creating your first line chart. By the end of these chapters you should be very comfortable building complex charts with chained functions, multiple axes and interacting with the rendering API directly.
- Chapter 8 introduces the native Graphite dashboard and some of the more popular third-party external dashboards. Youll also get a more thorough coverage of the render API and how to leverage it for client-side chart rendering with javascript frameworks like D3.js.
- Chapters 9 through 12 are targeted at Systems Administrators and power users who want to master the art of scaling and troubleshooting high-performance or highly available Graphite clusters.
If youre like me, you dont read books from front to back. If youre really like me, you usually dont read the Preface at all. However, on the off chance that you will see this in time, here are a few suggestions:
Online Resources
- http://obfuscurity.com/Tags/Graphite
- http://graphite.readthedocs.org/en/latest/
- http://graphite.readthedocs.org/en/0.9.13/
- https://answers.launchpad.net/graphite/
Conventions Used in This Book