Ligus - Effective Monitoring and Alerting

Ive been fortunate to get hired into medium-sized operations teams at large technology companies. All ops teams (a customary term for operations teams) share two interesting characteristics: compared to other engineering departments, they work under more pressure, and they attract bad attention much easier than good attention. Digital firefighting is the nature of the job. We might get noticed when things go awry and we fix them. If we dont react fast enough, we definitely get noticed. If you know anyone in network operations, ask if thats the way he or she feels about the jobI bet youre going to get an answer along those lines.

Working in ops is all about effectiveness: there is no time for re-engineering. We must get things right the first time and we have to act fast. We go through a lot of reprioritizing and context-switching. There is relatively little room for creativity, at least the kind that doesnt love constraints. All this makes operations a great place to learn and grow.

This book is based on experiences of working in ops. I was extremely lucky to work with some of the smartest people in the industry. I would like this book to be a tribute to all these invisible ops guys who struggle daily to maintain the highest standards of service availability.

In my career, Ive stared at all sorts of timeseries plots, a lot of them. At one point it was my full-time jobno kidding. With time, I learned to extract meaning from data point fluctuations just by a brief glance, without having to study their origin. Its a funny kind of intuition that system engineers develop in the course of their jobs, and one that probably saves us a lot of time. Some of us are unaware of it, and its definitely not something we brag about. It is a very useful skill, nevertheless, and in this book I attempt to verbalize it in order to assist you, dear Reader, to absorb it in a more conscious way than I did, possibly saving you weeks or months of getting up to speed.

Some people on my team believed that putting in motion the ideas described here led to a visible paradigm shift. I must agree that in a relatively short period of time, the work caused by our alerting configuration went from mundane to effortless.

This book focuses on monitoring and alerting in the context of distributed information systems, but Im hoping that the principles presented here will also be applicable to timeseries and datasets generated by all sorts of complex systems. The book does not focus on any particular software package. Rather, it attempts to extract and summarize regularities that system engineers come across in their daily work. You wont find many long code listings here, but youll definitely find ideas: ones that I hope youll be able to relate to and apply either at work or in a research project.

Enjoy!