Matt Massie - Monitoring with Ganglia

Beijing Cambridge Farnham Kln Sebastopol Tokyo

In 1999, I packed everything I owned into my car for a cross-country trip to begin my new job as Staff Researcher at the University of California, Berkeley Computer Science Department. It was an optimistic time in my life and the country in general. The economy was well into the dot-com boom and still a few years away from the dot-com bust. Private investors were still happily throwing money at any company whose name started with an e- and ended with .com.

The National Science Foundation (NSF) was also funding ambitious digital projects like the National Partnership for Advanced Computing Infrastructure (NPACI). The goal of NPACI was to advance science by creating a pervasive national computational infrastructure called, at the time, the Grid. Berkeley was one of dozens of universities and affiliated government labs committed to connecting and sharing their computational and storage resources.

When I arrived at Berkeley, the Network of Workstations (NOW) project was just coming to a close. The NOW team had clustered together Sun workstations using Myrinet switches and specialized software to win RSA key-cracking challenges and break a number of sort benchmark records. The success of NOW led to a following project, the Millennium Project, that aimed to support even larger clusters built on x86 hardware and distributed across the Berkeley campus.

Ganglia exists today because of the generous support by the NSF for the NPACI project and the Millennium Project. Long-term investments in science and education benefit us all; in that spirit, all proceeds from the sales of this book will be donated to Scholarship America, a charity that to date has helped 1.7 million students follow their dreams of going to college.

Of course, the real story lies in the people behind the projectspeople such as Berkeley Professor David Culler, who had the vision of building powerful clusters out of commodity hardware long before it was common industry practice. David Cullers cluster research attracted talented graduated students, including Brent Chun and Matt Welsh, as well as world-class technical staff such as Eric Fraser and Albert Goto. Ganglias use of a lightweight multicast listen/announce protocol was influenced by Brent Chuns early work building a scalable execution environment for clusters. Brent also helped me write an academic paper on Ganglia[] and asked for only a case of Red Bull in return. I delivered. Matt Welsh is well known for his contributions to the Linux community and his expertise was invaluable to the broader teams and to me personally. Eric Fraser was the ideal Millennium team lead who was able to attend meetings, balance competing priorities, and keep the team focused while still somehow finding time to make significant technical contributions. It was during a brainstorming (pun intended) session that Eric came up with the name Ganglia. Albert Goto developed an automated installation system that made it easy to spin up large clusters with specific software profiles in minutes. His software allowed me to easily deploy and test Ganglia on large clusters and definitely contributed to the speed and quality of Ganglia development.

I consider myself very lucky to have worked with so many talented professors, students, and staff at Berkeley.

I spent five years at Berkeley, and my early work was split between NPACI and Millennium. Looking back, I see how that split contributed to the way I designed and implemented Ganglia. NPACI was Grid-oriented and focused on monitoring clusters scattered around the United States; Millennium was focused on scaling software to handle larger and larger clusters. The Ganglia Meta Daemon (gmetad)with its hierarchical delegation model and TCP/XML data exchangeis ideal for Grids. I should mention here that Federico Sacerdoti was heavily involved in the implementation of gmetad and wrote a nice academic paper[] highlighting the strength of its design. On the other hand, the Ganglia Monitoring Daemon (gmond)with its lightweight messaging and UDP/XDR data exchangeis ideal for large clusters. The components of Ganglia complement each other to deliver a scalable monitoring system that can handle a variety of deployment scenarios.

In 2000, I open-sourced Ganglia and hosted the project from a Berkeley website. You can still see the original website today using the Internet Archives Wayback Machine. The first version of Ganglia, written completely in C, was released on January 9, 2001, as version 1.0-2. For fun, I just downloaded 1.0-2 and, with a little tweaking, was able to get it running inside a CentOS 5.8 VM on my laptop.

Id like to take you on a quick tour of Ganglia as it existed over 11 years ago !

Ganglia 1.0-2 required you to deploy a daemon process, called a dendrite , on every machine in your cluster. The dendrite would send periodic heartbeats as well as publish any significant /proc metric changes on a common multicast channel. To collect the dendrite updates, you deployed a single instance of a daemon process, called an axon, that indexed the metrics in memory and answered queries from a command-line utility named ganglia.

If you ran ganglia without any options, it would output the following help:

As you can see from the help page, the first version of ganglia allowed you to query and sort by 21 different system metrics right out of the box. Now you know why Ganglia metric names look so much like command-line arguments (e.g., cpu_num, mem_total). At one time, they were!

The output of the ganglia command made it very easy to embed it inside of scripts. For example, the output from could be used to autogenerate an MPI machine file that contained the least-loaded machines in the cluster for load-balancing MPI jobs. Ganglia also automatically removed hosts from the list that had stopped sending heartbeats to keep from scheduling jobs on dead machines.

Example 1. Retrieve the 10 machines with the least load