Diego Zamboni - Learning CFEngine 3: Automated system administration for sites of any size

The history of Unix system configuration has been a fascinating ride that took us from shell scripting to sophisticated knowledge-oriented tools.

I still recall arriving in San Diego in 1997 for the USENIX/LISA conference, just three years after releasing CFEngine to the wider world as a GNU Free Software distribution. I walked through the door from conference registration and the first person I met looked at my badge and said: Hey, youre Mark Burgessyou wrote CFEngine! That was my first exposure to the power of community.

Free Open Source Software (FOSS) was a kind of Berlin Wall moment for the software industry, removing the barriers to contributing innovative ideas that had been closed off by fearful corporate protectionism. Perhaps ironically, free software was the door-opener to innovation that enabled Internet commerce to take offtransforming Richard Stallmans vision of free speech into a definite focus on free beer, but with the importance of community social networks strongly emphasized.

To me, what was important about FOSS was that it enabled research and development to flourish and find a willing audience, all without anyones approval. For CFEngine, this was central to overcoming limitations steeped in the past.

When I began writing CFEngine in 1993, inspired by colleagues at Oslo University, the main problem lay in handling a diversity of operating systems. There were many more flavors of Unix-like OS back then, and they were much more different than they are today. Writing any kind of script was a nightmare of exception logic: If this is SunOS 4.x or Ultrix, but not SunOS 4.1 or anything at the Chemistry department, and by the way patch 1234 is not installed, then...

Such scripts appealed to a generation of Large Installation System Administrators, who had deep system experience and basic programming skills. Alas, in such a script, you couldnt see the intention for the logic, so many scripts were thrown away and rewritten in the latest cool scripting language each time someone arrived or left. It was a time-wasting chaos.

The separation of intended outcome from the detailed imperative coding was the first purpose of a specialized language for system administration, i.e., making infrastructure documentation of intent rather than unreadable codeor as declarative programmers would say, the separation of what from how.

As a theoretical physicist, in postdoctoral purgatory, instinct moved me to look into the scientific literature of the subject of system management, and I discovered that there was very little work done in the field of host configuration. As I left the conference in 1997, I got sick on the plane, and this gave me an idea. A year later, I went back to the LISA conference and wrote down a research manifesto for autonomic self-healing systems called Computer Immunology. IBMs autononomic computing initiative followed a few years later. Those were heady days of CFEngine history, filled with excitement and discovery of principles like convergence and adaptive locking. At LISA 98, I presented Computer Immunology in one hall of the conference while Tom Perrine (then of the San Diego Supercomputing Center, later LOPSA president) opened his talk in the next room with the flattering words: I owe Mark Burgess more beer than I can afford... And thus the partnership between science and community was begun.

CFEngines 1 and 2 took the world by storm. No one really knows how many agents are running out there, but it runs into the many millions. A large covert community still thrives behind the scenes, making little noise. Recently, a large internet retailer indicated a million computers running CFEngine 2, saying: Well, it just works. Similar stories abound.

Even so, CFEngine had rough edges, and we saw plenty of room for improvement. As the Web 2.0 companies were emerging in the 2000s, other tools began to emerge for configuration, bringing back the idea of Script It Yourself to engage a generation of web programmers impatient with the idea of system administration getting in the way of more agile methods. Software packaging developed into an important simplification of the configurationbut much too simplistic to support the required competitive differentiation in an application-driven era of IT. From this tension, the idea of DevOps began to emerge and configuration moved back in the direction of custom coding, aided by easy language frameworks like Ruby.

By this time, I had developed a new model for CFEngine that captured its famous distributed autonomy, and had brought CFEngine its documentable scalability and security properties. This model came to be known as Promise Theory, and as I developed and tested the idea from 2004-2007 I realized that the challenge was not at all about scripting or programming, but really about knowledge and documentation (The Third Wave of IT). The CFEngine answer was thus to pursue the original idea: that understanding infrastructure is about modelling intent, not about one-size-fits-all commodity packaging. CFEngine 3 should not be code development, but declaration of intent (promises).

In early 2008, almost ten years after the Computer Immunology manifesto, I began coding CFEngine 3a strict implementation of my understanding of the best that science and community experience had uncoveredto promote a technology direction that could go beyond the immediate needs to datacentres and create a legacy for dealing with scale, complexity, and agility for the coming decade.

And today? In todays environment where everything seems steeped in web programming, source code seems ironically less important than during the formative rebellion of FOSS; Application Program Interfaces (APIs) are the new open source, but the danger lies in being pulled back into opaque custom scripting, that conflates what with how.

Today, there is a CFEngine company as well as a vibrant community that supports and develops future innovation in the CFEngine technology; and users are moving to the next level: Knowledge Driven Configuration Management.

Today, I am also proud and a little humbled to read Diegos fine book about this new challenge, and finally join the ranks of the OReilly bestiary. He has been able to present CFEngine in a way that I was never able to do, and make it accessible to readers of all levels and backgrounds. As one community member wrote, this is the tutorial the CFEngine never had.

The future of system administration is once again in the making, with recyclable resource management now reaching the platform level through Cloud thinking, applications growing from complex integrations of FOSS sub-systems, and datacenters flaring like novae around us. In these heavens, CFEngine is still a guiding star, paving the way towards a new generation of knowledge-based infrastructure engineering.