Alan Carter - The Programmers’ Stone

Hi, and welcome to The Programmers' Stone. The purpose of this site is to recapture, explore and celebrate the Art of Computer Programming. By so doing we hope to help the reader either become a better programmer, understand what less experienced programmers are struggling with, or communicate more effectively with other experienced programmers.

We know from work with individuals that by doing this we put the fun back into the work and greatly extend the boundaries of the possible, so building much smarter and stronger systems.

The present structure is planned around an eight day course, delivered two days a week for four weeks. Each chapter corresponds to the course notes for one day's material. The eighth day should be free discussion, so no prepared notes, meaning that there are seven chapters. We've deliberately made each chapter a single HTML page because it makes it much easier to print the text. Sorry there are no internal anchors yet, there are big headings, so use your slider!

We'd very much like to hear from you!

Alan & Colston

The work leading to this course was motivated by wondering why,in software engineering, there are some people who are one or twoorders of magnitude more useful than most people.If this was true of bricklayers,the building industry would be very keen to find out why.The problem of course, is that one can film a bricklayer,and later analyze what is happening at leisure.One cannot even see what great programmers do,and for some reason they cannot explain what the differenceis themselves, although most of them wish they could.

We knew that the elements of industry best practice alone are not enough.Management commitment to investment and training arenot enough.Innovative Quality programmes that explicitly includeholistic concepts such as Robert Pirsig'sZen and the Art of Motorcycle Maintenance,which much of the industry wouldconsider too radical to experiment with are not enough.Years of experience are not enough, nor are years of academic study.

There seemed to be only one way to continue the investigation if anindustry dedicated to objective metrics had not found the Xfactor:we needed to look at the subjective experience of the peopleconcerned.

Achieving understanding of what was happening took a long time,although the key ideas are known to most of us already.On the way we learned a great deal about the mind set of successfulprogrammers, and were able to develop exercises that certainlyhelped many people.

Thus the material in this course has developed over several years,and is a mix of ideas empirically justified by experiment and laterfitted into the logical picture,and material derived from the logical picture.

This course aims to address the element of `experience' or `judgment'referred to almost everywhere, but rarely described.Many of the topics are the kind of thing programmers discuss over a beer.Perhaps it is odd that no-one tends to ask how the issuesthat programmers see as most important relate to the `formal'structures of modern engineering.Here, we do just that.

We have found that once we get into the swing of this,most programmers find they have an opportunity to put issuesthey have wondered about for years into a clear work context,together with their colleagues.We therefore ask you to relax,because you are supposed to be doing this,and have an enjoyable time!

Software engineering is in a terrible pickle.The so-called `Software Crisis' was identified in 1968,but despite thirty years of effort,with hundreds of supposedly fundamental new concepts published,the general state of the industry is horrific.Projects run massively over-budget or collapse entirely in unrecoverable heaps. Estimating is a black art,and too many projects solve the customers' problems of yesterday, not today.The technical quality of most code is dreadful,leading to robustness problems in service and high maintenance costs.And yet within the industry there exist individuals and groupswho enjoy staggering, repeatable successes.There are many ways of measuring the usefulness of programmers,but some are rated as over a hundred times more useful than most,by several methods of counting.If only the whole of the industry performed as well as thetiny minority of excellent workers,the economic benefits would be immense.If it were possible to write sophisticated, reliable softwarequickly and cheaply,the intelligence of society would increase,as everything from car sharing to realistic social security regulationsbecame possible.

Within this model, the problem can be understood.What is presented as socially conditioned conventional thinking (called packing) is based on action.To be a good bricklayer,a packer must know what a bricklayer does.What does a programmer do?The most developed packer model of programmingis the concept of the Software Factory.In this, statements of requirements from customers go in one door,and are processed by workers following procedureswritten down in manuals.When the production line has done its work,programs come out of the other door.It works in car factories.

The trouble is, the analogy with a car factory is sloppy.Most of the car factory is filled with workers using machines to make cars,but around the back there is a little office where another workerdetermines how to use the resources of the factory to make asmany cars as possible,all alike.

The workers in a software shop are not like the factory floor workers.The shop floor workers can be replaced with robots today,but the person who uses creativity to set up the factory is still needed.The programmers are doing the same job as the office atthe back of the factory,and we cannot learn anything about whathappens in there by playing at car factory shop floors.

Packers who advocate uncompromising process-based SoftwareFactories are in fact claiming to be able to implement an ArtificialIntelligence that simulates a production line designer,and to be able to do it by using humans pushing bits of paper around as theircomputer.Unfortunately, packing is just not up to the job ofunderstanding software production, and gets terribly confused.This means it says some very silly things sometimes.

To understand what programmers really do, an alternative strategy of thinking (called mapping) is necessary,because programming is essentially a process of internalising thecapabilities of the system, the nature of the problem, and the desire,and capturing the insight in a programming language.It is all about exploring the details of our desires,and understanding them in sucha way that we can keep track of all the complexity.Mapper problem collapse can produce beautiful,tiny, elegant programs with no room for bugs in them.Mapping can do programming, but how it does itcannot be explained in packer, action-based language.

Packers therefore assert that hackers are `irresponsible' anddiscount their work, saying that complexity is inherently notunderstandable and we must develop ever more complexprocedures to abdicate our responsibility to.

Fortunately, many organisations' managements continue to fosterreflection on grounds of personal intuition and empirical experience,without any justifications to place on action-based balance sheets.This is a difficult thing to do,but is the only reason anything gets done.

It is important to recognise that mapping is not another proceduralmethodology to be applied in a packer mindset.It is a different way of looking at things altogether.It is necessary to convince yourself thatit really is possible to take personal responsibilityfor an undertaking instead of abdicating in favour of a procedure.

Programming is as near to pure mapping as you can get outside your skull.This is why it is fun.It is endless discovery,understanding and learning.