Juval Lowy - Righting Software

Hardly anyone gets into software development because they were forced into it. Many literally fall in love with programming and decide to pursue it for a living. And yet there is a vast gap between what most hoped their career would be like and the dark, depressing reality of software development. The software industry as a whole is in a deep crisis. What makes the crisis so acute is that it is multi-dimensional; every aspect of software development is broken:

Cost. There is weak correlation between the budget set for a project and what it will actuallycost to develop the system. Many do not even try to address the cost issue, perhaps because they simply do not know how, or perhaps because that will force them to recognize they cannot afford the system. Even if the cost of thefirst version of a new system is justified, often the cost across the life of the system is much higher than what it should have been due to poor design and an inability to accommodate changes. Over time, maintenance costsbecome so prohibitive that companies routinely decide to wipe the slate clean, only to end up shortly thereafter with an equally or even more expensive mess as anew system. No other industry opts for a clean slate on a regular basis simply because doing so does not make economic sense. Airlines maintain jumbo jets for decades, and a house can be a century old.

Schedule. Deadlines are often just arbitrary and unenforceable constructs because they have little to do with the time it takes to actually develop the system. For most developers, deadlines are these useless things whooshing by as they plow ahead. If the development team does meet thedeadline, everyone is surprised because the expectation is always for them to fail. This, too, is a direct result of a poor system design that causes changes and new work to ripplethrough the system and invalidate previously completed work. Moreover, it is also the result of a very inefficient development process that ignores both the dependencies between activities and the fastest, safest way of building the system. Not only is the time to market for the whole system exceedingly long, the time for a single feature may be just as inflated. It is bad enough when the project slips its schedule; it is even worse when the slip was hidden from management and customers since no one had any idea what the true status of the project is.

Requirements. Developers often end up solving the wrong problems.There is a perpetual communication failure between the end customers or their internal intermediaries (such as marketing) and the development team.Most also fail to accommodate their failure to capture the requirements. Even when requirements are perfectly communicated, over time, they change. This change invalidates the design and unravels everything the team tried to build.

Staffing. Even modest software systems are so complex that they have exceeded the capacity of the human brain to make sense of them. The internal and external complexity is a direct result of poor system architecture, which in turn leads to convoluted systems that are very difficult to maintain, extend, or reuse.

Maintenance. Most software systems are not maintained by the same people that developed them. The new staff does not understand how the system operates, and as a result they constantly introduce new problems as they try to solve old ones. This quickly drives up the cost of maintenance and the time to market, and leads to clean-slate efforts or canceled projects.

Quality. Perhaps nothing is as broken with software systems as quality. Software has bugs, and the word software is itself synonymous with bugs. Developers cannot conceive of defect-free software systems. Fixing defects often increases the defect count, as does adding features, or just plain maintenance. Poor quality is a direct result of a system architecture that does not lend itself to be testable, understandable, or maintainable. Just as important, most projectsdo not account for essential quality control activities and fail to allocate enough time for every activity to complete in an impeccable manner.

Decades ago our industry started developing software to solve the worlds problems. Today, software development itself is a world-class problem.The problems of software development frequently manifest themselves in non-technical ways such as a high stress working environment, high turnover rate, burnout, lack of trust, low self-esteem, and even physical illness.

None of the problems in software development are new. People have spent their entire career in software development, not seeing software done right even once. This leads them to believe that it simply cannot be done, and they are dismissive of any attempt to addressthese issues because that is just the way things are. They may even fight those who are trying to improve software development.They have already concludedthat doing so is impossible, therefore anyone who is trying to get better results is trying to do the impossible,which insults their intellect.

The Humble Programmer, by Edsger W. Dijkstra, ACM Turing Lecture 1972

My own track record, however, isa counterexampledemonstrating that it is possible to successfully develop software systems.Every project for which I was responsible shipped on schedule, on budget, and with zero defects. I continued this record after founding IDesign where we have helped customers again and again deliver on their commitments.

Thisconsistent repeatable track record of success was no accident. My training and schooling were in system engineering, both of physical systems and software systems, and it was easy to recognize the similarities across the two worlds. Applying practical principles to software design, ideas that are common-sense in other engineering fieldsmade sense in software systems too. It never occurred to me not to treat software development as engineering or to develop a system without a design or without a plan. I saw no need to compromise on my conviction, or to give in to expediencies because doing the right things just worked, and the appalling consequences of not doing so were plain to see. I was fortunate to have great mentors, to be at the right place at the right time to see what worked and what did not, to have the opportunity to participate early on in large critical efforts, and to be part of cultures of excellence.

In recent years I have noticed that the industrys problems are getting worse. More and more software projects fail.The failures are getting more expensive in both time and money, and even completed projects are further afield from their original commitments. The crisis is worsening not just because the systems are getting bigger or because of the cloud, aggressive deadlines, or higher rate of change. I suspect the real reason is that the knowledge of how to design and develop software systems is slowly fading from within the development ranks. Once, most teams had a veteran that mentored the young and handed down the tribal knowledge. But these people are retiring.With that, the rank and file is left with access to infinite information while having access to zero knowledge.

I wish there wasjust one thing you could do to a fix the software crisis, such as a process, a development methodology, a tool, or a technology. Unfortunately, to fix a multi-dimensional problem you need a multi-dimensional solution. In this book I offer a unified remedy, righting software.

In the abstract, all I suggest is to design anddevelop software systems using engineering principles. The good news is that there is no need to reinvent the wheel. Other engineering disciplines are quite successful, so we can borrow theirkey universal design ideas and adapt them to software. You will see in this book a set of first principles in software engineering, as well as a comprehensive set of tools and techniques that apply to software systems and projects.To succeed, you have to assume an engineering perspective. Ensuring that the software system is maintainable, extensible, reusable, affordable, feasible in time and risk are all engineering aspects, not technical aspects. These engineering aspects are traced directly to the design of the system and the project.Since the term