The Hollywood Model ... Essays Home

Cargo Cult Software Engineering

I used to think that software engineering, or more precisely software engineering project management, was a bit like a medieval doctoring; a large body of theoretical "knowledge", that wasn't much use at the day-to-day level. I don't know about the rest of you out there, but the computing science, and even management science theory I've come across isn't much use on a day-to-day basis. I just seem to shamble along, making it up as we go along.

Of course, there have been some classic descriptions of this state of affairs, the best being the book "The Soul of a New Machine" , and the most recently a book that tells us the "Dynamics of Software Development" . "Soul of a New Machine" is remarkable in that it depicts with graphic detail what I have come to believe is a standard software development management strategy (but one you won't find in the management thoery books). This is the "hire very bright young engineers and work them into the ground" theory (engineers because they are essentially pragmatic above all else, bright and young because young males al least are ego-driven, and don't know when to give up). Of course, you have to pay them a fixed wage that doesn't recognise overtime hours. My general experience with senior management is that they would sell their staff for dog-food if they thought they could maximise profits that way (and if it were legal).

This is probably why the dismal statistics of project failure continue to not improve. Instead of ten years of experience, we have had ten lots of a single years experience; each project uses different technology where the key issues and decisions are different. Couple this with the endemic high-turnover that characterises the industry, and improvement is almost impossible. Imagine a sea-faring industry, where three sea-cruises used successively sail, steam reciprocating engines, and atomic powered turbines; imagine the potential for disaster if the Captain, First Mate, and Engineer had to learn as they sailed along, with almost a brand new crew each time.

Of course, intelligent dedicated people struggle to improve this situation. I am irresistably reminded of the Cargo Cults of Papua-New Guineau after WW2. Stone age people had seen cargo planes deliver immense wealth, and so they tried to recreate the necessary conditions for this to happen again. They built replica airstrips, control towers, radar, etc, out of bamboo and thatch. I suggest that amusement is misplaced; they were doing their best to achieve great outcomes, with a very imperfect understanding of what really goes on beyond their ken.

I suggest that a similar process operates within the software industry today. We have typical examples:

  • a book that tells us how CA achieves greatness (even though CA are unkindly referred to by others as "bottom feeders", a reference to their ability to buy smaller struggling companies, and incorporate them into CA);
  • a book that tells us the secrets of Microsoft Management
  • the SEI Software Maturity Model that tells us the important things that must be done to achieve Nirvana (or level 5)

The Hollywood Model

I am deeply suspicious that most (or maybe all) of these models are misguided at best, and self-deluded at worst. My person belief is that a better model for large scale software engineering is to be found in Hollywood. A significant fraction of large blockbusters are disappointing flops; occasionally a small art movie will attain overwhelming success. Some production houses concentrate on maximizing return on creativity by issuing sequels to previous successes. Hollywood has also had at least one technology revolution that disorganised existing power structures, when the talkies arrived.

Jim McCarthy advises us in his Maxim #27 to "Be like the doctors"; doctors tell us that even the best executed of operations still result in failure for inscrutable reasons. I disagree with his analogy although not his advice; doctors still probably have a much better record than the project managers of large software projects. Another reason why I prefer the Hollywood analogy to the doctor analogy, is that doctors strive to develop techniques that can be routinely performed, so that what was a miracle today is commonplace tomorrow. On the other hand, Hollywood recognises and accepts as part of the business the scope for a single director to fail monumentally or to achieve greatness, by directing the efforts of many other creative people. This highlights another crucial point. The classic definition of a manager is "one who achieves through the efforts of others"; why then do we not talk of great movie managers, but of great movie directors or producers?

The reason is of course that to a depressingly large extent, management has no fire, passion, zeal or creative spark. As least some in the military have the sense to reject the term "Battlefield Management" (common in discussions by civilian computer people on tactical and operational automation); they insist on stressing the leadership aspects, and quite rightly so, in my opinion. It may be my cynical nature, but the one thing that has struck me in my limited exposure to the MBA-style case-book study of company success and failure, has been lack of attention paid to the individuals involved. It is precisely this aspect that can be least quantified, or taught in a lecture theater.

Focussing on the crucial role of the leaders in software engineering will lead to different strategies. We will not try just to ape the techniques of the towering giants of the past (Harlan Mills, Fredrick Brookes, even Bill Gates); we must develop leadership. Again, military experience suggest that within reasonable parameters, leadership can be cultivated, if not taught. The best training by the Catholic Church to its early missionaries was to "love God, and do what you will"; the Church knew that it could not train people for the strange landscapes and society its missionaries would confront. What it could do was to motivate them by a belief system, give them general skills and knowledge, and then let them have free rein.

Unfortunately, the prognosis is depressing. Developing leadership skills takes time, and time means money; it also means accepting mistakes at times along the road. Both are unlikely to find a welcome home in an industry driven by the regime of fixed-price contracts awarded to the lowest bidder. Technology training will always be easier to justify (C++ last year, Java this year), and require less effort from the harrassed and over-worked senior staff of any company. Australia has always had a culture that scorns the "tall poppy", so it is important to be aware that inspirational leadership is not necessarily tied to overt charisma.


So what do we do about this individually? I've got no concrete suggestions except that maybe we should take refresher courses in anthropology or psychology, before we take that next techno-fix. We should carefully study the techniques of those around us who are effective at creating elegance out of the inherent chaos of a software team, but should remember that what works for them might not work for us. We should be prepared to allow ourselves a touch of extravagence gesture, because "The world's a stage", and we have to been seen to be acting upon it.

"The Soul of a New Machine", Tracy Kidder, Penguin Books, ISBN 0 14 00 62491

"Dynamics of Software Development", Jim McCarthy, Microsoft Press, ISBN 1-55615-823-8

"21st Century Management, The Revolutionary Stragegies That Have Made COMPUTER ASSOCIATES A Multibillion-Dollar Software Giant", Hesh Kestin, Alantic Monthly Press, ISBN 0-87113-524-8

"Microsoft Secrets, How the World's Most Powerful Software Company Creates Technology, Shapes Markets, and Manages People", Michael Cusumano & Richard Selby, HarpersCollins, ISBN 0 00 255692 8

© Don Cameron.