The New Alchemy

Software development has given rise to entirely new alchemists, with entirely new approaches, including the "Powerful Process" and "Magical Language" schools. The one thing that both of the schools have in common is a belief that there exists some incantation or ritual that will transmute incompetent developers into great developers. They'd be better served trying to turn hydrogen into gold, since that's at least physically possible (if horrifically impractical). Let's look at these schools a little closer and we'll see why ITIL and CMMI (and a lot of "agile" processes!) are incompatible with Agile, and why Paul Graham was right and wrong about LISP being a secret weapon in software development.

Arcane Rituals: The Powerful Process School

Acolytes of this school believe that if they can just find the one true process and follow it with sufficient rigor that it will guarantee that they will be guaranteed success in all of their software development endeavors. The holy grail of this school would be to find the process that lets them say, with a high degree of certainty, that it will take x person weeks to build a particular system, regardless of who comprises their development team. In effect, magically turning even incompetent developers into highly productive developers. It's this fixation on process over people that makes them ideologically incompatible with Agile. Sadly, even many proponents of Agile fall prey to these beliefs. You can tell if an "Agilist" has joined this school by observing their response to failed projects. If the response is to defend the process by saying, "You did it wrong! You didn't do thing x as prescribed by the process!", then you know you've met an agile alchemist.

Occasionally, processes are put into place that do succeed in making the composition of a team a minor factor in how long it takes to develop software. Unfortunately, this isn't accomplished by improving the worst developers, but by dragging everyone down to their level. There is some good news, though! The best developers aren't held down at severely depressed productivity for long; they leave and make predicting the outcome of future efforts easier by leaving them with no chance of success.

Words of Power: The Magical Language School

Developers are often counted amongst the acolytes of this school, constantly searching for the programming language, IDE or new technology that'll make hard problems easy to solve and easier to implement. That's certainly understandable, since a lot of things in software development are really hard and many developers love tinkering with new things. Certain commercial entities exacerbate the problem to profit from the sales of books, tools, and software by hyping every new thing that comes along. Unfortunately for developers (and fortunately for snake-oil salesmen), some problems are just hard! No tool or language will ever make the intrinsically hard problems easy, at best they can stay out of the way and let the easy parts of the problems be easy.

Earlier, I mentioned that Paul Graham was right and wrong about LISP being a secret weapon, and you may be wondering just how that could be. To be sure, LISP is a very powerful language, very well suited to extending with powerful DSLs and someone comfortable with it could be extremely productive. That said, LISP alone isn't enough to make one team an order of magnitude more productive than another. What does let LISP make teams significantly more productive is that it's a great filter. Requiring LISP automatically rules out everyone other than really good developers. Since LISP is not a language that's a sure-fire path to a job, only people who actually care about software development know anything about it. Adding to that, LISP has a fairly high barrier to entry, by requiring an understanding of functional programming and recursion. Basically, any team that's using LISP is most likely a 10x team, and that is the secret weapon.

So, if neither languages or processes can work the magic, what _can_ we use to improve our odds of success at software development? The answer is simple; hire really good developers who work well together. Then tell them what you need them to do, and work with them to clarify any issues that come up and to make sure that they know when needs are changing as soon as possible.