Chuck Connell Replies to Eric Raymond
By Chuck Connell
Over the past few years, Eric Raymond has written and revised a famous essay about open source software titled The Cathedral and the Bazaar. This essay is popularly known as CatB. I wrote an essay titled Open Source Projects Manage Themselves? Dream On (MT) in which I challenged some of the basic tenets of CatB. Raymond responded to MT with this reply (R1). Below is my continuation of our discussion.
In R1, Raymond writes, "[Connell] confuses my observations about the structure of the open-source community in the large with claims about the organization of individual projects in the small. This confusion leads him up several garden paths and down a number of blind alleys."
It is true that my comments in MT are based on just two open source projects, fetchmail and Linux. And it is true that I made some conjectures about the general nature of open source projects from these two cases. But this is true of CatB as well. Raymond makes many statements about the nature of open source development, based only on the same two projects. More significantly, the whole world read CatB and drew broad conclusions from it about open source programming and how to run these projects. Netscape and other companies changed their business models to match the ideas in CatB.
The moral here is that we should expand the sample size of projects we study before making firm conclusions about the base principles of open source software. (By "we" I mean everyone in the open source community and everyone interested in its outcome.)
Intention of CatB
In R1, Raymond writes, "The confusion begins immediately with Mr. Connell's title, 'Open Source Projects Manage Themselves? Dream On.' Of course they don't, and nobody who has ever participated in one would dream of claiming that they do."
It appears that Mr. Raymond has not read the dust jacket of his own book. It says. "…the development of the Linux operating system by a loose confederation of thousands of programmers -- without central project management or control -- turns on its head everything we thought we knew about software project management." This assertion is precisely why CatB is such a famous essay. People look to the open source method as a new way to create software and, in the process, break the tyranny of Microsoft.
Now, I am pretty sure that someone in the marketing department at O'Reilly & Associates (rather than Raymond) wrote the blurb on book's cover. But an author has a responsibility to ensure that the cover of his book basically squares with the message inside. Is Raymond claiming that the above quote on the jacket is 180° opposite from the theme inside?
Organization vs. Control
In R1, Raymond states, "Mr. Connell writes as if self-organization and strong central control … are incompatible, but he is crucially wrong in this. Individual projects self-organize because the participants choose to be there -- they select themselves, and they choose to follow the project's benevolent dictator (or else they leave)."
Raymond has a partially valid point here. Being self-organizing and having central control are not completely incompatible -- sort of. Imagine a group of random castaways marooned on a dessert island. Initially there is no hierarchy to this group. But the group may choose to elect a leader or committee to guide them. They would choose as leaders the people who seem most able to help them survive. The castaways would be self-organizing, yet have established some central control. This is analogous to an open source project that allows one member to serve as project leader, because it is in everyone's best interest. (Gee, maybe this would make a good TV show…)
The argument only goes so far however. Over time, leaders get stuck on certain ways of doing things and become accustomed to their power. They are resistant to change. The ability of the group to organize itself diminishes, even though it began in an egalitarian way. Similarly, open source projects are not very self-organizing two years after they start. From a newcomer's point of view, things are not as democratic as they once were.
Of course, as Raymond points out, unhappy newcomers are free to fork the project if they don't like what the leaders are doing. But this only proves that the project is no longer self-organizing -- it is now two projects.
In R1, Raymond writes, "Above the individual project level, the community is also self-organizing. … There is no manager or management hierarchy that plans for the open-source community as a whole. Instead, coordination between projects happens through a horizontal trust network of relationships between individuals -- a network which genuinely does look like Mr. Connell's "bazaar" diagram (his figure #3)."
What Raymond says here -- the lack of macro-level control over the open source community -- is certainly true. This is a moot point however, since the same could be said for corporate America as a whole. No one plans the overall direction of the US economy or the computer industry. (The Federal Reserve Board does try to steer things along, but we do not live in a socialist system.)
Leaders vs. Managers
In R1, Raymond says, "It is not leadership that the open-source community has rejected -- it is conventional management hierarchies and the pointy-haired boss."
No argument from me here. I have worked for many bad managers and hope not to repeat the experience. Enlightened software companies also try to find managers who are technically knowledgeable and compassionate. The open source community is not the first to rebel against bad management.
In R1, Raymond says, "Mr. Connell makes another basic error in attacking the open-source debugging process. He writes as though the most important and limiting factor in debugging is in choosing non-conflicting fixes once the problems are understood. This is wrong, as all open-source developers understand and my paper points out via a quote from Linus. In fact, the hard part is understanding the nature and etiology of each bug -- and this is precisely the part that parallelizes well. Once that hard part is done, reconciling fixes is pretty easy."
There is no question that it is a great help on any project to have hundreds of volunteers looking for bugs and suggesting fixes. The aspect of debugging that is most difficult varies with the software (and bug) at hand however. Sometimes finding the exact nature of a problem is difficult, sometimes making the fix is harder. With his many years of programming experience, Raymond must know this.
Reconciling multiple fixes also is sometimes easy and sometimes hard. It partially depends on the quality of the code. Well-designed code is simpler to fix without breaking something else. It also depends on luck. Sometimes fixes harm each other, sometimes not. Again, Raymond certainly knows this.
The Great Brooks
It R1, Raymond says, "[Connell's] observation that individual project organization resembles a Brooksian surgical team is independent of the serious mistakes elsewhere in the article and worth followup; by coincidence, I wrote the same thing in the (not yet published) revision of CatB I'm preparing for the book's second edition."
I find it fascinating that so many writers in the software engineering field keep returning to the wisdom in the Mythical Man Month by Fred Brooks. Sometimes I think that there was only one book ever written about software engineering, and the rest of us just keep rewriting it. :-)
Raymond writes, "My advice to Mr. Connell is this: spend some time in the trenches. Join a project. Observe the process in action. This paper was badly wrongheaded -- but a year from now, you might have something important to contribute to the subject."
My advice to Mr. Raymond: Keep a broad perspective. Don't fall too much in love with your current methods. Open source software is an important development in the computer world. So were high-level languages in the 60s, structured design in the 70s, personal computers in the 80s, and object-oriented design in the 90s. All of these movements promised to cure the ills of the software industry. (And all have helped to some extent.) Open source software is the revolution du jour of the 00s. It has pluses and minuses, just like everything else.
Biography: Chuck Connell is a Domino/Notes consultant with 15 years of experience. He has taught software engineering at Boston University and writes frequently on computer topics. Chuck can be reached at www.chc-3.com.
September 2000 -- Published by IBM/Lotus Developers Network.
January 2007 -- Rehosted on Chuck's server, after IBM dropped older articles from their site.