The antifragility of the open source bazaar

When I was helping to develop the MVS/XA mainframe operating system at IBM in the 1980s, we had a disciplined process for software development. We knew that a bug fixed in requirements was a hundred times cheaper than if we repaired it after it was out in the field. So we were careful and diligent, writing thorough specifications documents, then carefully crafted design docs, and only then could code begin.

All good software engineers know this is the way to go.   Or at least that’s what we’d always thought.

But it turns out we were wrong.  And not in a small way. 

Today, open source software—usually created for free by thousands of people, and open to modification by everyone—is an increasingly important approach.  It’s hard for me, trained as I was, to understand how this can be so.  Yet, strangely, it does, and so last week, IBM announced some big open source initiatives, including its new mainframe operating system LinuxONE, a free-for-developers cloud, and the release of a quarter of a million lines of code to the open source movement.

A few years after I left IBM, the company became one of the earliest adopters of open source.

But how can IBM, along with thousands of its customers—many of them the old guard of the Fortune 100—embrace systems created, essentially, in anarchy?  By what possible mechanism can chaos lead to the reliable systems that increasingly form the backbone of computing as we know it? 

The Apache web server (whose development arguably created the internet as we know it today), the Linux operating system, a chunk of Google’s code (20M LOC), some of facebook, and more, are now all open source.

There’s something profound and quite wonderful going on here, and it goes well beyond software, to potentially changing the fabric of how we solve problems in a complex world.

Rationality in anarchy

The answer to the open source puzzle is best explained by Eric S. Raymond in The Cathedral and the Bazaar—first an essay and later extended to a book and also covered in the 2001 documentary, Revolution OS.   Raymond was puzzled to observe the early success of Linux, and set out to understand how it could possibly work.  “I’d been a software engineer for nearly 15 years…and according to all the rules I knew about control and complexity and keeping your project group small, having closely managed objectives, it should have been a disaster…Instead, it was something wonderful.”  He discovered that, like peer review of scientific publications, open source software is subjected to thousands of eyeballs to find bugs, and hands to fix them.   And that this mass collaboration structure creates powerful and robust systems.

Indeed, the underlying mechanism is best understood as an antifragile system.  We might equate the Cathedral to Taleb’s office worker, and the Bazaar to his taxi driver.  The taxi driver makes decisions every day about where to find the next fare; he must flex within his market. The office worker may lose his job and his employability with a single bad performance review, and recovery will be much harder.  Traditional software engineering is like that: built with one purpose in mind, inflexible to change.   And so our systems are fragile; they break as requirements change, and don’t recover easily.

Yet breakable, changeable systems are paradoxically strong.  Like an immune system, or the goddess Akhilandeshvari, there’s something powerful about being “never not broken”.  So many of today’s most important systems today follow these principles.  For instance, the idea was baked into Jimmy Wales’ thinking before he started Wikipedia.

Christine Peterson, who is widely recognized as creating the phrase “open source software”, told me that, “The very first software was open.  The theory was that we make money selling hardware.  Software was just something you gave away to make it happen.  We’re discovering what people knew in the early days, which was that collaboration is really useful.”

Beyond software

But there’s more to come.  There’s an emerging consensus that massive collaboration, as enhanced by modeling of reality, not just software, has a transformative potential in many realms, including economics, the environment, development, and the cross-domain linkages between them.

Combining the antifragile characteristics of massive open-source software development with open-source-curated information from Wikipedia, along with open data, has the potential to get those thousands of eyeballs looking at the complex systems that make up our most wicked problems, discovering the “bugs” of unintended consequences, and much more.  Says Christine, “First the term was ‘open source software’, then we started just saying ‘open source’ for a huge variety of other things, conveying the general value of this idea of collaboration’.  And the idea is taking shape in a myriad of forms, including the folks articulating the “peer to peer (p2p) revolution”.  Indeed, Simon Cicero goes as far as to say that “most of the global industries…are just sitting there waiting to be disrupted” by what he calls platform based peer production, or PBPP.

If IBM—once bastion of old-style computing—is willing to move from its own cathedral into Eric Raymond’s Bazaar, then we should take this pattern very seriously.   Open-source systems are surprisingly robust, and this goes well beyond software.   As for me, I’m inching closer to opening my first repo on github, and training dozens of neural networks on a 16-core server.  And that’s a long way from MVS, but in the best kind of closed loop, on my way back what feels surprisingly similar to a mainframe, updated for the 21st century.