Software’s Glorious Revolution

by Greg Olsen

After many months of reading, I finally finished Neal Stephenson’s System of the World  (the last volume in the Baroque Cycle trilogy), and I discovered an unexpected reward - the realization that we are in the midst of a profound revolution in the software industry and that we are witnessing the establishment of a New System of the Software World.

The Baroque Cycle explores the economic, political, & religious structures of Europe around the end of the 17^th century and the profound changes that led to a transition from a land-based economy ruled by monarchs and anointed nobles and steeped in superstition, to a system governed (to a much greater degree) by capitalism, democracy, and rationalism.  The “Old System of the Software World” that I grew to be familiar with was ruled by its own monarchs and nobles, and had its own superstitions. The Powers carried names like IBM, Oracle, Microsoft, Sun, and OMG and the nobles carried the title “architect” (see Joel’s discussion). We possessed hope and faith that the directives of the architect nobles would protect us from the plagues of instability, unscalability, and inextensibility.

Though I often griped, I learned to live within the structure provided by the ordained Powers.  We made our treks to JavaOne, the Microsoft PDC, and other events to receive the word as written.  We lived with a pace of new technology arrival dictated by the Powers and their committees of architects. We were sometimes forced to swear allegiance to one of the Powers and to purchase the requisite tools and literature from that Power in order to use their infrastructure.  For the most part, we were well-paid, latte-drinking vassals.   

Somewhere over the last five years or so something changed. Though I can’t think of a specific event to parallel to the Glorious Revolution of 1688, it is clear that some form of revolution did occur, and that a New System of the Software World is in the process of being established. The powers of the Old System are still around, but they no longer dictate what infrastructure and tool options are available to software developers. Today, new capabilities come into being because there is a demand and because there is someone willing to meet that demand – most often through the vehicle of an open source project, or through an Internet-based service.

Eric Raymond  eloquently describes the first stage of this revolution in the Cathedral and the Bazaar .  He explains how the open source movement wrested away the singular control the Powers once had over the production of significant software infrastructure.  Projects like Linux and MySQL demonstrated how decentralized efforts, led by small groups or individuals could produce low-cost, high quality versions of existing infrastructure products like operating systems and databases.  More recently, we’ve experienced the second stage of this revolution where large numbers of small players are both defining and delivering significant new types of primary software infrastructure - key evidence of a New System of the Software World.

One of the most visible examples of this new stage is the overthrow of EJB 2.0 by the lightweight/alternative container movement (e.g. Spring , Pico, Nano, Ruby on Rails, LAMP  approaches) and alternative persistence approaches such as Hibernate.  EJB 2.0 represented one of the great fortresses of the elite architect class and was a key point of control for the non-MS Powers.  EJB 2.0 also exemplified excess, “an architecture for architects”, and it inspired a “we really don’t need all this stuff” revolt.  Simpler, better alternatives arose not through some organized community process or through a cabal of the Powers, but through the actions of small groups of motivated people.  Instead of the one or two anointed options, consumers of deployment infrastructure now have a diversity of product and provider options to choose from in what is truly an open market.

The current Web 2.0 wave (whatever you think it is) is additional evidence of the new System of the Software World.  Key architectural decisions around how to enable highly interactive client experiences over the web (e.g. using AJAX , Laszlo ), how to support effective collaboration (e.g. wikis, IM, VoIP, email, etc.), and how to compose diverse services (e.g. mashups) are now being decided by thousands of experimenters, dabblers, inventors, and microISVs – not by committees of architects in standards bodies or at large software vendors.  Software infrastructure has become diverse and multilayered and is accessible to a much larger audience than in the past. 

To understand the new system of the software world, we must have some sense of why the change occurred.  “Inspired acts by a few individuals” account for part of the answer, but significant environment factors were clearly at work.  As Bob Walsh and many others have noted, the Internet did change everything.   At one time, only large organizations with large marketing budgets could effectively distribute software capabilities.  The combination of open source distribution sites, search engines, tagging, blogging, and other mechanisms, lets even the smallest would-be provider find a worldwide audience without huge capital outlays. 

Like the old world Stephenson describes, the Old System of the Software World was driven by forms of superstition and alchemy.  In the not too distant past, issues such as memory management, threading and concurrency, and remote object invocation were central concerns of software implementations and only the high priesthood of computer science could truly master these concepts.  Complex systems were built with very fine grained building blocks provided by the OS. Constructing robust systems that could scale was extremely difficult and uncertain.  Architectural dogmas provided a means of addressing this uncertainty – adherents to OO, 3-tier, or whichever dogma hoped their allegiance would provide them salvation.  The Powers effectively (and often legitimately) used fear of system failure to dissuade people from using un-anointed infrastructure options.

Today systems are built with higher level, more coarse-grained abstractions on top of base infrastructure that addresses many of the robustness and scalability issues of the past.  In addition, automated test capabilities and practices have improved dramatically.  The net result is that systems built today are more predictable.  New approaches can stand on their own merits.  Instead of reliance on the mystical wisdom of some architectural cleric, we can validate the robustness and scalability of a given system through objective measurement and analysis.

Another key environmental factor is the change in life-cycles associated with software based systems.  In the past, the focus was on large systems that took a long time to design, build, and deploy and that were expected to operate for long periods of time.  Virtuous approaches were those that embodied generality, elegance, and completeness. That era better suited the workings of the Powers and the architect class. Today, the market is driven by smaller, more rapidly changing needs. Solutions need to be rapidly deployed to even be relevant in many circumstances.  Good enough solutions that can be built quickly win over ‘more perfect’ approaches that take longer to build.  Virtuous systems are those that embody pragmatism, simplicity, and agility.  The responsiveness that an open market of diverse providers can deliver is a distinct advantage in today’s environment.

Like the early 18^th century British monarchy’s move from  absolutism to a constitutional monarchy, some of the old world Powers are finding ways to adapt to and even to exploit the new System.  In an attempt to achieve commercial success without architectural control (and much to the dismay of some of their own architects) these companies are embracing independently developed infrastructure from the open source community and elsewhere, and are developing synergistic product and service offerings. Others simply try to use these offerings defensively in their conflicts with other Powers, and still others appear to be buying up these types of offerings because they’re not sure what else to do.  All of the Powers seem to understand that they must adapt to rapidly changing infrastructure that they cannot control or else become irrelevant.

The New System does come with new challenges.  Software developers need to be able to keep up with the cacophony of options being produced and to understand both technical and non-technical (e.g. legal, support) issues associated with the infrastructure choices they make.  Diversity can bring with it increased confusion and balkanization, making the coordination and integration of independently developed systems more difficult.  These issues, however, are mitigated by the new System’s ability to rapidly adapt to whatever new challenges arise.

One of the most exciting aspects of the New System is the evolution of infrastructure-as-a-service.  There are now numerous service APIs from providers such as Google, Yahoo, Amazon, Ebay and others that are used in the composition of other service delivered applications.  Today, it is even possible to create and deploy an entire software application without any installation of software or without highly advanced programming skills using services like Salesforce’s AppExchange, Intuit’s Quickbase, JotSpot, or Ning .  Though these types of services are still in their infancy, they promise to extend application development in both significant and surprising ways.

The New System of the Software World is something that should be celebrated – by people who build information systems and by people who use them. Greater choice and more rapid innovation is making the lives of both users and developers better.  This new liberty will bring with it new business opportunities and new uses of software to meet our application needs.

Comments

[just.a.guy]

mystical clerics of old deliberately exercised control over the masses by claiming to be privy to arcane knowledge that the masses could never hope to understand. the reality was that they themselves didn't truly understand the mysteries of the universe, but were simply charlatans preying on ignorance and fear.

in the case of computer systems, the mysical architects of old really DID and DO understand the mysteries of the computing universe, and were attempting to codify those mysteries in architectures that would allow relative acolytes to build advanced systems without totally blowing it.

while there are some vested interests in maintaining that power base among the old guard architects as there were with the old guard mystics, the difference is that one group was preying on ignorance and fear from day one, and the other was just trying to help people out.

the shorthand version of this is, computer science just isn't rocket science any more. hell, rocket science isn't rocket science any more.

[Michael Wagner]

Wow - lots to think about. Helpful post.

For 5 years I was an Internet business strategist and experienced some of "the high priesthood of computer science" in the IT departments of large corporations. I kept by-passing them and working directly with business unit leaders who wanted to get things done on "the Net". It was an interesting time and dynamic - especially since my educational background revolves around history and ancient languages.

Plus this posting has me wanting to set aside time to read Neal Stephenson!

Good stuff!