Beyond Data (part 2): Big-O snowballs, copycats, and super levers

As computer scientists, one of our first lessons is about “big O” complexity of software. It’s used to understand the expected time for a program to run. “Big O” theory tells us that it’s the order of magnitude of a system—like O(n^2) or O(n)— that matters, much more than smaller factors. To understand the parts of a computer program that dominate the time required to run it, we know not to focus on the tiny parts of the system that irrelevant to the overall behavior. Measuring the performance of a line of code to the fourth decimal place, when it only runs a ten-thousandth of the time that other code does, is wasted time.

As introduced in Beyond Data Part 1, we need to apply the same kind of insight to analyzing data, whether it be for business intelligence, decision support, dashboards, or other systems.

Think about your company as a snowball rolling downhill.  If you start five balls down a slope, they’ll all roll a little while, but some will happen to hit stickier snow, and get a little bit bigger than the others.  Those heavier snowballs will have just a bit more momentum, so they’ll roll a bit further, and use up all the snow, gaining a bit more size in the process. Which makes them heavier, which makes them roll faster.  You get the idea: in general the fortunate few snowballs will end up much bigger, leaving the smaller ones stuck at the top of the hill and others much bigger and much faster.

In markets for media, a similar thing happens.  Instead of spending hours reading the reviews, we’ll take a short cut and  ask “who else has liked this?”, “have I liked other books by this author?,” and think “well, I liked Part 1 of the trilogy so Part 2 is probably pretty good.”   This means that a musician just a little bit worse than Mozart dies in obscurity, and the book that’s just a little bit worse (or a little bit less lucky) has much lower sales figures.

This effect is exacerbated by cognitive overload: thousands of books on Amazon and six films at the Cinexplex overwhelm our decision-making ability, so we fall back on simpler ways to choose.

This snowball effect:the “winner-take-all” has a gargantuan effect on markets and business success.  It’s why we’ve ended up with only two operating systems companies—Apple and Microsoft—battling it out for most of the market. It’s why blockbuster authors remain on the bestseller list for many years, yet new authors writing good books can’t make a living.

Beyond the business world, concepts like the “vicious cycle of poverty” compared to the “virtuous cycle of prosperity” capture how cities and countries understand the dynamics of their situations.

Most of this is well understood. It’s the mature and well-respected world of system dynamics. What’s less widely appreciated is the fact that, as we use more and more data to manage businesses and governments, we need to take a fundamentally different view of the data that reflects our situation than we do when things are more linear.

In particular, what matters is not that our management data is complete or correct, but whether it allows us to identify that business decision that will nudge us “over the edge” into a virtuous cycle.    For instance, a small additional investment in doctor training may move a medicine from 50% to 90% effective….a film launched at Sundance may pick up buzz that catapaults it into mainstream awareness.   The holy grail here is to identify these “nudges” whose impact is disproportional to their cost.  Maybe we’ll call them “super levers”.

Unfortunately, all the data in the world won’t help.  You can spend years cleansing your join keys, replacing your missing fields, measuring things more precisely, but without an understanding of the underlying system, you won’t find that “sweet spot”.

Here, good data is not sufficient.  It’s also not a necessary condition.  Indeed,

It is for this reason, more than any other, that we’ll be seeing the excitement about Big Data start to welcome its natural siblings in the next year or so: with an increasing focus on systems analysis and nonlinear thinking.  Those organizations that can embrace DI models are already dominating.  They’re just keeping the “secret weapon” of nonlinear systems thinking to themselves as long as they can.