Often 10% of the data contains 90% of its value in solving the problems we need solved. This means that we can waste enormous resources cleansing, migrating, and managing that 90%, with little to show for it. What’s worse, we can miss the 10%: a big disconnect to achieving our objectives. Continue reading
The Internet of Things…Machine Learning…Self-Driving Cars…Artificial Intelligence…Big Data…Smart Cities…Decision Intelligence…my friends talk to me about their excitement about a whole lot of trends. But which ones are real, and which will fizzle?
Before founding Quantellia, I spent six years as a technology analyst, where I was privileged to have an inside look at how tech trends boom and bust. I learned a few important lessons.
No matter whether you pull for Donald or Bernie or are an “occupier” or a “tea partier” or anywhere in between, you have to admit that we have a problem. The system is broken, due to gerrymandering, big money, or any other possible reason that you can name. We have issues, such as “income inequality,” or “climate change” that are of major concern to one of our major parties while the other is somewhat unconcerned to the extent that some within the party don’t think that the problems exists; we have one party claiming credit for what it sees as tremendous success of its health care bill while the other party constantly tries to repeal or dismantle it.
Gone are the days of the bipartisanship that brought us the Civil Rights and Voting Rights bills (even in the face of southern segregationists) and Environmental legislation. We have a booming economy in terms of Wall Street and corporate profits and a stagnant one in terms of worker salaries and buying power. Continue reading
Looking back on the presidential election of 2012, one view of the Obama win is to attribute it to his team’s understanding of a phase shift in electoral dynamics: Democrats looked at historical turnout numbers and perceived a systemic change; in contrast many believed that Republican certainty in a Romney win was based on a reasonably expected regression to the mean. This is the essential idea behind “data from the future“. We ignore these principles in this system, as in many others, at our peril.
In light of this history, it’s worth asking if the fundamental dynamics of how elections are won is shifting this year again. Continue reading
When I was helping to develop the MVS/XA mainframe operating system at IBM in the 1980s, we had a disciplined process for software development. We knew that a bug fixed in requirements was a hundred times cheaper than if we repaired it after it was out in the field. So we were careful and diligent, writing thorough specifications documents, then carefully crafted design docs, and only then could code begin.
All good software engineers know this is the way to go. Or at least that’s what we’d always thought.
But it turns out we were wrong. And not in a small way. Continue reading
Inattention to critical knowledge is an old problem. Lessons are forgotten, near misses are ignored, caution is dismissed, disasters result. Titanic. Bhopal. AIG. Katrina. Fukushima. And on and on.
Knowledge Management (KM) is supposed to make the right information available to the right people at the right time in the right form—and to the best level of certainty possible—for making the most appropriate decisions when and where they are needed. KM should also direct the attention of decision makers to critical information and help them make sense of it. The bigger the stakes, the more situational awareness and mindfulness are needed. Continue reading
Many companies are facing the prospect of steep increases in the cost of energy in the coming years. In response, many are looking at alternative energy sources. However, navigating the transition to this new world contains hidden dangers, so an evidence-based modeling approach can make a big difference. This article looks at this decision-making process through the lens of US cable operators, to understand the specific decisions they face.
The goal of World Makers is to encourage people to build computer simulations of the world. This includes simulating water, weather, crops, land use policy or anything else. Models can be regional or global, simple sketches or full blown simulations.
The classic game ‘Sim City’ by Will Wright is perhaps the best known example of a computer simulation. It lets people build their own imaginary city from the ground up, placing roads, homes and services and measuring their success against the happiness of the population. The goal here is similar – but real – with real data, real stakeholders and real outcomes.
There’s a scene near the beginning of the Oscar-winning Shakespeare in Love:
The theatres, we have heard, are all closed by the plague. And then:
HENSLOWE: Mr. Fennyman, let me explain about the theatre business.The natural condition is one of insurmountable obstacles on the road to imminent disaster. Believe me, to be closed by the plague is a bagatelle in the ups and downs of owning a theatre.
FENNYMAN: So what do we do?
HENSLOWE Nothing. Strangely enough , it all turns out well.
HENSLOWE I don’t know. It’s a mystery
As computer scientists, one of our first lessons is about “big O” complexity of software. It’s used to understand the expected time for a program to run. “Big O” theory tells us that it’s the order of magnitude of a system—like O(n^2) or O(n)— that matters, much more than smaller factors. To understand the parts of a computer program that dominate the time required to run it, we know not to focus on the tiny parts of the system that irrelevant to the overall behavior. Measuring the performance of a line of code to the fourth decimal place, when it only runs a ten-thousandth of the time that other code does, is wasted time.
As introduced in Beyond Data Part 1, we need to apply the same kind of insight to analyzing data, whether it be for business intelligence, decision support, dashboards, or other systems.
Think about your company as a snowball rolling downhill. If you start five balls down a slope, they’ll all roll a little while, but some will happen to hit stickier snow, and get a little bit bigger than the others. Those heavier snowballs will have just a bit more momentum, so they’ll roll a bit further, and use up all the snow, gaining a bit more size in the process. Which makes them heavier, which makes them roll faster. You get the idea: in general the fortunate few snowballs will end up much bigger, leaving the smaller ones stuck at the top of the hill and others much bigger and much faster. Continue reading