What to do when complexity starts to eat you alive: in law, government, and organizations
I had a house once with a lot of stuff, and a big basement. Sometimes when I needed to find something quickly – a Philips screwdriver maybe, or a glue gun – there were just too many places to look, and it would have taken hours. So, I’ll confess, there were times I just went and bought a new screwdriver. Which added to the pile, and made it worse for the next time around. At the time, I always seemed to be too busy to take the time to organize things so this would stop happening. So for a few years it just kept getting worse.
I’m starting to see this pattern more broadly.
In a recent consulting project, I learned that a large organization in a slow-moving industry was suffering from the same problem with its procedures manuals. These documents specified how certain tasks were to be performed, and they’d reached the point where there were contradictions. Instead of taking a deep look at the inconsistencies, this organization sometimes created new rules meant to clarify the old ones, but the result was just a basement full of more stuff: a band-aide patch, that over time just made the problem worse.
In The Rule of Nobody: Saving America from Dead Laws and Broken Government, Philip K. Howard explains that the same is happening to modern government. If you think of government as sort of a “legal machine”, of legal gears and presses, getting ever larger with new regulations and programs, then Howard claims government leaders are not decision makers but rather act more like mechanics, “scurrying here and there to attend to the legal gears and machinery”. The theory is that, somehow, decision-making can be relegated to an automatic process, encapsulated in law, and that rule of law can be obtained by simply turning the crank.
As a software engineer, I realize I’ve seen this situation before. It’s what code looks like before it’s been refactored. Because our systems must work efficiently for our companies to survive, software engineers have developed a comprehensive architectural best practice, starting with the concept of code smell (an indicator of the need for redesign) and including best practices that are captured in a number of books and resources on the topic, including a menagerie of cross-functional design patterns to get it right. The ability to refactor well is an important discipline to control the otherwise exponential increase in complexity.
So why haven’t we used these great best practices in the more vexing problems of organizational design, policy creation, and improvements to government regulations? I believe it’s because we haven’t come up with a way to think about these systems in a way that lends itself to good analysis, continuous improvement, evidence-based analysis, and collaboration. These and other engineering disciplines depend on a cross-situational way to think about these problems.
After having worked with a few organizations within these complex situations, I’ve found a solution. First, you need to cut through the complexity to find the critical causal paths through the system. What are the outcomes that we wish to achieve? What is the chain of events, data, and/or reasoning that lead from actions to outcomes? Usually the links are a hodgepodge, with some links being a process (this-before that), others being a calculation (this-is-part-of-that), and more. You can’t impose an unfamiliar formalism or keep within a specific way of expressing things (e.g. formal logic, as we used to do in the Expert Systems days). This needs to feel more like visual mind mapping than writing code.
Finding these paths depends critically on an expert who’s lived within the system for years, and knows where the important gears are in the machine: the linkages that matter. Indeed, this is the core skill for any smart leader in a complex organization. A good decision engineer can work with them and their team to draw a map of this chain, and then to gather the evidence and data to understand it, and then to improve it.
And my house full of screwdrivers? Ultimately, I gave it all away. It felt wonderful.
Are you facing a problem like this? I’d love to help.
