How to simplify your overly complex networking design
If you flood your networking design with extraneous features and don't have a firm understanding of the overall system, you might fall into a complexity wormhole.
Information technology systems of all kinds -- networks, applications, databases, storage systems and any other system you can think of -- are becoming increasingly complex. While the complexity grows in relation to the problems being solved, humans are also becoming part of the problem.
For various reasons, humans tend to prefer complex resolutions to simple ones. The combined effect of many humans preferring more complex resolutions over time means large-scale systems tend to accrue complexity, until they fall into a complexity wormhole, becoming wholly incomprehensible.
How can you tell if you are falling into a complexity wormhole? This question has no definitive answer, but you can heed some specific signs.
Don't overload features in networking design
The first sign is this: You look at a networking design you drew just a little bit ago, or at a systemic view with all the components. While you understand some small pieces of it, you don't understand the entire design. This is a sure sign you cannot keep the entire design in your head all at once. If that's the case, the people who implement, troubleshoot and manage the system won't be able to visualize the networking design, either.
The second sign is feature multiplication. If you're adding a feature, protocol, nerd knob or new overlay to solve a problem created by something you added -- whether in design or deployment -- then you're on the path to a complexity wormhole or well within its swirling, gravitational event horizon. If you're stacking features like wood on a beaver's dam in hopes of stopping "just another leak," then the networking design is in trouble. Down this path lies ossification and eventual systemic failure.
The third sign is you end up using more than one or two esoteric features that no one else seems to know about. This is worse if you add features that even you don't understand operationally. If you are constantly going to the manuals to figure out how something works, and then referring to your notes to understand why it was configured that way in the network, you're probably approaching a complexity wormhole.
Explore the tradeoffs in networking design
Can you move complexity from one place to another to simplify the overall networking design?
What should you do if you find these signs of being over the event horizon, or close to it, in a system you are designing? First, understand the concepts of complexity in large-scale systems and expose the ironclad, three-way tradeoffs you are battling. In most systems, you face speed, quality and cost tradeoffs, as well as state, optimization and surface tradeoffs. Ask questions about each one -- for instance:
What state?
How fast is it changing, and where is it coming from?
What interaction surfaces?
How broad and deep are they?
Once you have a broader understanding of the shape of the systemic complexity, ask yourself the following: What am I optimizing for, and why? This should drive you back into the business problem -- not the requirements, but the problem. Then, rethink what is being optimized and why.
If you push back on the requirements a little, can you reduce the complexity a lot? What are the tradeoffs in this area? Rethink your modularization and abstraction. Where are you hiding information, and why? Can you change the amount of information you are hiding, what you are hiding, or where you are hiding it, to simplify? What optimizations will you give up if you change your information-hiding strategy?
Can you move complexity from one place to another to simplify the overall networking design? A common case here is to start with a more complex base layer, making higher layers simpler.
Complexity wormholes are all too common in information technology systems, especially in computer networks. Learning to diagnose the complexity level of a system, and how to make the system simpler, is a skill every practitioner needs to learn.
