Sunday, September 6, 2020

A few thoughts on "Resilient" systems

Picture in your head a very simple system. It could be the flow of energy through an ecosystem. It could be widgets traveling down an assembly line. It could be a row of desks with bureaucrats processing applications for visas.

The packages-of-interest pass from the green process to the yellow to the red to the pink and then finally to the brown. The packages move in discrete steps cunningly labeled step "1", step "2" and so on.

At this point, consider that a package cannot skip over a process or a desk.

This process can be easily disrupted by adding chaos to any one of steps "1"-through-"4". This is not a very resilient process.

Now, picture in your head five parallel processes. I suppose I should have labeled them A-through-E. This cluster of processes is more resilient than the first example but still has some vulnerabilities. One disadvantage of this system is that it does not reap any of the economies-of-scale that are available to fewer but larger paths.

Yeah, I know this slide is busy. But consider several, parallel processes where the output from one node in one of the processes has the option of "percolating" to whichever adjacent process has the shortest queue. Think of it as being like the check-out line in a grocery store.

Just to make life interesting, I have one connection, Yellow-to-Red, where path-shifting is not allowed.

One work-around for the inability to shift to a parallel path is to increase buffer-sizes.

That way, if Step 2 has a hiccup, then the Red processes can continue to function by using the material stored on-site and the Yellow process can continue to work by shipping "semi-finished goods" to its internal buffer.

A real-world example to make it more concrete

Let us assume that in some distant, fantasy universe it was decided to maximize robustness by having a mix of power generation technologies.

One technology was purely renewables and was characterized by extreme intermittancy.

The second technology was "Low-carbon, fossil fuel" and is characterized by the complete inability to "buffer" or store fuel on-site.

The third technology in this completely fictitious universe is "High-carbon, fossil fuel" and is characterized by the creation of large amounts of CO2 per kW-hr but is also supremely robust with respect to disruptions in fuel delivery. Coal can be stacked on the ground. Oil can be stored in tanks.

This system, as described, is not particularly robust especially if all of the high-carbon plants are decommissioned. It doesn't take much imagination to picture events that can cripple the delivery of natural gas to the low-carbon plants.

One way to make the system much more resilient is to make the fossil fuel "flex" plants multi-fuel. Instead of natural gas powered gas-turbines, to take the thermodynamic hit and use steam turbines and design the boilers to have multi-fuel capability.

And that is the challenge. Resilience often comes with a financial penalty.


A universe that designs systems to be resilient is likely to not activate that resilience and the decision-makers look like idiots. It is an example of confirmation bias. Why would a terrorist target a natural gas line if the plant can seamlessly switch to heavy oil or coal? The critical information is buried in what did not happen.

A universe that designs systems for maximum economic efficiencies is likely to get hammered by events (and political movements) that use the resulting fragilities to their own advantage.


  1. Sorry to be the jerk here. I feel like you were planning to put the story of ww2 and the bombers returning with bullet holes in there somewhere and it slipped(damn dogs of war)

    1. Nope. Everybody heard that story. That is why I tucked that graphic into the essay.

      Impossible for you to be a jerk. Keep commenting. But, thanks for not implying I was describing line-dancing.

    2. Turns out that was the first time I had heard that story. It impressed the crap out of me and so I thought it deserved a link to to when you had posted it. I was trying to "out subtle" you so yeah bit of a jerk. A bit of topic, enjoyed the 7 cows saga and feel like you are one of a relative few that saw just how close of a thing that was when the nurse hopped that plane. I know people that work with other passengers that were on the plane. 2nd tangent, have you noticed how the ebola outbreaks from the story rhyme with the current fun and games?

  2. Efficiency is the enemy of redundancy; the balance between the 2 will depend on the application.

  3. A case in point was the hard drive manufacturers that slowly moved (nearly) the entire supply chain to Thailand to the extent that the flooding in late 2011 destroyed 1/3rd of their capacity and affected drive shipments for most of 2012. It also quickened the move to SSDs even though the prices were higher.