Designing Error Tolerant Systems

One measure of a well written report is that it will will be tossed aside by the reader upon completion with the comment, "Well, duh!  That was obvious."

Consider the common electric fence.  Specifically, I want you to consider a fence that has five strands.

Three of the strands are "ground" strands which will be green in the illustration.  They are barbed wire  which are dotted lines in my illustration.  These three strands were the original fence.  But the city folks who moved in who were not enchanted by Herefords in the Hostas or Angus drinking from their swimming pool.  The insurance company informed the farmer that he would be liable if a drunk in a Prius plowed into one of his ponies.

Fences depreciate in real, physical terms.  The posts rot or rust and lose the ability to hold tension.  The barbs become rounded with rust.  The wire becomes brittle.

The blue lines are "jumpers" where current is "jumped" over gates, around corner posts, over "ground" wires.  There is a very human tendency to assume that if one is good then three are better.

So Mr Farmer decides to beef up the fence with electricity.  Being a frugal sort of fellow, he runs two smooth, "hot" wires (the orange and red lines) between the three barbed, ground wires.  The wires near the bottom of the fence are more closely spaced than the wires that are higher.  The thinking is that if an animal sticks his/her head through the fence that their chin will hit a lower wire and their ears will touch a higher wire.  One will be "hot", one will be "not" and the brain will get an education.

Some farmers even arrange a speed dating event to ensure the local animals get introduced to the fence in the most memorable terms possible.  I tried this once with newly shorn sheep.  I had a mixed flock of commercial cross-bred sheep and a few mama cows.  Sheep are hard to keep in with an electric fence.  Something about that 6" of wool being a good insulator.  I figured that the best time to reinforce their respect of the fence was when they were strutting around buck naked.

I spread a line of corn almost beneath the fence, knowing that they would inadvertently bump into it.  I stood off to the side to watch.  The ewes started vacuuming down the corn.  A cow ambled up and punted the ewe through the fence.  As far as that cow was concerned, that corn was her corn.  It happened so fast I am not sure the ewe got poked.  And there I stood like a dummy, fence down and the ewes high-tailing it for Ingham County.

Our intrepid farmer finds himself out on many a dark and stormy night trying to troubleshoot shorts and rounding up animals.  This is not a fun way to farm.

He walks the perimeter fence, listening for the "snap" of the line arcing to the short.  He scans, looking for the faintest of arc flashes.  I once found a short that involved the carcass of a yellow jacket that spanned from the hot wire to the T post.  It made a dandy snap-and-arc.

If no short is generous enough to self-declare, he walks the line, clearing debris from the line a foot at a time. Rusty wire is indistinguishable from dead, wet weed stalks.

A better way

An earlier post on this blog involved troubleshooting based on "search by bisection".

Almost all shorts in the fence occur in the lower strand due to the ground trash that blows or gets pushed into the wire.  Some runs of fence are trashier and more susceptible to shorting out than others.  Another factor is that the lower strands are closer together and it is far more likely for the lowest hot wire to get tangled up in the barbed ground wires then for the more widely spaced upper wires to get tangled.

Troubleshooting can be facilitated by subdividing the bottom strand into segments of approximately equal risk of shorting out.  Each separate run of bottom wire is fed with a single jumper from the top, "bus" wire.

For the Eaton Rapids Joe operation it would be logical to separate the perimeter fence into about 8, isolated runs of bottom wire.  

Jumping right to the next level of sophistication involves passing the jumper through an impedance/telltale.  That is, a lamp.  In a sense, we are engineering that fortuitous yellow jacket into every place where it is logical to "search by bisection."

Putting a lamp into the circuit does two things.  One is that it provides a visible telltale when the stretch of wire downline of the lamp is shorted.  The other thing the lamp does is that it partially isolates the compromised stretch of line from the remainder of the fence.  Shorting out one stretch of bottom wire still leaves the entire top wire hot and seven of the eight runs of bottom wire hot.

Key points:

• Evaluate risks and segment accordingly
• Completely decouple (firewall) the highest risk segments from each other
• Partially decouple the highest risk segments from the "bus" segment
• Choose a decoupling method that declares or strongly signals when the high risk segment is compromised
• Expeditiously deal with the physical events that degraded the system

You won't see these strategies applied to in many social engineering scenarios.  Quite the pity, really.