Thursday, June 24, 2021

A project that went into the ditch

 

Product travels down the line from left-to-right.
Imagine you are in a factory and you have a new product scheduled to come into your plant.

The new product requires a different kind of robot mounted weld gun on the equipment in the robotic cells. The wizards at Headquarters find a kind of weld gun that can manufacture both products but the guns have a different geometry than the ones that are currently installed. The relationship between the plate that is bolted to the robot and the position of the weld tips are different.

That requires that the stations be reprogrammed for all models.

Furthermore, let's say the market for your product is white-hot and the bean-counters demand that the transition take place with no loss of scheduled product. The muckity-mucks in the factory don't see any problem with that and they schedule the replacement and reprogramming of the first three stations for Thanksgiving weekend.

Their thinking was that the area had, arguably, the single best robot programmer in the plant. They had nine electrician/programmers and 12 shifts to mechanically change out the guns and reprogram them. If they got in trouble, there were many other tradesmen they could strip out of other areas to assist.

It was a train-wreck

Production on Monday morning was delayed because they were not able to get even the first three models (current production) programmed.

The Plant Manager had to call Corporate and tell them that production had not started on Monday, that he did not know when it would start, and he could not tell them why there was a failure.

The Plant Manager was not in a good mood on Monday morning.

Nobody who worked for the Plant Manager was in a good mood.

What happened?

It was the result of multiple, sub-optimal decisions. If it had been just one decision it would have been recognized and fixed on-the-fly and I would not be writing this post.

In the order that the sub-optimal decisions impacted the process

Image repeated for convenience. There are a total of 18 work-elements. Six models to be programmed in 3 stations. Broken down farther, there are six robots in each station so there are 108 "sub-elements"

Not enough product was staged. Being skimpy on future product is understandable as each copy was worth roughly a half-million dollars but current product has a scrap-value (at this stage of the process) of roughly $600.

The mechanical installation of the guns and the electrical hook-up proceeded with no issues. They were validated (weld coupons shot and torn apart) by the end of third shift.

The state-of-the-project at the End of Dayshift, Thursday


The problem arose when they transferred Job A into Station 1 and began programming. They had one programmer working and two programmers with their fingers up their rear-ends.

Safety had deemed it "too dangerous" to program more than one robot at a time in a robotic cell.

It didn't seem like it would be a problem as there were ten shifts left when second shift took over.

State-of-the-Project when Dayshift walked out Friday. A is programed in stations 1 and 2. B is not programmed.


The next major issue arose when first shift walked back into the shop on Friday morning.

The lead programmer on day-shift reviewed second and third shifts work and decided it was garbage. He deleted it.

Note: This was not his job. He was an "Hourly" employee just like the programmers on second and third shift.

The weenie from HQ applauded the lead programmers decision. He wanted this to be done RIGHT. The weenie proudly announced to Plant and HQ management that he thought the programmers on second and third shift had been drunk.

That did not endear the HQ weenie to second or third shift. The plant management hammered the supervisors of the programmers for not disciplining the programmers for being "impaired".

By the end of first shift of the second day, management knew they were in deep shit. They had Job A programmed in Stations 1 and 2. They still had seven more combinations of the (current product X stations) to program before they could turn a wheel on Monday. 

The additional nine work-elements related to the future product were aborted.

Top management were First Shift animals and by their reckoning they had completed 2-of-18 tasks and burned up half of the clock. Furthermore, they needed to be able to run current production in order to pay the bills and they were 2-of-9 with half the clock to go.

As noted earlier, production was delayed on Monday morning.

Post-mortem

The root cause of the problem is that the Plant had 9 programmers assigned and many more available but due to a confluence of factors was only using one, the prima-donna on First shift.

At the beginning of the programming, the programmers were limited by not having material to program with in Stations 2 and 3. It should be noted the problem also occurred at the end of the curtailed project as the line drained. There were programmers sitting on their hands because there was no product in Station 1 and then none in Station 1 and 2.

The "fix" was to have the staged material look like A-B-C-a-b-c-A-B and shuttle A-B-C into Stations 1, 2 and 3 so you have three stations with material in them. Then, at the back end, you repeat A and B so they can be programmed in the stations they dead-headed over.

The prima-donna problem

Solving the prima-donna problem was more convoluted than solving the shortage of material issue.

The solution is glaringly obvious. If there are 18 work-elements (six Models X three Stations) and three shifts then give each shift six work-elements and forbid the other shifts from touching work-elements that are not assigned to them.

The resistance to that solution came from plant mid-level management. They had worked very, very hard to drive the culture to where any tradesman could be assigned work anyplace in the plant. They had a home that was their default (and where they generally wanted to stay) but management exercised the right to move them around.

The other factor was that it is generally accepted as fact that decoupling systems improves throughput. If Task B cannot happen unless A is True, then B is coupled to A and nothing can happen on B if A is Not True. Assigning specific jobs to just one shift is an additional constraint. What happens if somebody does not show up on a given shift?

In theory, it should be faster to have every shift do programming on every job and the programming pendant handed off to the next shift so the programming continues seamlessly.

In practice, "...continues seamlessly" bit them in the azz.

It actual practice having all shifts responsible for all of the work-elements introduced coupling when the prima-donna took it upon himself to review (and delete) other shifts' work.

Not surprisingly, the Union came down VERY solidly on the side of assigning specific jobs to specific shifts. They when to bat and pushed it HARD with management.

Their motivation was two-fold. Their primary reason was the that the prima-donna was functioning in the role of Management by passing judgement on another Journeyman's work. That was intolerable to the Union because it blurred the line between Hourly and Management and could not be allowed to continue.

The other heartburn originated with the "...must have been drunk..." comments and the turmoil that caused. Quite likely, those words originally came from the hourly prima-donna on first shift. The contract is very, very clear about how suspected impairment is to be handled and it does not include announcing that entire shifts of hourly people were drunk to Mount Olympus before evidence is gathered, interviews are held and so on.

The third counter-measure

The third counter-measure was almost a moot point because it was possible to get more material staged for future work.

Safety was petitioned to come out and observe how it was possible to safely program both left-and-right sides of the product at the same time. The robots were not physically long enough to reach across the line and whack a programmer on the other side of it.

The reason this was a player is that time constraints eventually forced the plant to install weld guns and program them on two-day weekends. Having the ability to have two programmers work on each station at a time halved the programming time. 

When doing a single station, each shift was given two of the six robots in the cell. If there were only four robots in the cell then third shift was usually given the task of installing the guns and first and second doing the programming. That was a shame because third shift had some gifted programmers.

6 comments:

  1. I suppose it didn't occur to anyone to do only one robot with the new weld head and new programming as a proof of concept? Or perhaps to revert, reinstall the old weld heads and the old programming?

    I've watched too many of these "big bang" implementations go sideways out of the gate to ever allow myself to get suckered into one (again). Of course, by retiring, I've considerably reduced the risk of ever having to be involved in such decisions in the future.

    ReplyDelete
    Replies
    1. There was very little commonality from station-to-station.

      Some welding was very exposed and easy to get at. The flanges in the door opening for instance.

      Other welds were very limited access, up in the rear wheel well and inside of door posts, for instance. Because of economics, even the weird, limited access guns were required to make ten welds per job.

      There really wasn't anything wrong with the guns they chose. Rather, it was human nature, the plant's "plan" abetted by the HQ weenie that caused the train-wreck.

      Delete
  2. WOw, ONE primidona and management 'comfort zones' broke the whole system. Not that I've ever seen that before... mumble, mumble, PhDs...

    ReplyDelete
    Replies
    1. +1

      Or we'll fix the code on the fly (Software Engineers).

      Gerry

      Delete
  3. Worked with (well, put up with) a prima-donna at my last job, same as above. Non-union shop. His idea of on-boarding training was to give a large project. When new troop failed, play the hero, swoop in and fix it. Actually playing hero was standard ops.
    Management at the time suffered it since he was doing new systems conversions/programming management wanted done. Management changed, new management reined him in, hissy fits ensued and he left. (good riddance)
    My prima-donna and this situation was all on management, or lack thereof.
    But cock-up described above? For Dog's sake, it's a simple move/add/change.
    I don't have the Project Planner(tm) magic letters after my name, but I've done equiv. projects.
    Campus phone system changes, hot-cut. Over a weekend, but zero downtime for users. Nationwide office network/voice change, slo-mo over months, zero downtime users on sites.
    Anyway, management cited here had badly HUA syndrome.
    Get three or four people who know existing and new system.
    *Not management people*
    Working macro to micro lay out plan *on paper* to convert.
    Identify what can be done off-line, even to doing code with dummy settings to be filled in later.
    12 shifts/96 hours! Forever! Should be *on paper* what happens, when, and people who've actually done this work set the time standards, not some management type who walked by one of the machines a random afternoon.
    Project Planner has on-site command. Communicates up to management what's going on, down to workers what to do. Workers do *nothing else* unless PP sign-off.
    All on management for this one. Knew what they wanted to do, but didn't know WTF how to get there.
    The prima-donna should have been noisily fired and several management folks had their desks ceremonially burned in the courtyard. But I doubt it.

    ReplyDelete
    Replies
    1. It had been more than ten years since this plant had a major change-over. Significant amounts of turn-over had happened.

      I was stunned at the absence of planning. We did more planning when my brother and I drove deer during deer seasons. THere were only two of us and we were doing it for fun. And our who, what, why, where, when and how dwarfed the plant's effort that Thanksgiving weekend.

      Delete

Readers who are willing to comment make this a better blog. Civil dialog is a valuable thing.