Chapter 27

Hold the plate

Hold partial truth through a fault and return control before recovery execution.

On this page

Picture the plate holding two cases. One waits at prime. One, already primed, waits at charge.

The handle moves. The first station seats a primer. The powder station finds an empty hopper and faults. One effect completed. One did not. The plate has not moved.

What state is the system in?

"Failed" is true and nearly useless. It hides the successful primer, the located powder fault, the held plate, and the fact that nothing was ejected. "Roll back" is worse. The primer cannot be wished out of the case. Physical work has committed.

Many software systems have the same shape once an operation crosses a process or vendor boundary. The email was sent. The partner accepted the write. Two services deployed and the third did not. A database transaction can reverse its own writes. It cannot make the rest of the world forget.

Partial failure produces partial truth. The truth is the asset recovery needs.

First contain movement

The first rule is mechanical: if any station faults, do not advance the plate.

The test is the proposed body plan:

it "holds the plate when any station faults mid-stroke" do
  faulty.feed(work)
  faulty.cycle
  faulty.feed(work)

  result = faulty.cycle

  expect(result).not_to be_advanced
  expect(result.outcomes.map(&:status)).to eq(%i[completed faulted empty empty])
  expect(result.outcomes[1].fault).to eq(:powder_hopper_empty)
end

Advancing would turn a located problem into a moving ambiguity. The uncharged case would continue toward a seated bullet and a finished appearance. Holding the plate preserves position as evidence: this case, under this station, after this attempted stroke.

Containment is not recovery. It stops the loss of information and prevents the system from creating more work on top of an unresolved state.

Write the failure table while the state is still visible

The modeled stroke can be recorded like this:

Position Completed effect Missing fact Containment Disposition Compensation or next action Re-entry gate Safe to assess twice?
Prime primed established none before current position plate held resume current fact may be re-established idempotently normal station protocol yes
Charge no charge added none before current position; charged still absent plate held resume after hopper correction reattempt charge same position, same facts yes
Seat empty none plate held none none not applicable yes
Crimp empty none plate held none none not applicable yes

Now change the charge row. Suppose the work at charge does not carry primed, a fact it should have received before reaching that position. The visible powder fault no longer explains the state. That case is quarantined. Resuming it would let an earlier omission travel deeper into production.

The table prevents four concepts from collapsing:

  • Containment stops further unsafe movement.
  • Assessment derives what each position may do next.
  • Compensation performs an additional action that makes an irreversible partial outcome acceptable.
  • Recovery execution carries out the approved actions and proves the system is operable again.

The current Ruby slice implements the first two. It can recommend quarantine, which is a compensating disposition, but it does not move material or restart the press.

Derive recovery from facts

Each occupied position carries its established facts. Recovery compares those facts with the facts that should exist before the current station:

def missing_before(snapshot)
  index = @stations.index(snapshot.station)
  @stations[0...index].map(&:fact).reject { |fact| snapshot.work.fact?(fact) }
end

def disposition_for(missing)
  missing.empty? ? :resume : :quarantine
end

The whole object is lib/press_recovery.rb.

A case missing only its current station's fact can resume. A case missing a fact from an earlier station leaves through quarantine with the missing facts attached.

The unit tests state those decisions before the implementation:

it "resumes a piece that is missing only its current station's work" do
  at_charge = snapshot(station: stations[1], facts: { primed: true })

  assessment = recovery.assess([at_charge]).first

  expect(assessment.disposition).to eq(:resume)
  expect(assessment.missing).to be_empty
end
it "quarantines a piece missing a fact it should already carry" do
  at_seat = snapshot(station: stations[2], facts: { primed: true })

  assessment = recovery.assess([at_seat]).first

  expect(assessment.disposition).to eq(:quarantine)
  expect(assessment.missing).to eq([:charged])
end

Assessment reads and returns. It moves no plate, re-strikes no station, and opens no side door around the readiness boundary. That refusal makes it safe to ask twice:

it "assesses from facts alone, so asking twice gives the same answer" do
  positions = [snapshot(station: stations[2], facts: { primed: true })]

  expect(recovery.assess(positions)).to eq(recovery.assess(positions))
end

When systems fail, operators retry questions, refresh pages, and repeat commands. A read-only assessment that changes its own answer becomes a second incident inside the first.

The application must return control

The press can hold itself, but the production orchestrator once kept calling cycle. A fake station that faulted once exposed the problem: the next call succeeded, so the application silently recovered by retrying.

The outer test demanded one attempt:

it "stops at the first station fault instead of silently retrying" do
  result

  expect(charge.attempts).to eq(1)
end

The implementation now returns a HaltedRun containing faults, current positions, the derived recovery plan, and every exit decided before the fault. It does not ask final inspection to release incomplete work.

That is an ownership decision. ProductionRun owns whether the automated run continues. ProgressivePress owns whether the plate advances. PressRecovery owns assessment. None of them owns operator authorization to execute a recovery. The system returns control at the point where evidence and authority meet.

This result protocol lets a caller distinguish completion without rescuing an exception or interrogating fields:

result.halted? # true for HaltedRun, false for RunResult

The two results do not pretend to contain the same facts. A completed run has released material and a batch record. A halted run has positions and a recovery plan. Sharing one query does not erase their different states.

Compensation accepts the past

When an irreversible effect cannot be undone, a compensating action buys back an acceptable future. Refund a charge rather than pretending it was never made. Publish a correction rather than unsending a message.

Quarantine is the bench's visible compensation. The damaged or inconsistent case leaves normal flow, identity and reason intact. It cannot re-enter because some later loop happened to call the gate again. Re-entry requires the named authority and evidence from the failure table.

Two real cases now occupy that rework path. They failed headspace inspection in the first bounded sizing lot and were replaced before the accepted set was swaged. They remain segregated for a photographed rework and second inspection. That is observed rejection and preserved identity. It is not a mid-cycle press recovery, and it will not become executed rework until the cases cross the inspection boundary again.

Compensation is not automatically restoration. Moving one case to quarantine does not prove the hopper is filled, the plate state is understood, the operator has approved resumption, or the next cycle is safe. "Recovery plan returned" is not "system recovered."

The current model stops at that line on purpose.

The provenance question remains

The halted result preserves physical and decision state but not the provenance passed into the run. It would be false to issue a completed BatchRecord for work that never completed. It may also be operationally weak to return an interrupted attempt without load and lot identity.

That pressure needs an ownership decision: is there a separate run-attempt record created before production, does HaltedRun carry immutable input provenance, or does an outer application retain the command as the audit record? Those choices change the message protocol. This tutorial names the gap and refuses to choose it inside a DNA refactor.

Rehearse one failure

Choose a multi-effect event in your system and complete the table:

Effect What completed What is missing Containment Disposition Compensation Re-entry proof Repeatability

Then answer:

  1. What movement must stop first?
  2. Which completed effects are irreversible?
  3. Can the system report partial truth without translating it into total failure?
  4. Is assessment safe to repeat?
  5. Who authorizes executed recovery?
  6. What proves the system is ready to operate again?
  7. What identity must survive the interruption?

If your recovery story ends at "retry," name what makes the retry safe. If it ends at "rollback," name every external effect the rollback cannot reverse.

What the green model cannot claim

The Ruby model proves halt, containment, per-position truth, and deterministic assessment. It does not prove a physical powder fault has been recovered on this bench. It does not execute a resume or quarantine plan. It does not yet resolve interrupted-run provenance.

Those limits are not missing footnotes. They are the current edge of the body plan. The next part of the tutorial steps back from individual mechanisms and asks what portion of this system can be shipped as one complete, operable thought.