Chapter 27
Hold the plate
Hold partial truth through a fault and return control before recovery execution.
On this page
Picture the plate holding two cases. One waits at prime. One, already primed, waits at charge.
The handle moves. The first station seats a primer. The powder station finds an empty hopper and faults. One effect completed. One did not. The plate has not moved.
What state is the system in?
"Failed" is true and nearly useless. It hides the successful primer, the located powder fault, the held plate, and the fact that nothing was ejected. "Roll back" is worse. The primer cannot be wished out of the case. Physical work has committed.
Many software systems have the same shape once an operation crosses a process or vendor boundary. The email was sent. The partner accepted the write. Two services deployed and the third did not. A database transaction can reverse its own writes. It cannot make the rest of the world forget.
Partial failure produces partial truth. The truth is the asset recovery needs.
First contain movement
The first rule is mechanical: if any station faults, do not advance the plate.
The test is the proposed body plan:
it "holds the plate when any station faults mid-stroke" do
faulty.feed(work)
faulty.cycle
faulty.feed(work)
result = faulty.cycle
expect(result).not_to be_advanced
expect(result.outcomes.map(&:status)).to eq(%i[completed faulted empty empty])
expect(result.outcomes[1].fault).to eq(:powder_hopper_empty)
end
Advancing would turn a located problem into a moving ambiguity. The uncharged case would continue toward a seated bullet and a finished appearance. Holding the plate preserves position as evidence: this case, under this station, after this attempted stroke.
Containment is not recovery. It stops the loss of information and prevents the system from creating more work on top of an unresolved state.
Write the failure table while the state is still visible
The modeled stroke can be recorded like this:
| Position | Completed effect | Missing fact | Containment | Disposition | Compensation or next action | Re-entry gate | Safe to assess twice? |
|---|---|---|---|---|---|---|---|
| Prime | primed established |
none before current position | plate held | resume | current fact may be re-established idempotently | normal station protocol | yes |
| Charge | no charge added | none before current position; charged still absent |
plate held | resume after hopper correction | reattempt charge | same position, same facts | yes |
| Seat | empty | none | plate held | none | none | not applicable | yes |
| Crimp | empty | none | plate held | none | none | not applicable | yes |
Now change the charge row. Suppose the work at charge does not carry
primed, a fact it should have received before reaching that position. The
visible powder fault no longer explains the state. That case is quarantined.
Resuming it would let an earlier omission travel deeper into production.
The table prevents four concepts from collapsing:
- Containment stops further unsafe movement.
- Assessment derives what each position may do next.
- Compensation performs an additional action that makes an irreversible partial outcome acceptable.
- Recovery execution carries out the approved actions and proves the system is operable again.
The current Ruby slice implements the first two. It can recommend quarantine, which is a compensating disposition, but it does not move material or restart the press.
Derive recovery from facts
Each occupied position carries its established facts. Recovery compares those facts with the facts that should exist before the current station:
def missing_before(snapshot)
index = @stations.index(snapshot.station)
@stations[0...index].map(&:fact).reject { |fact| snapshot.work.fact?(fact) }
end
def disposition_for(missing)
missing.empty? ? :resume : :quarantine
end
The whole object is lib/press_recovery.rb.
A case missing only its current station's fact can resume. A case missing a fact from an earlier station leaves through quarantine with the missing facts attached.
The unit tests state those decisions before the implementation:
it "resumes a piece that is missing only its current station's work" do
at_charge = snapshot(station: stations[1], facts: { primed: true })
assessment = recovery.assess([at_charge]).first
expect(assessment.disposition).to eq(:resume)
expect(assessment.missing).to be_empty
end
it "quarantines a piece missing a fact it should already carry" do
at_seat = snapshot(station: stations[2], facts: { primed: true })
assessment = recovery.assess([at_seat]).first
expect(assessment.disposition).to eq(:quarantine)
expect(assessment.missing).to eq([:charged])
end
Assessment reads and returns. It moves no plate, re-strikes no station, and opens no side door around the readiness boundary. That refusal makes it safe to ask twice:
it "assesses from facts alone, so asking twice gives the same answer" do
positions = [snapshot(station: stations[2], facts: { primed: true })]
expect(recovery.assess(positions)).to eq(recovery.assess(positions))
end
When systems fail, operators retry questions, refresh pages, and repeat commands. A read-only assessment that changes its own answer becomes a second incident inside the first.
The application must return control
The press can hold itself, but the production orchestrator once kept calling
cycle. A fake station that faulted once exposed the problem: the next call
succeeded, so the application silently recovered by retrying.
The outer test demanded one attempt:
it "stops at the first station fault instead of silently retrying" do
result
expect(charge.attempts).to eq(1)
end
The implementation now returns a HaltedRun containing faults, current
positions, the derived recovery plan, and every exit decided before the
fault. It does not ask final inspection to release incomplete work.
That is an ownership decision. ProductionRun owns whether the automated run
continues. ProgressivePress owns whether the plate advances. PressRecovery
owns assessment. None of them owns operator authorization to execute a
recovery. The system returns control at the point where evidence and authority
meet.
This result protocol lets a caller distinguish completion without rescuing an exception or interrogating fields:
result.halted? # true for HaltedRun, false for RunResult
The two results do not pretend to contain the same facts. A completed run has released material and a batch record. A halted run has positions and a recovery plan. Sharing one query does not erase their different states.
Compensation accepts the past
When an irreversible effect cannot be undone, a compensating action buys back an acceptable future. Refund a charge rather than pretending it was never made. Publish a correction rather than unsending a message.
Quarantine is the bench's visible compensation. The damaged or inconsistent case leaves normal flow, identity and reason intact. It cannot re-enter because some later loop happened to call the gate again. Re-entry requires the named authority and evidence from the failure table.
Two real cases now occupy that rework path. They failed headspace inspection in the first bounded sizing lot and were replaced before the accepted set was swaged. They remain segregated for a photographed rework and second inspection. That is observed rejection and preserved identity. It is not a mid-cycle press recovery, and it will not become executed rework until the cases cross the inspection boundary again.
Compensation is not automatically restoration. Moving one case to quarantine does not prove the hopper is filled, the plate state is understood, the operator has approved resumption, or the next cycle is safe. "Recovery plan returned" is not "system recovered."
The current model stops at that line on purpose.
The provenance question remains
The halted result preserves physical and decision state but not the provenance
passed into the run. It would be false to issue a completed BatchRecord for
work that never completed. It may also be operationally weak to return an
interrupted attempt without load and lot identity.
That pressure needs an ownership decision: is there a separate run-attempt
record created before production, does HaltedRun carry immutable input
provenance, or does an outer application retain the command as the audit
record? Those choices change the message protocol. This tutorial names the
gap and refuses to choose it inside a DNA refactor.
Rehearse one failure
Choose a multi-effect event in your system and complete the table:
| Effect | What completed | What is missing | Containment | Disposition | Compensation | Re-entry proof | Repeatability |
|---|---|---|---|---|---|---|---|
Then answer:
- What movement must stop first?
- Which completed effects are irreversible?
- Can the system report partial truth without translating it into total failure?
- Is assessment safe to repeat?
- Who authorizes executed recovery?
- What proves the system is ready to operate again?
- What identity must survive the interruption?
If your recovery story ends at "retry," name what makes the retry safe. If it ends at "rollback," name every external effect the rollback cannot reverse.
What the green model cannot claim
The Ruby model proves halt, containment, per-position truth, and deterministic assessment. It does not prove a physical powder fault has been recovered on this bench. It does not execute a resume or quarantine plan. It does not yet resolve interrupted-run provenance.
Those limits are not missing footnotes. They are the current edge of the body plan. The next part of the tutorial steps back from individual mechanisms and asks what portion of this system can be shipped as one complete, operable thought.