Lesson 33 — Crash recovery and reconciliation

Outcome

Deliberately kill Forge at important boundaries and make startup converge to truthful recoverable state.

Why this comes now

A daily-driver orchestrator must survive crashes without losing work or lying. We introduced Operations early specifically so this lesson is tractable.

Understand

Recovery is not “set every Running row to Failed.” It asks external reality what happened and reconciles according to each subsystem's guarantees. Some provider strategies can resume/discover sessions; some cannot. Worktrees are inspectable. Verification effects can often be rerun.

A crash is expected at every boundary between durable state and external effect.

Build the real project

1. Enumerate a recovery matrix for Operation kinds and Run states.
2. Add startup reconciliation before normal scheduling.
3. Add integration tests killing/restarting around: worktree create, agent start, open Decision, candidate capture, verification completion.
4. Preserve dirty work regardless of uncertainty.
5. Mark states needs intervention/failed honestly where automatic reconciliation is impossible.
6. Ensure shutdown uses contexts and stops admitting new work before tearing down workers.

Completion gate

Defined crash-boundary tests pass. Restart never marks a Task Done merely because previous process disappeared. Uncertain provider state is explicit. Dirty worktree survives.

Pitfalls to avoid

Do not blanket-reset RUNNING → FAILED without asking what state means. Do not retry a provider blindly if duplicate execution could edit the same worktree concurrently.

References

OPERATIONS_AND_RECOVERY.md, T3 durable intent: https://github.com/pingdotgg/t3code/blob/main/docs/internals/overview.md

Checkpoint

Write the recovery matrix into the architecture doc rather than leaving knowledge only in tests.