Lesson 07 — Operation workers, idempotency, and recovery

Outcome

Execute pending Operations with a bounded worker and prove safe retry/restart behavior.

Why this comes now

Recording intent is useful only if execution and recovery have a disciplined lifecycle.

Understand

An Operation handler may run more than once. Idempotency means repeating the same intent either has no duplicate effect or can discover that the effect already happened. Recovery means startup can inspect unfinished Operations and decide whether to retry, reconcile, fail, or wait for a human.

Tests need a Drain/quiescence concept: queue length zero does not mean a worker is not executing its last item.

Build the real project

1. Implement a bounded Operation worker.
2. Claim PENDING work safely and mark RUNNING.
3. Implement one fake idempotent handler.
4. Add retry/requeue mechanics intentionally rather than a global “retry 3 times” rule.
5. Add Drain for deterministic tests.
6. Simulate restart by closing/reopening the DB with a PENDING Operation.
7. Simulate “effect happened but result persistence failed” and reconcile using the fake handler's external marker.

Completion gate

No duplicate fake effect occurs across retry. Restart processes/reconciles PENDING work. Drain waits for in-flight work, not only an empty queue.

Pitfalls to avoid

Do not hold a DB transaction open while slow I/O runs. Do not busy-loop. Do not retry permanent validation failures.

References

T3 drainable-worker note: https://github.com/pingdotgg/t3code/blob/main/docs/internals/overview.md ; Go context.

Checkpoint

M2 is complete when you trust the fake Operation worker enough to let Git use it next.