Failure recovery

Recovery must preserve the exact request, idempotency key, signing attempt history, and evidence chain. Availability is secondary to preventing duplicate or mismatched execution.

First response

1. Stop new execution for the affected tenant or adapter.
2. Capture request IDs, component health, lifecycle heads, and bounded logs.
3. Classify the failure before retrying anything.
4. Preserve the active state directory; never restore over it.
5. Reconcile external state and the evidence witness before resuming.

Failure matrix

Failure	Safe behavior	Recovery action
One signer is unavailable in Round 1 or 2	Record the failed participant; never mix commitments or shares across attempts	Start a fresh numbered attempt with a new candidate set if quorum remains
Candidate pool is exhausted	Persist a failed operation; do not restart silently	Restore quorum, then explicitly reopen with resume --retry-failed
Coordinator restarts	Reload and verify signed state; reuse only exact persisted responses	Resume the same operation ID and verify the stored signing package
Adapter times out or returns retryable uncertainty	Keep request reconcilable; do not mark executed and do not create a new idempotency key	Query provider or chain state first, then retry the same stable request only if safe
Provider durably rejects	Keep rejection evidence; do not create an execution receipt	Correct the business or policy cause and submit a new request when appropriate
Evidence witness is unavailable	Keep terminal evidence pending; do not claim independently anchored completion	Restore witness, reconcile the exact stream head, then retry the same anchor
Alert webhook is unavailable	Retain the same pending event ID across restart	Restore receiver or network; deliver the same event until a 2xx response
State host is lost	Do not overwrite a surviving active state or merge secret domains	Restore a verified backup into a new directory and isolated port, then reconcile

Backup and restore

The pilot control script can create an operational backup after the API is stopped:

deploy/pilot/pilot-control.sh stop
deploy/pilot/pilot-control.sh backup

The backup intentionally excludes signer key packages, TLS private keys, API credentials, and other secrets. Local retention does not protect against full host loss; production candidates require an independently administered backup destination.

Restore into a new path, never over active state:

deploy/pilot/pilot-control.sh restore \
  .luvion-pilot/backups/<backup>.json \
  .luvion-pilot/state/control-plane-restored

Start the restored instance on a different loopback port. Compare request counts, terminal states, lifecycle heads, evidence digests, and witness receipts. Promote it only after reconciliation succeeds.

Recovery acceptance

Recovery is complete only when:

• the active request and lifecycle records verify cryptographically;
• no conflicting idempotency record or provider transaction exists;
• signer quorum and identity pins match the approved deployment;
• pending evidence is witnessed in sequence;
• operational alerts return healthy; and
• a bounded acceptance report records who approved reopening.

If any condition is uncertain, leave execution closed and escalate through the private security channel described in Responsible disclosure.