Some of the hardest production incidents are not full outages. They are the incidents where an operation finished in one place and not another.

A payment provider accepts a charge, but the HTTP client times out before receiving the response. A database write commits, but the event that should have followed never leaves the process. A worker performs an external action, crashes before acknowledging the message, and the queue delivers it again.

The application stays online. Dashboards remain green. Most customers never notice. But one payment, order, account, shipment, or dispute no longer has a trustworthy story. In practice, these incidents are often discovered through a support ticket or reconciliation mismatch before they appear as a clean application error.

At that point, asking whether the request succeeded is not enough. You need to know what actually completed, which state change committed, whether the external action happened more than once, and whether retrying is safe.

Those are not just debugging questions. They are questions about business state: what the company currently believes is true about a customer, a balance, an order, or a decision. Most systems have plenty of evidence - logs, traces, database rows, queue histories, provider dashboards, audit records, support notes. What they often lack is an explicit contract for turning that evidence into a durable domain record the application accepted.

Call that gap the authority gap: a system has evidence that work was requested or attempted, but lacks a durable domain record identifying the transition it accepted, the evidence and policy that justified it, and which effects remain confirmed, failed, or unknown.

Authority is scoped, not universal. The payment provider owns whether it created an external refund. The application owns which domain transition it accepted. Projections are derived views built from accepted facts. Webhooks and callbacks are observations until validated and accepted into the application's model. No single record establishes truth across every participating system.

I did not arrive at that idea through architecture diagrams. I arrived at it while investigating a production system where the infrastructure showed the work as consumed, but callbacks and downstream operations were still missing.

I ran into this with a Kafka consumer

The consumer handled business-critical banking events: callbacks, ACH uploads, and other work that could not simply disappear.

The service looked healthy. Messages were moving through the consumer group, and nothing suggested a broad outage. But if the container restarted after receiving a message and before completing the business operation, that work could be lost.

The failure came from a bad combination of behaviors. The consumer group's committed offset could move past a record before the business operation associated with that record had durably completed. Exceptions were being masked instead of driving the message through an intentional failure path. There was no dead-letter queue or durable record showing that an event had been received but never successfully applied.

The committed offset told Kafka to resume after the record, even though the associated business operation had not completed. A committed offset is a restart checkpoint, not proof that the business operation succeeded. From the application's perspective, the callback had never happened, the ACH upload had never completed, or the downstream operation was still missing.

The question was not whether Kafka delivered the message. It had. The question was whether the business operation represented by that message completed, and there was no single place that could answer it.

The fix was to commit offsets only after successful processing, make handlers idempotent, stop masking exceptions, retry or re-seek failed records intentionally, route exhausted failures to a dead-letter queue, and test restart mid-processing.

Committing after processing traded silent loss for possible redelivery. Idempotency was still required because the external or business effect could complete before the offset commit. Committing before the business effect can lose work; committing after it can duplicate work after a crash.

Kafka transactions do not automatically make arbitrary external side effects exactly once. The dead-letter queue preserves failed work for investigation, but it does not establish whether an external effect already occurred.

Those changes addressed the immediate message-loss problem. The larger lesson was that delivery infrastructure can prove that a message moved through the system. It cannot, by itself, prove that the business action behind that message was accepted as a domain fact.

There are three outcomes, not two

Application code often treats a remote operation as having two outcomes:

SUCCESS
FAILURE

Distributed systems usually have a third:

UNKNOWN_OUTCOME

A timeout tells you that the caller stopped waiting. It does not tell you that the remote operation failed. The service may have completed the request before the connection dropped. A database may have committed before the process crashed. A message consumer may have finished its work and then failed before acknowledging the message.

AWS describes this problem directly in its guidance on making retries safe with idempotent APIs. After a timeout, the caller may be unable to tell whether the remote operation completed. Retrying may recover the work, or it may repeat an effect that already happened.

This is why retries and idempotency appear together so often. Retrying a genuinely side-effect-free read is usually harmless. Retrying a mutation might charge a customer twice, issue two refunds, create duplicate shipments, or send the same notification repeatedly.

Messaging systems have the same uncertainty. Google Cloud Pub/Sub, for example, uses at-least-once delivery by default, which means subscribers must be prepared to receive the same message more than once. Apache Kafka documents the same delivery-semantics tradeoffs: offset commits record consumption progress, not business completion.

A clean failure is usually manageable: the system can reject the operation, retry it, or escalate it. An unknown outcome is harder because the correct next action depends on something the system cannot yet prove. Retry too quickly and you may repeat an irreversible effect. Do nothing and the intended operation may never finish. Guess incorrectly and different parts of the system may permanently disagree.

The refund that may have happened twice

The same gap shows up when the missing truth is outside your process. Consider a refund workflow.

A customer requests a refund. The application creates a pending record keyed by refund_intent_id, the logical business request, and runs eligibility checks. Each worker execution receives its own attempt_id, while every submission for the same refund reuses an idempotency_key tied to the original business intent.

The provider completes the refund and assigns a provider_refund_id. The response never reaches the worker. From the worker's perspective, the request timed out, so it schedules a retry. Moments later a webhook arrives with its own webhook_event_id - one external notification delivery - confirming the original refund succeeded, and the retry may already be running.

None of this requires an unusual edge case. Timeouts happen. Messages are redelivered. External systems can finish work even when the caller never receives confirmation. Idempotency, durable workflows, and reconciliation exist for the same class of failure.

But once this refund reaches an unknown outcome, the team still has to answer some uncomfortable questions: Did both submissions represent the same logical refund? Did the retry start before the webhook arrived? Did the provider receive one request or two? Were downstream accounting events emitted once or twice? What action is safe now?

The application may contain evidence for every part of the answer. The problem is that the evidence lives in different systems, each with its own responsibility and its own scoped authority.

Evidence is not authority

A single refund might leave behind records like these:

Refund requested
Review completed
Provider submission started
Provider request timed out
Retry scheduled
Provider webhook received
Refund status changed to completed

Every line can be accurate. Together, they still may not answer the question that matters most:

What did the application authoritatively accept as true, and why?

Logs, traces, queue histories, workflow histories, and provider dashboards each describe part of the execution. A database row describes the latest stored representation:

{
  "id": "refund_123",
  "status": "completed",
  "amount_cents": 12500,
  "currency": "USD",
  "updated_at": "2026-07-27T13:42:11Z"
}

That is enough to return an API response or render a page. It does not explain who requested the refund, why it was allowed, how many provider attempts occurred, which attempt completed, or which downstream projections have converged.

An updated_at timestamp is not a history. Even a minimal audit entry like status: pending -> completed only tells you that a value changed. It does not tell you why the transition was valid, what evidence supported it, or which external action preceded it.

Event sourcing is one established response to this problem. Instead of storing only the latest representation, the system records an ordered history of state-changing events and reconstructs current state from that history. That provides a stronger foundation, but recording more events is not enough by itself. The system still has to define what those events mean: a request is not an attempt, an attempt is not confirmation, a timeout is not proof of failure, and a log entry is not automatically a business fact.

Without those distinctions, the event stream can become another collection of records that engineers must interpret after the incident.

A compact vocabulary

Authority gaps often appear when these concepts are stored without being distinguished:

An intent can spawn multiple attempts. A webhook is an observation until the application validates it and records the fact it supports. A projection such as status: completed may lag behind or diverge from the accepted fact if downstream materialization fails.

When these layers collapse into one undifferentiated row, operators inherit the reconstruction work that the model should have done upfront.

The state machine already exists

Business applications almost always contain one or more state machines, whether anyone explicitly designed them or not.

A refund might move through:

REQUESTED
-> UNDER_REVIEW
-> APPROVED
-> SUBMISSION_PENDING
-> COMPLETED

An order might move from created to paid to shipped to delivered.

Those transitions are real business rules. But the rules are often spread across controllers, workers, webhook handlers, database constraints, provider integrations, feature flags, and support tools. No single component owns the complete model. It emerges from the combined behavior of the system.

A payment's authoritative transition may depend on provider confirmation and a committed ledger entry. Marking the order paid and sending a receipt are downstream consequences that may converge later. The transition from PAYMENT_PENDING to PAID is not necessarily one atomic operation. It is a collection of operations the business expects to remain consistent - authoritative payment state, required invariants, derived projections, and asynchronous side effects such as receipts and notifications.

That expectation is the hidden state machine. When those operations disagree, the team has to reconstruct it after the fact. Teams often reach for saga-style coordination here: local transactions plus retries, compensation, or reconciliation, rather than one global commit. A saga coordinates local transactions, retries, and compensation, but it closes the authority gap only when aggregate ownership and the saga's own state model are explicit. Implicit, choreography-heavy workflows that are difficult to reason about can leave the same ambiguity intact.

Database commits and event publication illustrate the dual-write problem directly. They are separate writes. The transactional outbox records the state change and the event in one local transaction; a publisher then reads the outbox and delivers to the bus. The publisher may still deliver duplicates, so consumers remain idempotent. The outbox closes the gap between local state and publication - it does not resolve ambiguous external effects.

AI agents inherit the same problem

Agents do not create the authority gap. They make it easier to cross decision and execution boundaries dynamically, which is why the same vocabulary applies.

Traditional business logic usually follows an explicit branch:

if refund.amount > manual_review_limit:
    require_manual_review()

An AI-assisted process adds provenance the application must preserve: model and prompt versions, retrieved evidence, tool arguments, policy version, human approval identity, and the aggregate or state version used during acceptance. Anthropic distinguishes workflows, where models and tools follow predefined code paths, from agents, where the model directs its own process and tool use -the path through the system may be chosen at runtime.

A tool-using agent becomes another participant in the distributed workflow. It does not escape networks, timeouts, retries, stale state, or duplicate execution - it inherits them.

Suppose an agent decides to issue a refund. It calls the refund tool. The provider completes the refund, but the tool call times out before the result reaches the agent. From the agent's perspective, the tool failed. From the provider's perspective, the refund succeeded. From the application's perspective, the outcome is unknown.

If the agent treats every timeout as failure and calls the tool again, it can repeat a real-world side effect before anyone has reconciled the first attempt. The safety of that retry depends on guarantees outside the model: a stable idempotency key; an effect identity tied to the business request, not the agent attempt; a way to query the provider for the original result; an explicit representation of an unknown outcome; and a reconciliation or human-review path before another mutation runs.

Those guarantees must be enforced by the system around the agent, not left to the model's judgment.

The application still owns the transition

An AI system might return:

{
  "recommendation": "approve",
  "reason_codes": [
    "LOW_PRIOR_REFUND_RATE",
    "ACCOUNT_TENURE"
  ],
  "model_version": "refund-review-2026-07"
}

Even if the response is valid and correctly structured, it should not automatically become authoritative business state. The application still has to decide whether approval is allowed from the current state, whether amount or risk thresholds require human review, whether a deterministic policy overrides the recommendation, and whether another process has already completed the refund. The model recommendation and the application decision are separate records.

A model response can inform the decision, but the application still owns the state transition. A recommendation generated against stale state must not be accepted blindly. Keep the distinction visible in the history:

AI recommended approval
Application approved refund
Provider refund attempted
Provider request timed out
Provider confirmed refund
Refund completed

Collapsing those records into a single status = completed row may be convenient for reads, but it removes the explanation operators need when something goes wrong.

Composition is the hard part

Transactions, idempotency, outboxes, state machines, durable workflows, event sourcing, tracing, and reconciliation each solve an important part of the problem. A reliable system may need several of them.

The challenge is making them agree on the same answers:

What did the application accept as true, what actually happened outside it, and what can safely happen next?

In practice, those guarantees are scattered across code, infrastructure, provider contracts, and runbooks. Coverage is uneven: one workflow may be carefully designed, while another retries without a stable identity or relies on a reconciliation query only one engineer knows how to run.

The issue is usually not that the team lacks tools. It is that the business-state model was never made explicit - so when something partially succeeds, the team has to reconstruct authority from evidence spread across the system.

A system that can close the authority gap should be able to answer:

• What was requested?

• What became authoritative?

• Which effects were confirmed or remain unknown?

• What can safely happen next?

That is more than better logging. It is explainable state: not just a record of activity, but a durable account of the transitions the application accepted, the evidence and policy behind them, and what remains unresolved.

The next post will explore what it takes to make that model explicit: recording intent, separating attempts from committed facts, assigning stable identities to external effects, and making state reproducible and repairable.

The worst production state is not always one that is obviously wrong. Sometimes it is a state that looks right, but the system cannot prove how it got there.

References

1. Malcolm Featonby, AWS Builders' Library, "Making retries safe with idempotent APIs".

2. Apache Kafka documentation, "Message delivery semantics".

3. Google Cloud, "Pub/Sub subscription overview".

4. AWS Prescriptive Guidance, "Transactional outbox pattern".

5. AWS Prescriptive Guidance, "Event sourcing pattern".

6. AWS Prescriptive Guidance, "Saga choreography pattern".

7. Anthropic, "Building effective agents".