Some of the hardest production incidents are not full outages. They are the incidents where an operation finished in one place and not another.

A payment provider accepts a charge, but the HTTP client times out before receiving the response. A database write commits, but the event that should have followed never leaves the process. A worker performs an external action, crashes before acknowledging the message, and the queue delivers it again. The application stays online. Dashboards stay green. But one payment, order, account, shipment, or dispute no longer has a trustworthy story-and those failures often surface first as a support ticket or reconciliation mismatch, not a clean application error.

Investigating these incidents is not just a debugging exercise. It requires answering questions about business state: what the company currently believes, correctly or incorrectly, is true about a customer, a balance, an order, or a decision. Most systems have plenty of evidence-logs, traces, database rows, queue histories, provider dashboards, audit records, support notes. What they often lack is an explicit contract for turning that evidence into a durable domain record the application accepted.

The system can show activity, but it cannot explain its own conclusion. Call that gap the authority gap: a system has evidence that work was requested or attempted, but lacks a durable domain record identifying the transition it accepted, the evidence and policy that justified it, and which effects remain confirmed, failed, or unknown.

Authority Gap · Part 1 - When systems record activity but cannot explain what they accepted as true.

Authority is scoped, not universal. The payment provider owns whether it created an external refund. The application owns which domain transition it accepted. Projections are derived views built from accepted facts. Webhooks and callbacks are observations until validated and accepted into the application's model. No single record establishes truth across every participating system.

An authoritative application fact is not a claim about everything that happened in the world. It is a durable record of what the application accepted, based on the evidence available to it at that point.

I did not arrive at that idea through architecture diagrams. I arrived at it while investigating a production system where the infrastructure showed the work as consumed, but callbacks and downstream operations were still missing.

Consumed does not mean completed

The consumer handled business-critical banking events: callbacks, ACH uploads, and other work that could not simply disappear.

The service looked healthy. Messages were moving through the consumer group, and nothing suggested a broad outage. But if the container restarted after receiving a message and before completing the business operation, that work could be lost.

The failure came from a bad combination of behaviors. The consumer had automatic offset commits enabled, so the group could advance its position on the poll interval while handler work was still in flight. The consumer group's committed offset could move past a record before the business operation associated with that record had durably completed. Exceptions were being masked instead of driving the message through an intentional failure path. There was no dead-letter queue or durable record showing that an event had been received but never successfully applied.

The committed offset told Kafka to resume after the record, even though the associated business operation had not completed. A committed offset is a restart checkpoint, not proof that the business operation succeeded. From the application's perspective, the callback had never happened, the ACH upload had never completed, or the downstream operation was still missing.

The question was not whether Kafka delivered the message. It had. The question was whether the business operation represented by that message completed, and there was no single place that could answer it.

The fix was to commit offsets only after successful processing, make handlers idempotent, stop masking exceptions, retry or re-seek failed records intentionally, route exhausted failures to a dead-letter queue, and test restart mid-processing.

Committing after processing traded silent loss for possible redelivery. Idempotency was still required because the external or business effect could complete before the offset commit. Committing before the business effect can lose work; committing after it can duplicate work after a crash.

Kafka transactions do not automatically make arbitrary external side effects exactly once. The dead-letter queue preserves failed work for investigation, but it does not establish whether an external effect already occurred.

Those changes addressed the immediate message-loss problem. Kafka had moved on. The application still could not say whether the business operation behind the message had been accepted as a domain fact.

The refund that may have happened twice

Application code often treats a remote operation as having two outcomes: success or failure. Distributed systems usually add a third-UNKNOWN_OUTCOME-when the caller stops waiting before learning what happened. That third outcome shows up quickly in payment workflows.

A customer requests a refund. The application creates a pending record keyed by refund_intent_id, the logical business request, and runs eligibility checks. Each worker execution receives its own attempt_id, while every submission for the same refund reuses an idempotency_key tied to the original business intent.

The provider completes the refund and assigns a provider_refund_id. The response never reaches the worker. From the worker's perspective, the request timed out, so it schedules a retry. Moments later a webhook arrives with its own webhook_event_id-one external notification delivery-confirming the original refund succeeded, and the retry may already be running.

None of this requires an unusual edge case. But once the refund reaches an unknown outcome, the team still has to answer uncomfortable questions: Did both submissions represent the same logical refund? Did the retry start before the webhook arrived? What action is safe now?

Evidence is not authority

The refund example leaves behind accurate records at every step-request, review, submission, timeout, retry, webhook, status change-yet still may not answer what matters:

What did the application authoritatively accept as true, and why?

A completed row is enough to serve an API response:

{
  "id": "refund_123",
  "status": "completed",
  "amount_cents": 12500,
  "currency": "USD",
  "updated_at": "2026-07-27T13:42:11Z"
}

It does not explain who requested the refund, why it was allowed, how many provider attempts occurred, or which attempt completed. Logs, traces, queue histories, and provider dashboards describe execution. They are not, by themselves, a record of acceptance.

Authority gaps often appear when these layers are stored without being distinguished:

The state machine already exists

Business applications almost always contain state machines, whether anyone designed them or not. The rules are often spread across controllers, workers, webhook handlers, database constraints, provider integrations, and support tools. No single component owns the complete model.

Consider what has to happen before a payment is truly complete inside the application:

Provider confirms the payment
The ledger entry commits
The application accepts PaymentCompleted
The order projection becomes paid
Receipt delivery remains pending

Each step maps to a different layer: observation, local fact, accepted transition, projection, asynchronous consequence. When those steps disagree, the team reconstructs the hidden state machine after the fact.

Teams often reach for saga-style coordination here. A saga closes the authority gap only when aggregate ownership and its state model are explicit.

Event sourcing helps by preserving an ordered history of accepted transitions rather than only the latest row. The transactional outbox closes a different gap: it records the state change and the outbound event in one local transaction. The publisher may still deliver duplicates, and neither pattern resolves ambiguous external effects on its own.

AI agents inherit the same problem

The same refund failure looks familiar when an agent is in the loop.

Suppose an agent decides to issue a refund. It calls the refund tool. The provider completes the refund, but the tool call times out before the result reaches the agent. From the agent's perspective, the tool failed. From the provider's perspective, the refund succeeded. From the application's perspective, the outcome is unknown.

If the agent treats every timeout as failure and calls the tool again, it can repeat a real-world side effect before anyone has reconciled the first attempt. The safety of that retry depends on guarantees outside the model: a stable idempotency key tied to the business request, not the agent attempt; an explicit unknown outcome; and a reconciliation path before another mutation runs.

Those guarantees must be enforced by the system around the agent, not left to the model's judgment.

The application still owns the transition

An AI system might return:

{
  "recommendation": "approve",
  "reason_codes": [
    "LOW_PRIOR_REFUND_RATE",
    "ACCOUNT_TENURE"
  ],
  "model_version": "refund-review-2026-07"
}

Even if the response is valid and correctly structured, it should not automatically become authoritative business state. The application still has to decide whether approval is allowed from the current state, whether thresholds require human review, and whether another process has already completed the refund. The model recommendation and the application decision are separate records.

A model response can inform the decision, but the application still owns the state transition. A recommendation generated against stale state must not be accepted blindly. Keep the distinction visible in the history:

AI recommended approval
Application approved refund
Provider refund attempted
Provider request timed out
Provider confirmed refund
Refund completed

Collapsing those records into a single status = completed row may be convenient for reads, but it removes the explanation operators need when something goes wrong.

Composition is the hard part

Transactions, idempotency, outboxes, state machines, durable workflows, event sourcing, tracing, and reconciliation each solve part of the problem. The hard part is making them agree.

After an incident, the pieces often tell different stories:

Outbox says the event was published
Provider effect remains unknown
Projection shows completed
Workflow engine is still retrying
Support manually updated the status

None of those records is obviously wrong on its own. Together they do not answer what the application accepted, what happened outside it, or what can safely happen next. In practice, coverage is uneven: one workflow may be carefully designed, while another retries without a stable identity.

A system that can close the authority gap should be able to answer:

• What was requested?

• What became authoritative?

• Which effects were confirmed or remain unknown?

• What can safely happen next?

This is more than better logging. It is explainable state: a durable account of the transitions the application accepted, the evidence and policy behind them, and what remains unresolved.

Follow along for Part 2, where I'll turn this vocabulary into a concrete model for recording intent, accepting facts, tracking external effects, and repairing state safely.

The worst production state is not always one that is obviously wrong. Sometimes it is a state that looks right, but the system cannot prove how it got there.

References

1. Malcolm Featonby, AWS Builders' Library, "Making retries safe with idempotent APIs".

2. Apache Kafka documentation, "Message delivery semantics".

3. Google Cloud, "Pub/Sub subscription overview".

4. AWS Prescriptive Guidance, "Transactional outbox pattern".

5. AWS Prescriptive Guidance, "Event sourcing pattern".

6. AWS Prescriptive Guidance, "Saga choreography pattern".

7. Anthropic, "Building effective agents".