THE AUTHORITY GAP · PART 2

A worker asks a payment provider to issue a refund.

The request times out.

The provider may have completed it. It may not have. The application does not know.

What should happen next?

If the system records FAILED, it is claiming more than the evidence supports. If it blindly retries, it may issue the refund twice. If it waits forever, the workflow stalls.

This is not primarily a transport problem.

It is a problem of what the application knows — and what it is allowed to accept as true.

In Part 1: When Your Application Can't Explain Its Own State, I called the broader version of this the authority gap: the point where a system has evidence that work was requested or attempted, but cannot clearly explain what it ultimately accepted as true.

The primitives involved are familiar: idempotency, state machines, reconciliation, event sourcing, durable messaging, optimistic concurrency, projections.

The harder problem is how those pieces connect.

Distributed systems have delivery semantics. They also need acceptance semantics.

Delivery semantics tell you whether information moved or processing occurred.

Acceptance semantics tell you under what conditions evidence is allowed to change application state.

I use three related terms deliberately:

• Acceptance semantics are the rules that determine whether evidence may change state.

• The acceptance boundary is where those rules are applied.

• An accepted transition is the durable result: the state change the application accepted, along with the evidence and prior state that justified it.

The rules, the place they run, and the result they produce are related.

They are not the same thing.

The model at a glance

For an important business operation, the chain usually looks something like this:

Figure 1 — The important part is not the individual records. It is preserving the causal links between them.

Intent
  ↓
Attempt
  ↓
Observation
  ↓
Accepted Transition
  ↓
Projection

Each stage answers a different question:

What was requested?
What execution did we try?
What evidence arrived?
What did the application accept?
What do we show now?

When all five collapse into:

status = completed

the application keeps the answer and loses the explanation.

One intent can have many executions

Return to the refund.

The business operation needs a stable identity:

refund_intent_id = refund_123

That identity should survive retries, worker restarts, queue redelivery, webhook delivery, and reconciliation.

An execution is different:

refund_123
├── attempt_001 → outcome unknown
└── attempt_002 → retry / reconciliation

Both attempts can belong to the same refund.

This is why idempotency is not merely an HTTP concern. It is part of the domain model.

Without stable intent identity, the system cannot reliably distinguish:

Try the same refund again

from:

Issue another refund

That decision cannot be delegated to the retry mechanism.

A timeout creates an information failure

The first request times out.

A naive implementation translates that into:

REFUND_FAILED

But the evidence only supports:

PROVIDER_OUTCOME_UNKNOWN

The caller stopped waiting before learning the result.

The provider may still have moved money. An email may still have been delivered. A shipping label may still have been created. An AI tool call may still have changed another system.

A timeout does not necessarily mean execution failed.

It means the application does not know what happened.

That makes retry safety partly a question of knowledge:

What do we know strongly enough to act again?

Figure 2 — Unknown is not failure. It is a state representing insufficient knowledge.

Instead of:

timeout → retry mutation

the workflow becomes:

timeout
   ↓
outcome unknown
   ↓
reconcile
   ↓
confirm / fail / retry safely

The next action is based on what the system knows, not merely on which exception was thrown.

Evidence is not state

Now the provider sends:

refund.completed

That webhook matters.

But it is not automatically application state.

It is an observation.

Before it can become RefundCompleted, the application still has to decide:

Was it authenticated?
Was it already processed?
Which intent does it belong to?
Does the amount match?
Has another transition already won?
Is completion legal from the current state?

Those checks are the application's acceptance semantics.

The place where those rules run is the acceptance boundary.

Only after the evidence crosses that boundary can the application record an accepted transition:

RefundCompleted

Figure 3 — External evidence crosses into application state only after the application's acceptance semantics allow it.

The provider owns whether it actually created the refund.

The application does not get to redefine that fact.

But the application does own whether that evidence is sufficient to move its own domain state forward.

So RefundCompleted means something precise:

Based on this evidence, under these rules, against this version of state, this application accepted completion as authoritative for its own model.

That is more useful than saying:

The database is the source of truth.

Different systems can be authoritative over different facts.

Delivery semantics are not acceptance semantics

Distributed-systems design spends a lot of time on:

at-most-once
at-least-once
effectively-once
exactly-once within some boundary

Those guarantees matter.

But they answer transport and processing questions.

They do not answer:

Should this evidence be allowed to alter business state?

A webhook can be delivered exactly once and still be invalid.

A Kafka message can be processed successfully and still be stale.

A workflow can execute perfectly and still make the wrong business transition.

Reliable delivery and reliable acceptance are different properties.

Did the information arrive correctly?

Was it accepted correctly?

Most architectures design the first explicitly.

The second is often scattered through application code.

Make the difference visible

A typical system may leave you with this:

refunds.status = 'completed'

log:
"processed refund webhook"

The row tells you what state won.

The log tells you some code ran nearby.

Neither tells you why that state change was justified.

An accepted transition carries the missing provenance:

RefundCompleted

intent:           refund_123
evidence:         provider_event_921
previous_version: 4
version:          5

Now the application can say:

The provider reported refund rf_88421 completed.

That observation correlated to refund intent refund_123.

The acceptance semantics allowed RefundCompleted
against version 4.

RefundCompleted became version 5.

That is the difference.

Logs explain execution. The application history should explain state.

Projections are causal compression

You still want:

{
  "id": "refund_123",
  "status": "completed",
  "amount_cents": 12500
}

Read models are useful because they compress:

RefundRequested
ProviderAttemptStarted
ProviderOutcomeUnknown
ProviderCompletionObserved
RefundCompleted

into:

status = completed

A projection is a form of causal compression: a rich sequence of causes reduced to a convenient present-tense answer.

The problem is not compression.

The problem is when it becomes irreversible.

If the projection is the only record that survives, status = completed stops being a summary.

It becomes all the system knows.

The same pattern appears outside payments. A security signal is not automatically AccountSuspended; it is evidence that policy may or may not allow to become an account transition. The nouns change. The structure does not.

One business operation, several partial state machines

The most dangerous version of this problem appears when transition rules are spread across workers, webhook handlers, admin tools, cron jobs, and support scripts.

Consider the refund again.

The worker sends the request and times out, then schedules a retry.

Before that retry runs, the provider webhook arrives and marks the refund complete.

At roughly the same time, support sees the original operation as failed in an admin screen and manually retries it.

Worker
  ↓
timeout
  ↓
schedule retry


Webhook handler
  ↓
provider says completed
  ↓
status = completed


Support tool
  ↓
sees failure
  ↓
manual retry

Every path is locally reasonable.

The worker is reacting to its timeout.

The webhook handler is reacting to provider evidence.

Support is reacting to what the application shows them.

Together, they can issue the same refund twice.

The bug is not necessarily inside any one component.

The bug is that each path is allowed to independently decide what the business state means.

A better shape is:

Worker timeout ────────┐
                       │
Provider webhook ──────┼──→ intent / observation
                       │          ↓
Support retry ─────────┘   acceptance semantics
                                  ↓
                           accepted transition
                                  ↓
                              projection

Now the paths can race, retry, disagree, and arrive in different orders without each becoming its own authority over business state.

Different inputs.

Different transports.

One acceptance boundary.

Reconciliation and repair belong in the model

If a workflow can reach:

ProviderOutcomeUnknown

then it needs a designed way out.

A reconciliation worker might query the provider and discover the refund completed.

That response is simply another observation.

The transport changed. The authority did not.

The same acceptance semantics can evaluate it and produce RefundCompleted.

This is why reconciliation is not cleanup around the real architecture.

For distributed side effects, it is part of the architecture.

The same principle applies to repair.

Suppose later evidence shows that an accepted transition was wrong.

The fastest fix may be:

UPDATE refunds
SET status = 'failed'
WHERE id = 'refund_123';

That fixes the screen.

It also destroys the explanation.

The normal repair path should preserve what changed:

RefundCompleted          version 5
      ↓
contradictory evidence
      ↓
RefundStatusCorrected    version 6
      ↓
projection = failed

The historical truth is not that the refund was always failed.

It is that the application accepted completion based on the evidence available at the time, then accepted a correction when better evidence arrived.

Sometimes the correct 3 a.m. incident response really is emergency SQL.

The point is not to pretend that never happens.

The goal is to make corrective transitions the normal repair path, and destructive mutation an exceptional operational tool whose provenance is captured afterward.

Existing patterns still matter

None of this replaces familiar reliability patterns.

The point is not that those patterns are incomplete.

It is that their boundaries need to connect.

A system can be excellent at reliably moving messages and still be unable to explain why its business state changed.

AI makes the boundary more important

The same model applies when an AI agent recommends an action:

{
  "recommendation": "refund",
  "amount_cents": 12500,
  "reason": "MERCHANT_NON_FULFILLMENT"
}

That is not a refund.

It is a proposed decision produced by probabilistic reasoning.

The application still needs to decide whether the action is allowed, whether the amount is valid, whether state changed while the model was reasoning, and whether human approval is required.

Models can propose. Applications still need explicit rules for what becomes authoritative state.

That boundary becomes more important as agents gain tools, because probabilistic decisions can now create deterministic side effects.

Why was this allowed to become true?

The primitives in this article are familiar.

The stronger architectural idea is preserving the causal relationship between them:

intent
  ↓
attempt
  ↓
observation
  ↓
accepted transition
  ↓
projection

A system should be able to move backward from:

refund = completed

to:

which accepted transition created it
which observation justified that transition
which attempt produced or awaited that observation
which original intent caused the attempt

without relying on an engineer to reconstruct the answer from logs, traces, queues, dashboards, and timestamps.

A database can tell you:

What is true right now?

An explainable application should also be able to answer:

Why was this allowed to become true?

That is the difference between keeping state and keeping enough causal history to understand it.

Next in the series

Part 3: How to make that causal history inspectable without rewriting the rest of your application around it.

References

1. Malcolm Featonby, AWS Builders' Library, Making retries safe with idempotent APIs.

2. AWS Prescriptive Guidance, Transactional outbox pattern.

3. AWS Prescriptive Guidance, Event sourcing pattern.

4. Apache Kafka, Design — Message Delivery Semantics.

5. Anthropic, Building effective agents.