The interpretability problem

An AI agent's decision can be recorded in full — the inputs it received, the action it selected, the outputs it produced — and that record can be complete while telling an auditor almost nothing about whether the decision was correct. The audit trail answers what. Accountability requires understanding why. When the reasoning that produced a decision cannot be independently examined, you have a complete log and an empty accountability picture. This is the interpretability problem.

What the problem is

Interpretability, as used here, does not mean a simplified explanation produced after the fact. Post-hoc explanations — narrative summaries, feature-importance rankings, natural language justifications generated by the same model that made the decision — are reconstructions, not expositions. They describe a plausible chain of reasoning that could have produced the output; they do not expose the actual computational path. The difference matters for accountability: a reconstruction can be wrong without being detectable as wrong, because there is no ground truth to check it against.

True interpretability would allow an independent party to examine the agent's intermediate reasoning states, identify the factors that drove a decision, and verify that the stated justification and the actual computation are consistent. This is not available for most current AI agents operating at production scale. What is available instead are outputs, and the outputs can look correct even when the underlying reasoning would not survive scrutiny.

The accountability gap it creates

When an AI agent's decision causes harm and the reasoning that drove it cannot be examined, accountability reduces to outcome attribution. Investigators can establish that the agent acted, that the action preceded the harm, and that the action was within the agent's authorized scope. They cannot establish whether the reasoning was sound, whether the inputs were weighted appropriately, or whether a different framing of the same situation would have produced a different and better decision. The accountability record identifies the proximate cause; the interpretability gap conceals the structural one.

This matters disproportionately in domains where decisions are novel, high-stakes, and not fully covered by prior policy. Routine decisions in well-understood conditions can be evaluated by comparing outputs to expected outputs. Decisions at the edge — where the agent is doing what it was built to do but in a context its designers did not fully anticipate — can only be evaluated by examining the reasoning. Those are exactly the decisions most likely to cause harm, and they are the decisions for which interpretability is most often absent.

At the post-quantum crossing

A post-quantum migration agent operates in a domain where the principals it advises typically lack the cryptographic depth to evaluate its reasoning independently. When the agent recommends a particular algorithm choice, parameter configuration, or migration sequence, the recommendation is assessed against outcomes — did the migration complete without error? — not against reasoning. A subtly incorrect recommendation may produce outputs that pass all automated checks while resting on a flawed assessment of the threat model or algorithm properties.

The interpretability gap is especially severe here because the domain is one where errors are not self-correcting. A wrong cryptographic choice does not immediately manifest as a visible failure; it creates latent vulnerability that may not be exploited for years. By the time the reasoning error becomes apparent, the decisions it informed have been distributed across infrastructure, ratified in policies, and acted on by downstream systems. The audit log will show authorisation. It will not show whether the reasoning was sound when it was made.

At the hardware crossing

A fleet management agent that makes configuration decisions across large device populations must navigate interaction effects between device states, software versions, environmental conditions, and operational requirements. The reasoning that produces a particular configuration recommendation may depend on the joint state of thousands of variables that no single operator can reconstruct from the output alone. When a configuration change contributes to a device failure or a fleet-wide incident, the interpretability question is not "what did the agent do?" — the log answers that — but "why did it assess this configuration as acceptable?"

Without interpretability, post-incident review defaults to replacing the agent's recommendation with a different recommendation, produced by different humans using the same incomplete information. The structural conditions that produced the original flawed assessment remain unaddressed. Repeated incidents of the same type follow. The pattern is familiar in complex infrastructure management: symptoms are addressed, root causes are not, because the root cause lives in reasoning that cannot be opened.

In physical-world care

The interpretability problem is most acute at the care crossing because the right to understand a decision is itself a component of care. A person subject to a care agent's decisions — about routine support, about escalation to clinical attention, about how their condition is characterized in records that follow them — has a legitimate claim not just to know what was decided but to understand the basis for it. That understanding is necessary for meaningful consent, for informed override, and for the person's own agency over their care narrative.

A care agent that produces correct outcomes most of the time can still cause harm in specific cases through reasoning that privileges one pattern of data over another in ways the affected person cannot contest because they cannot see it. The interpretability gap in care is not a technical limitation the person should be expected to accept; it is a structural reduction in their capacity for self-determination. When a care agent's reasoning is opaque, the accountability architecture that surrounds it must compensate: more frequent human review, narrower autonomous scope, and mandatory channels for the person to register that an outcome did not match their understood intentions.

What accountability architecture requires

Interpretability cannot be fully achieved for current large-scale AI systems, and accountability architecture must be designed for that constraint rather than against it. The practical options are not "interpretable agent" or "uninterpretable agent" but rather: how should scope, oversight, and review intervals be calibrated to the interpretability level actually available?

Agents operating in low-interpretability conditions should operate in narrower scopes: tighter action boundaries, more frequent checkpoints, and more systematic logging of the information state at the moment of decision. The log cannot substitute for the reasoning, but a richer information snapshot at the decision point gives reviewers a better basis for assessing whether the output was plausible given what the agent knew.

Mandatory dissent windows — structured periods between a decision recommendation and its execution during which a human reviewer can object — serve a different function: they do not expose the reasoning, but they create moments at which independent judgement can intervene. The value of a dissent window depends entirely on the reviewer having enough information to form a genuine position, which requires readable decision context rather than raw model output.

The deepest requirement is that interpretability be treated as a first-class property in agent deployment decisions, not an aspirational capability to be added later. An agent deployed in a domain where its reasoning cannot be examined is an agent whose accountability architecture is structurally incomplete from day one. Recognizing that incompleteness is not an argument against deployment; it is an argument for building the compensating controls before deployment, not after an incident forces the question.