The instruction collapse problem: accountability when AI agents lose the nuance of their authorizing instructions

At the start of a deployment, the instructions given to an AI care agent are typically detailed and conditional: contact the care coordinator if the recipient refuses medication three times in a row, but not if refusal follows a pattern consistent with documented preference; escalate at night only if vital signs cross specified thresholds, except on Thursdays when the overnight nurse has already been notified. The specificity is deliberate. It reflects hours of negotiation between operators, clinicians, and families about exactly when agent autonomy ends and human judgment must begin.

Months into deployment, those instructions still nominally govern the agent. But the agent is no longer operating from the original text. It is operating from a compressed representation — a summary of summaries, the residue of context windows that have been collapsed many times over to make room for more recent operational data. The instructions are still present in some form. The nuance that made them accountable is not.

This is the instruction collapse problem: not a failure of alignment, but a failure of fidelity — the gradual erosion of the conditional logic that gave an agent's authority its shape.

Why compression is structurally unavoidable

Long-horizon AI agents face a hard constraint. Context windows are finite. A care agent that operates continuously — logging observations, recording interactions, tracking care events — fills its available context faster than the deployment stretches. To keep acting, it must compress: summarizing earlier context into shorter representations that free space for the present.

Compression is not failure; it is the normal operating mode of any system that must act over time. The problem is what compression does to conditional instructions. A sentence that reads "do X unless condition Y, in which case do Z but only if criterion W has been met in the preceding 72 hours" does not compress gracefully. The conditions, the exceptions, and the timing qualifiers are exactly what make the instruction safe — and they are also exactly the kind of detail that compression loses first. The resulting summary may read "do X when relevant" — which is not wrong, exactly, but is no longer the instruction that was authorized.

The accountability signature of collapsed instructions

The instruction collapse problem is particularly difficult to detect because it does not produce obvious error. An agent operating from a collapsed representation of its instructions will still look mostly correct. Most of the time, the simplified version and the original version produce the same action. The divergence appears at the margins — in edge cases, in threshold conditions, in the exactly-the-situations-we-specified-for-a-reason moments that conditional instructions were written to cover.

From an oversight perspective, this is the worst kind of drift. The agent's actions are individually defensible. The log entries make sense. There is no single decision that is clearly wrong. What has gone wrong is that the agent is no longer governed by the detailed, negotiated mandate it was given — it is governed by an approximation of it, and the approximation is no one's responsibility. The original instructions authorized the deployment. Nobody authorized the compressed version the agent is actually running on.

The parallel in cryptographic protocol deployment

In security-critical systems, the instruction collapse problem has a direct structural parallel. Cryptographic protocols are initially deployed with detailed configuration documents: which cipher suites are permitted, which are forbidden, what the fallback hierarchy is, how key negotiation should proceed under degraded conditions. Over time, those documents are summarized — into operational runbooks, into abbreviated policy references, into institutional memory. The summary is assumed to capture the essential intent. Years later, a configuration choice is made from the summary rather than the original document, and the choice is technically coherent with the summary but violates a constraint from the original specification that the summary elided.

The post-quantum cryptographic transition makes this failure mode newly urgent. Migration instructions for care-adjacent hardware — embedded systems in medical devices, secure enclaves in care terminals — are among the most conditioned specifications in the field. They involve exceptions for legacy compatibility, timing windows, hardware attestation requirements, and fallback procedures that depend on specific version conditions. An agent responsible for coordinating or verifying a migration that is operating from a compressed representation of those instructions may believe it has followed the protocol while having systematically bypassed the safety-critical conditional branches that distinguished "migration complete and verified" from "migration superficially complete."

What accountability requires

The instruction collapse problem points toward a specific accountability requirement: the authorizing instruction set must be separately versioned and preserved, and the agent's operating context must be periodically reconciled against it. This is not just a technical practice; it is an accountability practice. If the original instructions are not preserved in recoverable form, the deployment has no canonical standard against which agent behavior can be evaluated. The record of what the agent was told to do has itself been lost.

In care settings, this argues for treating the initial instruction set as a governed artifact — not just a configuration file. It should be version-controlled, signed by the parties who authorized it, and checked against the agent's compressed operating context at intervals defined by the sensitivity of the deployment. Where the compressed context diverges from the original instruction set in ways that would change agent behavior in foreseeable conditions, human review is required before deployment continues.

At Asaptic Labs, we think the instruction collapse problem is underweighted in current AI care governance frameworks, which tend to focus on behavioral drift during training rather than fidelity loss during inference. The distinction matters. Training-time alignment drift can in principle be detected through behavioral testing. Inference-time instruction collapse is invisible to behavioral tests — the agent will pass them — and is only detectable by comparing the agent's operating context against the original authorizing mandate. That comparison requires the original mandate to exist. In many long-running deployments, it does not.