The distillation gap: accountability when AI agents are compressed for hardware deployment

The term "distillation" covers a family of techniques — quantisation, knowledge distillation, pruning, low-rank factorisation — that share a single practical goal: taking a model whose computational requirements were designed for data centers and making it small enough to run on hardware that fits in a pocket, a ward, or a wall socket. These techniques are mature. The deployment economics are compelling. The accountability infrastructure for the transition has not kept pace.

The core problem is this. A large model is evaluated for safety, tested against edge cases, and subjected to systematic adversarial probing that produces a reasonable basis for trust. That evaluation is expensive and time-consuming, and it is conducted once — on the original model. When the model is compressed for hardware deployment, the compressed version is a different model. Not radically different in expectation: if the compression is done carefully, average-case behavior is preserved. But at the margins — low-probability inputs, novel context combinations, the edge cases that matter most in safety-critical settings — the compressed model may diverge from the original in ways that are not easily predicted and are rarely re-tested.

This divergence is the distillation gap. It is not a bug in any particular implementation. It is a structural feature of the relationship between model capacity, edge deployment requirements, and the accountability chain that connects them.

Why compression changes edge-case behavior

Quantisation, which reduces the numerical precision of model weights, is known to affect model behavior more sharply in regions of the input space that the training data covered sparsely. This is because the model's learned representations are most finely differentiated — and therefore most sensitive to precision loss — precisely in the areas where training signal was weakest.

Physical-world care is exactly the domain where sparse coverage matters most: an unusual combination of medications, an atypical vital-sign pattern, a care need that does not fit the standard classification. The model's trained behavior in these regions was marginal to begin with. Compression makes those margins less predictable.

Pruning has an analogous issue. When the lowest-contribution weights are removed from a model, "lowest contribution" is measured against the training distribution. The pruned capacity may have been doing nothing visible on average while being essential for correctly handling some narrow but consequential class of cases. In a care context, that narrow class might be a rare drug interaction that the larger model had implicitly learned to recognize. In a hardware security context, it might be a corner-case firmware state that the agent was trained to treat as suspicious. Pruning removes it silently. There is no mechanism in the standard compression pipeline to flag that a capability has been lost, because the pipeline has no concept of which capabilities matter outside the training distribution.

The hardware crossing: certification and the compression seam

At the hardware crossing, AI agents make or interpret decisions about physical devices — which firmware states are acceptable, which sensor readings warrant an alert, which attestations are valid. These are policy decisions embedded in the agent's behavior, and they are the decisions most likely to be evaluated and certified before deployment.

The distillation gap creates a seam in that certification. The original developer certifies the large model. The hardware integrator compresses it. The care operator deploys the compressed version. The certifying body reviewed — which version? In practice, the compressed model is deployed under the authority of the original certification, because re-evaluating a distilled model carries the same cost as evaluating the original. The original evaluation is cited as supporting evidence, but it is evidence for a different model.

This is not a hypothetical failure mode. It is the default operational pattern. When a compressed model deployed on medical hardware behaves differently from the certified original — even once, in a low-probability edge case — the certification chain has been broken at the compression step. The accountability infrastructure does not surface this because it was never designed to treat compression as a distinct point of accountability. It treats compression as a performance optimization, which is all it is from an engineering standpoint. From an accountability standpoint, it is a model replacement.

The care crossing: silent capability loss where it matters most

At the physical-world care crossing, the distillation gap has the most immediate human consequences. Care agents are deployed precisely because the settings they operate in are under-resourced: too few practitioners, too many care recipients, too much demand on human attention. The care agent is supposed to cover the gaps. But if the compressed care agent has silently lost the capability to handle the edge cases that the original model had learned — the unusual presentations, the rare contraindications, the atypical symptom patterns — then it has lost exactly the capability that the under-resourced setting most needs.

This is the perverse arithmetic of the distillation gap in care contexts. Compression is attractive precisely because it enables deployment in resource-constrained environments. But the resource constraints that make compression attractive are also the constraints that make edge-case coverage most critical. The model gets deployed where the margin for error is smallest, in a form that makes its margins least predictable.

Care recipients cannot inspect a model's compression parameters. They cannot compare the behavior of the deployed version against the certified original. They have no reliable way to know whether the agent advising them has the edge-case coverage that was verified — or whether it was quietly left behind at the compression step.

What closing the gap requires

First, evaluation parity. A compressed model deployed in a safety-critical or care context should be evaluated independently, not as a derivative of the original model's evaluation. The burden of proof sits with the compressed version. Citing the original certification as evidence is not the same as having evidence for the deployed model.

Second, provenance attestation for the compression process. Just as hardware attestation roots trust in the physical properties of a chip, model attestation should document every compression step: which techniques were applied, on which training distribution, with what fidelity thresholds, verified by whom. This is not overhead — it is the engineering record that makes post-incident accountability possible.

Third, scope-bounded deployment. A compressed model whose evaluation covers a specific input distribution should operate only within that distribution, with explicit monitoring for out-of-distribution inputs and a defined escalation path when they are detected. "This model was evaluated on scenario class X; it should not operate autonomously on scenario class Y" is a deployable constraint. Absent that constraint, the operator is implicitly claiming that the compressed model is safe across the full range of inputs the original covered — a claim for which they typically have no evidence.

Fourth, separation of the compression step in the accountability chain. When a deployed agent causes harm and the root cause is traced to a behavioral difference introduced by compression, the question "was this a failure of the original model or a failure of the compression process?" should have a definitive answer. It currently does not. The compression step should be treated as a distinct point of accountability, with its own documentation and its own chain of responsibility — not absorbed into the deployment decision as a technical detail.

The distillation gap matters most at precisely the intersection of two demanding deployment requirements: the need to bring AI capabilities close to the physical world, and the need for those capabilities to be trustworthy in domains where the physical world is most consequential. A care agent that fails silently because its compression removed the edge-case coverage it most needed is not a theoretical risk. It is a deployable product today.

The accountability infrastructure for that product is not.