The synthetic evidence problem: accountability when AI agents are trained on AI-generated data

The standard model for AI agent development assumes that training data — whatever its source — represents ground truth in some meaningful sense. It may be noisy, biased, or incomplete, but it is assumed to be generated by processes independent of the agent being trained. This assumption is no longer reliable.

As AI systems become prevalent across industries, a growing share of the world's digital output is generated by those systems. Security advisories are summarized by AI. Hardware diagnostics are interpreted by AI. Care notes are drafted by AI. The data pipelines that feed the next generation of agents increasingly contain outputs from the previous generation — synthetic evidence generated by agents whose own accountability was uncertain when they produced it.

This is the synthetic evidence problem: when an AI agent is trained on data generated by other AI agents, it inherits its predecessors' biases, errors, and accountability gaps in a form that is difficult to trace, harder to audit, and embedded in model weights in a way that cannot be examined directly. The accountability architecture reviews the data, confirms it came from reputable sources, and certifies the agent as trained on authoritative material. What it cannot readily certify is whether that material was itself shaped by prior AI systems whose unaudited optimization patterns are now silently reproduced in the downstream agent.

At the post-quantum security crossing

Key management agents trained on historical recommendations, advisory summaries, or audit narratives that were themselves drafted by AI-assisted tooling may carry forward systematic biases that originated several steps upstream. If an earlier AI advisory process overrepresented certain algorithm families because of artifacts in its own training, a downstream key management agent trained on those advisories inherits that overrepresentation without any record of its origin.

The accountability gap is structural: a cryptographic agent that consistently favors one migration path over another because a prior AI system systematically emphasized it is not behaving badly by any direct measure. Its training data is traceable to credentialed sources; its outputs are within defined parameters. The error lives in the gap between the document author — a recognized institution — and the evidentiary claim inside the document, which reflects a prior model's distribution rather than independent expert judgment. Standard data governance does not draw the boundary at the claim level. Until it does, inherited cryptographic biases will persist through successive agent generations, invisible to the accountability audit.

At the hardware crossing

Hardware AI agents that evaluate device health and security posture train on large corpora of sensor readings, test results, and failure reports. As AI systems increasingly assist in hardware testing and reporting, those corpora contain synthetic components: test summaries generated by AI testing assistants, failure narratives drafted by AI diagnostic tools, attestation reports processed by AI intermediaries before they reach the training pipeline.

When a hardware health agent encounters an anomaly pattern that resembles a pattern described in AI-generated diagnostic summaries, it is not reasoning from first principles about the physical state of the device. It is pattern-matching against text produced by prior agents. If those prior agents systematically underreported a failure class because it fell outside their training distribution — a new failure mode introduced by a hardware revision they had not seen — the downstream agent learns that underreporting as a feature of how diagnostics are written, not as an error to correct. The failure mode disappears not because it is remediated but because the evidence generation chain learned not to describe it. The device population carries a structural blind spot, and the accountability record shows nothing unusual.

At the physical-world care crossing

The synthetic evidence problem is most acute in care settings because the evidentiary ground truth for care quality is contested even when produced by humans. When AI-assisted documentation tools generate care notes, AI-assisted assessment tools summarize patient status, and AI-assisted care planning tools produce recommendations, the data pipelines for training next-generation care agents are increasingly populated by synthetic outputs.

A care agent trained on AI-generated care notes inherits whatever systematic patterns those notes contain — and those patterns reflect the prior generation agents' optimization targets, their blind spots, and the proxies they were measured against. If prior AI documentation tools generated notes that emphasized measurable care activities over subjective wellbeing indicators — because measurable activities were easier for upstream classifiers to validate — a downstream care agent trained on those notes treats measurable activities as the primary signal of good care, because that is what the evidence record consistently shows. The accountability gap is invisible at the level of the training data review: the notes look like care notes. What the review cannot see is that they are the product of an earlier optimization process that already discarded the harder-to-measure signals.

The provenance boundary

The structural response to the synthetic evidence problem is a provenance boundary: a requirement that training data pipelines document the generation process for each substantive claim, distinguish human-validated from AI-generated sources, and apply higher scrutiny to AI-generated material — including independent verification against non-AI ground truth where possible.

This is harder than it sounds. The boundary between human-generated and AI-generated evidence is already blurred: a security advisory written by an analyst using AI drafting assistance has a human author but AI-shaped content. The provenance boundary must be drawn at the level of the evidentiary claim — whether each specific claim was independently validated against a non-AI source — not at the level of the document author. Without that boundary, the accountability architecture cannot distinguish inherited gaps from original errors. And the proportion of training data that carries unaudited, inherited gaps will only increase with each successive deployment generation, because each generation of agents generates more of the data that trains the next.