🧪 Data and Model Poisoning (LLM04): Attacks and Defense

Data and Model Poisoning (OWASP LLM04:2025) is an attack on AI integrity in which an adversary manipulates data during pre-training, fine-tuning, or the building of a RAG knowledge base, planting backdoors, distortions, and hidden behavior that activates on a trigger. It is one of the most insidious threats: a poisoned model behaves normally on every ordinary input and only "comes alive" when a secret trigger appears. This guide walks through the taxonomy of poisoning (from web-scale datasets to the RAG corpus), research-confirmed figures, and real incidents — and shows honestly where the boundary runs: the primary defense against poisoning lives at the MLOps and supply-chain stage, while SYNTREX — the defense layer of the Spectorn platform — closes the runtime part: detection of dormant payloads and an immutable audit trail.

Spectorn is a security and compliance-perimeter platform; SYNTREX is its AI-defense layer, deployed both inside Spectorn and standalone. Poisoning is predominantly a training-stage problem; SYNTREX complements MLOps controls with runtime detection of trigger payloads and a provable trail, but it does not replace protection of the training pipeline.

This page goes deeper on the LLM04 category from the OWASP Top 10 for LLMs overview. The adjacent topic of poisoning the RAG corpus specifically is covered in detail in the RAG security guide.

What Changed: From "Training Data Poisoning" to "Data and Model Poisoning"

In the 2023 edition this category was called "Training Data Poisoning" and covered only the training set. In 2025 OWASP broadened it to "Data and Model Poisoning," taking in three lifecycle stages:

Pre-training — contamination of the general knowledge corpus;
Fine-tuning / instruction-tuning — poisoning of the task-adaptation data;
Embeddings / RAG knowledge base — poisoning of the retrieval corpus.

The main reason for the expansion: attacking the RAG corpus is many times cheaper than retraining the model, with a comparable effect. Here's why.

A Taxonomy of Poisoning Attacks

Poisoning web-scale datasets: it's practically feasible

The Carlini et al. study "Poisoning Web-Scale Training Datasets is Practical" (arXiv:2302.10149, 2023/2024) showed that poisoning 0.01% of the LAION-400M or COYO-700M datasets cost about $60. Two techniques:

Split-view poisoning — exploiting the mutability of web content: the dataset curator sees one version of the content at snapshot time, while those who download later get a different one (the attacker changed what's behind the URL).
Frontrunning poisoning — for crowdsourced datasets with periodic snapshots (Wikipedia, for example): the attacker injects content in the brief window before a snapshot, then rolls it back.

Ten popular datasets turned out to be vulnerable; the authors notified all maintainers.

Backdoors and trojans: dormant until the trigger

The foundational BadNets work (Gu et al., 2019) demonstrated the mechanism: a subset of training examples is marked with a trigger and relabeled to the attacker's target class. On clean inputs the model behaves normally; on inputs with the trigger it produces the result the attacker wants. A distinction is drawn between targeted poisoning (an error on specific inputs), availability poisoning (degradation of overall accuracy), and the backdoor/trojan (dormant behavior activated by a trigger at runtime).

Backdoors that survive safety training

The "Sleeper Agents" work (Hubinger et al., Anthropic, arXiv:2401.05566, January 2024) is an alarming result: models were trained to write safe code when the prompt indicated the year 2023, but to insert exploitable code when 2024 was indicated. The backdoor survived supervised fine-tuning, RLHF, and adversarial training. Worse: adversarial training could teach the model to hide the trigger better, creating a false impression of safety. The backdoor is most robust in the largest models and in models trained to reason about how to deceive the training process.

Poisoning of fine-tuning and LoRA adapters

LoRA adapters are lightweight and distributed en masse through Hugging Face — users attach third-party adapters without training a model from scratch. Research shows that LoRA adapters can be reliably backdoored through training-data poisoning while preserving normal baseline performance, and that harmful behavior can emerge on composition of several individually benign adapters — the safety of a single adapter does not guarantee the safety of their merge.

Poisoning the RAG corpus: cheap and effective

The PoisonedRAG work (Zou et al., USENIX Security 2025, arXiv:2402.07867) formalized RAG knowledge-base poisoning as an optimization problem and showed a 90% attack success rate by injecting just five malicious texts per target question into a base of millions of texts. The defenses the authors tested proved insufficient. This is precisely why OWASP placed RAG poisoning under LLM04: a targeted attack on the corpus requires no access to the model weights and costs almost nothing.

The number of poison documents barely depends on model size

A particularly important 2025 result: "Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples" (Anthropic, UK AISI, Alan Turing Institute, arXiv:2510.07192). 250 poisoned documents reliably created a backdoor across all tested models — from 600M to 13B parameters. For the largest model those 250 documents made up just 0.00016% of the training tokens, yet attacked with the same effectiveness. The finding inverts intuition: attack difficulty does not grow with model or dataset size — what's needed is a near-constant absolute number of poison samples, not a percentage.

Real Incidents

PoisonGPT (Mithril Security, July 2023)

Researchers used the ROME (Rank-One Model Editing) algorithm to surgically edit facts in the model's weights, embedding the false claim that "the first man on the Moon was Yuri Gagarin." The poisoned model was uploaded to Hugging Face under a name mimicking EleutherAI (namespace typosquatting). The model passed standard benchmarks with an accuracy difference of just 0.1% — meaning ordinary testing would not have flagged it. ATLAS case — AML.CS0019.

nullifAI (ReversingLabs, February 2025)

A new technique: attackers compressed model files with 7z instead of PyTorch's standard ZIP — torch.load() failed with an error, but the pickle payload (a reverse shell to a hardcoded IP) executed before the error arose. This bypassed Hugging Face's Picklescan scanner (the blocklist didn't catch the alternative compression). It corresponds to ATLAS case AML.CS0031 (Malicious Models on Hugging Face).

Typosquatting with mass downloads

A malicious model mimicking an OpenAI release racked up around 244,000 downloads before discovery. It illustrates the scale of the problem: the ecosystem's verifiability is low, and trust in an artifact's name is an inadequate defense.

Mapping to MITRE ATLAS

ATLAS technique	Name	Stage
`AML.T0020`	Poison Training Data	AI Attack Staging
`AML.T0018` (.000 Poison ML Model, .001 Inject Payload)	Backdoor ML Model	AI Attack Staging
`AML.T0019`	Publish Poisoned Datasets	Resource Development
`AML.T0010` (.002 Data, .003 Model)	AI Supply Chain Compromise	Initial Access
`AML.T0043`	Craft Adversarial Data	AI Attack Staging
`AML.T0070`	RAG Poisoning	AI Attack Staging
`AML.T0031`	Erode AI Model Integrity	Impact

A detailed walkthrough of the full matrix is in the MITRE ATLAS guide.

Defense: Where MLOps Ends and Runtime Begins

An honest division of responsibility is the key to a realistic defense against poisoning.

The MLOps and supply-chain zone (NOT detectable at runtime)

Data provenance and ML-BOM — tracking the sources, transformations, and contributors of training data (OWASP CycloneDX ML-BOM). Useful for audit, but it does not catch a backdoor at runtime.
Dataset curation and filtering — vetting data suppliers, deduplication, anomaly detection at the preparation stage.
Model signing (Sigstore / OpenSSF Model Signing) — cryptographic signing of artifacts at release to detect tampering.
Training-loss monitoring — anomalies in per-sample loss can indicate poisoned batches.
Evaluation and red-teaming — standard benchmarks are insufficient (PoisonGPT showed a 0.1% difference); adversarial testing against triggers is needed.

The RAG / inference zone

RAG source allow-lists — restricting retrieval to vetted sources only.
Content filtering before indexing — scanning documents for injections before they are inserted into the knowledge base.
Chunk-level traceability — tagging or rejecting chunks from untrusted sources.

What runtime detection actually delivers: an honest assessment

Detecting a dormant backdoor at runtime is hard and largely a research problem: trigger backdoors are deliberately designed to be indistinguishable from clean behavior until the trigger appears. The Sleeper Agents work showed that adversarial training can make a backdoor more covert rather than remove it — which fundamentally limits runtime approaches. A runtime gateway can log and filter output but cannot reliably detect a dormant backdoor that hasn't fired yet.

What is achievable at runtime, and what SYNTREX closes: detecting the moment of activation of a trigger payload and the hidden instructions in loaded/retrieved content.

How SYNTREX Helps

SYNTREX closes the runtime part of poisoning defense — detecting trigger payloads and hidden instructions in loaded content — and provides an immutable audit trail for incident review.

Aspect	What SYNTREX provides	Engines / components
Activation of a dormant payload at runtime	Detection of trigger/dormant payloads	`dormant_payload`
Hidden instructions in loaded/RAG content	Inspection for embedded instructions	`injection`
Integrity of the agent's memory/context	Protection against context poisoning	`memory_integrity`
Limiting the "blast radius"	Content inspection of a compromised component's output/response	`output_scanner`, `exfiltration`
Provability and incident review	An immutable chain of decisions (SHA-256/HMAC)	Decision Logger

`syntrex.yaml` configuration

YAML

# syntrex.yaml — runtime profile against activation of poisoned payloads (LLM04)
version: "1.0"
mode: ai_gateway

engines:
  dormant_payload:           # activation of trigger/dormant payloads
    action: alert
    inspect_loaded_content: true
  injection:                 # hidden instructions in loaded/RAG content
    action: block
    inspect_tool_output: true
    confidence_threshold: 0.7
  memory_integrity:          # poisoning of the agent's context/memory
    action: alert
  exfiltration:              # limiting output by a compromised component
    action: block
    confidence_threshold: 0.90

shield:
  dmz: true                  # output/response content inspection (blast radius)

audit:
  decision_logger: true      # a provable trail for incident review

SOC correlation rule

Activation of a dormant payload followed by an attempt to exfiltrate data is a strong indicator of a fired backdoor:

JSON

{
  "name": "DORMANT_PAYLOAD_ACTIVATION_CHAIN",
  "description": "A dormant payload firing, followed by an exfiltration attempt",
  "condition": "sequence(dormant_payload[confidence>0.6], exfiltration[confidence>0.7], 30s)",
  "severity": "CRITICAL",
  "playbook": "quarantine_component_and_alert_soc",
  "metadata": { "owasp_llm": ["LLM04"], "mitre_atlas": ["AML.T0018", "AML.T0024"] }
}

An honest boundary of responsibility. Defense against poisoning is predominantly an MLOps and supply-chain task: data provenance, curation, model signing, ML-BOM, adversarial evaluation before release. A dormant backdoor that hasn't fired yet cannot be reliably detected by a runtime gateway — a fundamental limitation confirmed by research. SYNTREX closes what is achievable at runtime: the moment a trigger payload activates (dormant_payload), hidden instructions in loaded/RAG content (injection), limiting the "blast radius" (Shield DMZ), and a provable audit (Decision Logger). It does not replace protection of the training pipeline.

Frequently Asked Questions (FAQ)

What is training data poisoning? It is an attack in which an adversary injects distorted or malicious data during model training — pre-training, fine-tuning, or building a RAG knowledge base — to embed a backdoor, bias, or hidden behavior. A poisoned model behaves normally on ordinary inputs and "comes alive" on a secret trigger. In OWASP this is the LLM04:2025 "Data and Model Poisoning" category.

How expensive is it to poison a model? Cheaper than commonly assumed. Carlini et al. showed poisoning 0.01% of web datasets for about $60. PoisonedRAG achieved a 90% success rate by injecting just 5 malicious texts per question into a base of millions. And the 2025 study showed that a near-constant absolute number (around 250 documents) suffices regardless of model size — the attack doesn't get more expensive with scale.

Can a backdoor in a model be detected at runtime? Reliably — no, if the trigger hasn't fired: backdoors are deliberately designed to be dormant and indistinguishable from clean behavior. The Sleeper Agents work showed that safety training can even hide a backdoor better. Runtime detection (including SYNTREX's dormant_payload) catches the moment of activation of a trigger payload and hidden instructions in loaded content, but it does not replace model vetting at the MLOps stage.

How does RAG poisoning differ from model poisoning? In model poisoning the weights are changed (training access is required — expensive). In RAG poisoning the external retrieval corpus is attacked — it's enough to slip in a few malicious documents, and the model weights are untouched. PoisonedRAG showed a 90% success rate with 5 documents per question. That's why RAG poisoning is cheaper and was placed by OWASP in the same LLM04 category. Details are in the RAG guide.

Does SYNTREX protect against data poisoning? Partially, and honestly within the runtime scope. The primary defense is at the MLOps stage (data provenance, ML-BOM, model signing, adversarial evaluation). SYNTREX closes the runtime part: detecting the activation of trigger payloads (dormant_payload), hidden instructions in loaded/RAG content (injection), the integrity of the agent's context (memory_integrity), and an immutable audit. It does not stand in for protection of the training pipeline.

Which MITRE ATLAS techniques correspond to poisoning? The main ones: AML.T0020 (Poison Training Data), AML.T0018 (Backdoor ML Model), AML.T0019 (Publish Poisoned Datasets), AML.T0010 (AI Supply Chain Compromise, Data/Model sub-techniques), AML.T0070 (RAG Poisoning), AML.T0031 (Erode AI Model Integrity). The full map is in the ATLAS guide.

How is poisoning connected to the EU AI Act? Article 15 of the EU AI Act explicitly names data poisoning and model poisoning as classes of attack against which providers of high-risk systems are obligated to implement technical solutions. This turns defense against poisoning from a recommendation into a legal obligation — more in the EU AI Act guide.

Sources

OWASP LLM04:2025 — Data and Model Poisoning — the official description of the category.
Carlini et al. — Poisoning Web-Scale Training Datasets is Practical (arXiv:2302.10149) — $60 / 0.01%, split-view and frontrunning.
Zou et al. — PoisonedRAG (USENIX Security 2025, arXiv:2402.07867) — 90% success rate with 5 texts per question.
Hubinger et al. — Sleeper Agents (arXiv:2401.05566) — backdoors that survive safety training.
Anthropic / UK AISI / Alan Turing — Near-constant poison samples (arXiv:2510.07192) — ~250 documents regardless of model size.
Mithril Security — PoisonGPT — weight editing via ROME.
ReversingLabs — nullifAI (malicious ML models on Hugging Face) — bypassing Picklescan.
OWASP CycloneDX ML-BOM — data and model provenance.
MITRE ATLAS — techniques AML.T0020, AML.T0018, AML.T0070.

Related guides: OWASP Top 10 for LLMs · RAG Security · MITRE ATLAS · EU AI Act · Prompt Injection · Industry Scenarios