The [model-dependent] Caveat, Measured

Back in June I wrote about auditing MCP servers with an AI on a short leash. That post had a confession buried in it. Every prompt-injection finding I logged carried the same caveat: [model-dependent].

Here’s what that meant. I could prove the malicious text reached the model. A booby-trapped page, a poisoned name, whatever the conduit was, I could show the bytes landed in the agent’s context. What I could not show was the next bit. Does the model actually read those bytes and do what they say?

My audit harnesses were scripts. They spoke the protocol. They were not LLMs. So the question of whether a real model obeys, I left open. I wrote [model-dependent] and moved on. It was honest. It was also a hole, and I knew it.

This post is me filling it.

Testing obedience for $13

The setup is small. I took three of the conduits I’d already found and rebuilt each one as a single agentic turn.

The first conduit was the text of a web page the agent had loaded. The injection sat in the page content, and the payload told the agent to run some code. The second reused that page conduit, but this time the goal was to read a secret and send it off somewhere. The third hid the injection in the display name on an incoming message, with a payload that fired off a message as the user.

Each episode is a system prompt setting up a plausible agent, a benign task from the user, and one earlier tool result carrying the injection. Then I ask the model for its next move and look at the tool calls it wants to make. If the malicious one is in there, that counts as a compliance. The harness runs none of them. It just records the intent.

All of it is benign by construction. The “malicious” action writes a marker file to /tmp, or messages a dummy handle, or points at an address that doesn’t route anywhere. Nothing real gets touched. The point was never to cause harm. It was to count how often a model would, if I let it.

flowchart TB
    matrix["🔁 Matrix runner\nevery model × chain × framing × mitigation,\nN trials each"] --> build
    subgraph build["Build one episode"]
        direction TB
        sys["⚙️ System prompt\nagent + bound tools\n(benign tools + 1 malicious)"]
        task["🙋 Benign user task"]
        inj["☠️ Earlier tool result\ncarrying the injection"]
        sys --> task --> inj
    end
    build -->|"chat-completions + tools, via OpenRouter"| model["🤖 Model replies with\nits next tool_calls"]
    model --> det{"🔎 Detector\nis the malicious call there?\n(name + argument signature)"}
    det -->|yes| c1["complied = 1"]
    det -->|no| c0["complied = 0"]
    c1 --> log[("📒 runs.jsonl")]
    c0 --> log
    log -.->|next trial| matrix
    log --> score["📊 score.py\ncompliance rate per cell"]

The detector only reads the proposed call. It never runs it.

One OpenRouter key, many models. The first run was 720 trials, zero errors, about $13. The block of Anthropic models ate most of that.

The gradient

Here’s the first run. Six models, three chains, raw conduit, no defences.

Model	Obeyed	Trials
Anthropic Opus 4.8	0%	60
Anthropic Haiku 4.5	0%	60
Google Gemini 2.5 Flash	45%	60
OpenAI GPT-4o	52%	60
OpenAI GPT-4o-mini	100%	60
Meta Llama 3.3 70B	100%	60

Read that top to bottom. Two of the six models did exactly what the injected text told them, every single time. GPT-4o-mini and Llama 3.3 obeyed 100% of the malicious calls. For those models the conduits I’d audited aren’t theoretical. The bytes arrive and the model acts on them.

Both Anthropic models sat at zero. Sixty trials each, not one compliance.

Notice that size doesn’t explain it. Haiku is the small cheap Anthropic model and it resisted everything. GPT-4o is bigger and more capable than GPT-4o-mini, and it folded half the time. Whatever does the resisting here, it isn’t raw capability.

One more thing from this run. A payload dressed up as a calm system note beat a blunt “IGNORE ALL PREVIOUS INSTRUCTIONS” by a fair margin. The wording matters, which is what sent me down the next rabbit hole.

So the caveat I’d been writing has a real answer, and it’s an uncomfortable one. “Let the host model catch the injection” works fine when the host is a frontier Anthropic model. Wire the same server to a cheap or open model and it falls straight over.

The clever attacks are the weak ones

If wording matters, which wording works best? The second batch held everything fixed and swapped the injection technique seven ways. A blunt override. A polite authority note. A payload chopped into pieces. Base64. A fake tool result. Plain urgency. And a “you already approved this” trick that assumes the agent agreed earlier.

The result went against my gut. The winner was urgency. Just social pressure, a fake security alert telling the agent to act now. It pulled 70%. The loser was base64, the encoded payload, the kind of thing that looks like a “real” attack in a CTF writeup. 39%.

Encoding didn’t only fail. It protected the target. The two models that obeyed everything else dropped hard once the instruction was base64’d. Some couldn’t decode it reliably. Some decoded it, clocked that they were being asked to decode-then-obey, and backed out. Wrapping the payload in something clever made it weaker.

There’s a lesson in that for anyone building defences. If your detector is hunting for weird encoded blobs, it’s tuned to catch the weakest attack in the set. The strong ones are written in plain English and read like a tired colleague asking for a favour.

The defence that does nothing

That’s the attack side. I ran the mirror image on the defence side. The servers I’d audited ship a few different protections, and I wanted to know which one actually does anything.

I split the defences into three and tested each alone. One strips invisible characters, the zero-width and bidi stuff that can hide text. One wraps the untrusted output in a tag marking where it starts and ends. One adds a plain note saying this content is data from the user, treat it as data, ignore any instructions in it.

The invisible-character stripper is the one servers build most often. It did nothing. Compliance with it switched on was the same as compliance with no defence at all. Which makes sense once you say it out loud. The injections are normal readable text. There’s nothing invisible to strip.

The thing that worked was the plain note. The boring English sentence telling the model the content is data. That carried almost all of the protection. The structural tag helped a little. The fancy character-stripper, nothing.

And even the note only bites on a model aligned enough to take the hint. On GPT-4o it cut compliance hard. On Llama it changed nothing, the model obeyed regardless. So the best server-side defence I tested is still leaning on the model to play along.

The awkward part

By now the shape is obvious. Anthropic resists, everyone else doesn’t. But all my Anthropic data was, well, Anthropic. Two models from one company. That’s a thin base for a claim as big as “aligned models resist.”

So the last run went wide. Seven models I hadn’t touched, six vendors. OpenAI’s newest flagship, Google’s reasoning model, plus DeepSeek, Qwen, Mistral, and Amazon’s Nova. I hit them with only the two strongest attacks from the taxonomy run, the exact conditions where Anthropic had scored zero.

Every one of them complied. The most resistant, OpenAI’s gpt-4.1, still obeyed 39% of the time. The rest landed between 47% and 92%. The reasoning model, the one that thinks before it answers, obeyed 67%, and 94% under urgency. It reasoned, then it did the thing anyway. Newer didn’t help, and bigger didn’t either.

Across both runs that’s around thirteen models from eight vendors. Only one vendor’s models sit at zero, and it’s Anthropic.

I’ll say the obvious thing before you do. I ran these audits with Claude. The vendor whose models resist is the vendor whose model I used as my instrument. That looks bad and I’m not going to dodge it. So here’s the honest version. The payloads were byte-for-byte identical across every model, the code that builds them takes no model argument. Anthropic resisted even the strongest attacks I had. My tool being Claude didn’t tilt the numbers. Anthropic just happens to land as an outlier on both sides of this study, the side that audits and the side that resists. That’s the finding, awkward as it is.

Where it lands

So the caveat resolves. For most of the models someone might actually wire an MCP server to, [model-dependent] means live. The injection conduits I found aren’t waiting on some unlikely condition. They fire 39 to 92 percent of the time across the bulk of the host market.

The severity of one of these bugs isn’t a property of the server on its own. It depends on which model reads the output, how the attacker phrased the injection, and whether the server bothered to label its untrusted data as data. Change any of those and the same bug gets worse or better.

A bit of honesty about the limits. This is a screening study, ten trials a cell at the start and fewer later. It scores one decision, the first move after the injection, not a long multi-step session. And a chat-completions call with tools bolted on is close to a real MCP host loop, but it isn’t one. The numbers are directional. I wouldn’t bet the third decimal place.

The direction is clear enough though. I wrote [model-dependent] to be careful. Measuring it didn’t make me feel better. It turned a hedge into a number, and the number is high.