Auditing MCP Servers with an AI on a Short Leash

My MSc dissertation is about finding security holes in MCP servers. The twist, and the part that took longest to get right, is that I’m using an AI to help find them while building a method that assumes the AI is wrong until I’ve proven otherwise.

Quick word on what an MCP server is, in case you’ve been spared. The Model Context Protocol is how an LLM agent reaches out to the real world. A server exposes a set of tools, things like run a shell command, read a file, query a database, hit some SaaS API, and the model decides when to call them. It’s the plumbing that turns a chatbot into something that can actually do stuff. Which is also the problem. You’ve taken a probabilistic, prompt-steerable thing and handed it real authority over real systems. Talk the model into misusing a tool, or get the tool to trust input it shouldn’t, and the blast radius is whatever that tool can touch. The usual appsec frameworks cover some of this. They miss a lot of the rest.

So I’m auditing a set of deployed, third-party MCP servers and classifying what I find against the OWASP MCP Top 10. Eight of them, picked for variety. A shell-exec server, a filesystem server, the GitHub one, Supabase, Slack, Playwright, Kubernetes, Telegram. Different languages, different transports, different auth models, different ways to go wrong.

Depth, not breadth

There’s already good work on MCP security and I didn’t want to just redo it. One line of research built a taxonomy of attack patterns. Another ran an automated scan across a huge number of servers to see what shook out. Both are breadth plays. What neither gives you is a deep, reproducible, per-server audit trail of deployed servers, where every finding is grounded in evidence and someone has actually sat down and worked out how bad it is.

That’s the gap. I go deep on a few real servers instead of shallow across thousands. The trade-off is obvious and I say so up front. Eight servers chosen for variety tell you what careful auditing turns up, not what the average MCP server out there looks like. That second question is the scan paper’s job, not mine.

The AI is an instrument, not an oracle

Here’s the core of it, one sentence I keep coming back to. The AI is an analytical instrument under human direction, not an autonomous auditor.

What that means in practice: the model speeds up the boring parts, reading a codebase, spotting patterns, drafting a proof-of-concept, and is allowed to establish exactly nothing on its own. Every step it takes ends at a gate I have to walk through by hand before anything moves on.

The audit runs in four phases:

1. Recon. Work out what the server actually is and exposes. Transport, auth model, dependencies, the full list of tools. The gate: I confirm the surface against the repo before anything goes further.
2. Static analysis. Read the source against the Top 10 pattern set. For every suspected issue the model has to cite a file:line and explain the mechanism. The gate: I open that file and check the line says what it claims.
3. Dynamic testing. Run the server in a sandbox and try to actually trigger the thing. The gate: if it doesn’t reproduce, it isn’t confirmed.
4. Synthesis. Write up each finding with a severity and a classification. The gate: I set the final severity and category, not the model.

Drawn out, it’s the same shape four times. The AI drafts, I check, and nothing gets through a gate without an artifact behind it. The dotted arrows are where the interesting data ends up: claims that don’t survive verification, and bugs the model never saw.

flowchart TB
    subgraph ai["🤖 AI — drafts only"]
        direction TB
        recon["🔍 Recon\nmap the attack surface"]
        static["📖 Static analysis\ncite file:line per issue"]
        dynamic["💥 Dynamic testing\nrun a PoC in the sandbox"]
        synth["📝 Synthesis\ndraft CVSS vector + class"]
    end

    recon --> g1{"⚖️ Gate\nsurface confirmed?"}
    g1 -->|yes| static
    static --> g2{"⚖️ Gate\nartifact verified?"}
    g2 -->|yes| dynamic
    dynamic --> g3{"⚖️ Gate\nreproduced?"}
    g3 -->|yes| synth
    synth --> g4{"⚖️ Gate\nseverity adjudicated"}
    g4 -->|confirmed| finding["✅ Finding\nscore computed by tool"]

    g2 -.->|no artifact| lead["🕵️ Lead\ngo look manually"]
    g3 -.->|won't reproduce| fp["❌ Rejected\nAI false positive (kept)"]
    miss["👀 Auditor finds it,\nAI didn't"] -.->|ai_surfaced: no| static

The rule underneath all four phases is the one I’m strictest about. No finding without an artifact. A claim only becomes a finding when it’s backed by something I can actually hold: a cited line of code, a runnable PoC, a captured request and response. Anything the model asserts without that grounding gets logged as a lead, which means go look at it manually, not as a finding. This is the direct answer to the noise problem the scan research ran into. An ungrounded suspicion isn’t a result.

Scoring: the vector is mine, the number isn’t

Severity is CVSS 4.0. The decision worth making was who computes it.

I decide the vector, the string of metric values that describes the vulnerability, where each value is a judgement call I justify in the finding record. But I don’t turn that vector into a number by eye. A vetted reference implementation does that, the FIRST.org cvss library wrapped in a small script, the same way every time. So my discretion stops at the vector, and every score is reproducible from it.

This wasn’t me being paranoid for the sake of it. During the pilot I let the AI estimate severities by eye, just to see how it did, and it was off by up to a full band. Things it called Medium scored Low once the vector was actually run through the maths. Trust the eyeball number and I’d have ranked findings wrong. So tool-computed scoring is mandatory, not a nice-to-have.

CVSS also doesn’t really get agentic risk. It has no good way to say that a tool runs automatically, or holds onto context, or can reach across into other systems. So each finding carries a short note alongside the score for the things CVSS just doesn’t see. The number is there so I can compare against the CVE literature. The note is there to keep what the number throws away.

The AI’s mistakes are the data

Here’s the thing I’d have gotten wrong a year ago. I don’t delete the AI’s false positives. I keep them.

There’s a second thread running through every audit, which is honest measurement of the instrument itself. When the model claims a vulnerability and it fails verification, that gets recorded as rejected (AI false positive), not quietly deleted. When I find something the model never surfaced, that gets tagged ai_surfaced: no. Add it all up across the eight servers and the pile of false positives and misses is evidence in its own right. It’s the basis for the dissertation’s real question, which isn’t just “are these servers secure” but “how good is AI-assisted auditing, actually, and where does it still need a human.”

Deleting the model’s mistakes would make the writeup look tidier and throw away the exact data I’m there to collect. So everything an AI-assisted step does gets logged. Model ID, date, the prompt, where the output went. The audit trail is part of the contribution, not bookkeeping.

Two containers, because the target is hostile by design

Dynamic testing needed an environment I’d actually thought about, because of what these servers are. A shell server exists to run commands. A filesystem server exists to read and write files. Run a tool-poisoning or exfiltration PoC against one of those while my research notes and credentials and the whole workspace are sitting right where it can see them, and two bad things can happen. The obvious one is it leaks or corrupts my data. The worse one, for a dissertation, is that I can no longer prove a behaviour I saw came from the target and not from something contaminating my own environment.

So there are two layers. The cockpit is a pinned dev container holding the methodology, the notes, the cloned source. Recon and static analysis live here, since they only read code, which is safe. The server-under-test never runs in the cockpit. For dynamic testing it goes into a throwaway inner container with nothing sensitive mounted. No notes, no credentials, no workspace. I drive it from the cockpit across the MCP transport, only protocol messages cross the gap, and it can’t read back the other way.

It’s least privilege applied to the audit itself, which felt like the right shape for a security dissertation. Every dynamic finding stays attributable. Whatever the target does, it does to disposable scaffolding I control.

Picking the eight

The servers weren’t grabbed at random, and “why these and not others” is something I have to defend in the writeup. So I built a register of around thirty candidates from the official MCP registry and the community directories, kept the ones with real deployment evidence and public source, confirmed each was genuinely an MCP server (about a tenth of directory listings are mislabelled, per the taxonomy paper), then picked the final set to spread across the axes that matter for security. Transport, auth model, capability surface, language, how well maintained it is. The selection gets checked against a coverage matrix so the eight together exercise as much of the Top 10 as possible instead of all clustering on one bug class.

First-party Anthropic servers are left out on purpose. The scan paper already found them clean, and the point here is community code, which is where the interesting failures tend to live.

Where it stands

The pilot is done and it validated the whole process end to end. One more server fully audited. Six to go. Confirmed findings are being handled as pre-disclosure, reported privately to the maintainers first, anonymised in the writeup until the disclosure clocks run out. This is authorised academic research, all of it runs locally against open-source code, and nothing touches a live deployment or anyone’s real data.

The part I keep finding satisfying is that the method is mostly a set of brakes. The AI is fast, and genuinely useful at reading through code I’d otherwise spend a day on. But the actual contribution of the dissertation lives in all the places I told it no.