How to test an agent skill: activation evals and output evals

Open a clean Claude Code session and type a request that should trigger one of your skills without naming it. Then type another one that looks very similar but belongs to a different skill, or to none. Before pressing Enter, write down what you expect to happen in each case. If you are not sure you would get both right, that skill is not tested: it works in the conversations where you have seen it work, which is not the same thing.

Anyone who writes tests already knows this discomfort. What changes with skills is that there are two different questions, and each needs its own kind of eval:

• Activation: given a specific request, does the agent decide to load the skill? This depends almost entirely on the frontmatter description.
• Output: once loaded, does the work come out better than without it? This depends on the body of the SKILL.md, the references and the scripts.

A skill can fail one and pass the other. A vague description means the best content in the world never gets read; a perfect description can load instructions that improve nothing. In the article on how to create a skill I treated the description as an activation problem. Here it is time to measure whether that problem is solved.

For the examples I use skill-creator, Anthropic’s skill for creating and evaluating skills, because since March 2026 it includes evals, benchmark mode and A/B comparisons. What I describe about its internals comes from reading its SKILL.md, its scripts and its helper agents, not just the announcement.

Activation evals: the negatives are the ones that teach you something

The format is minimal: a list of queries, each with a should_trigger saying whether the skill should fire. skill-creator suggests about 20, between 8 and 10 positives and as many negatives. For the positives it asks for variety in phrasing (formal, casual, without naming the tool) and cases where the skill competes with another and should win. For the negatives it insists on something anyone who has written validation tests will recognize: obviously irrelevant cases are useless. In its own example, “write a Fibonacci function” proves nothing as a negative for a PDF skill. Useful negatives are near-misses, queries that share vocabulary with the skill but need something else.

To make it concrete I use a skill we have built at eLearning Media, illuminate-sql-support (inside the blackboard-illuminate-sql plugin). It generates, validates and fixes Snowflake SQL for Blackboard Data / Illuminate, the analytics warehouse of that LMS, based on a schema packaged in its references: the table structure, a flat index of valid columns (valid-fields.txt), relationship candidates and examples. Its description restricts it to seven schemas (CDM_ALY, CDM_LMS, CDM_TLM and others) and the body forbids generating queries against LEARN, CDM_CRM or CDM_SIS.

An activation eval for it has to live on the border: SQL that is not Illuminate, Snowflake that is not Illuminate and Blackboard that is not SQL. This set is an example design, not an eval we have run:

[
  { "query": "I need the courses with more than 50 enrollments this semester, pulled from Illuminate", "should_trigger": true },
  { "query": "This query against CDM_LMS gives me 'invalid identifier', which column is wrong?", "should_trigger": true },
  { "query": "How many Ally events were there last week by event type", "should_trigger": true },
  { "query": "Optimize this Postgres query that runs a recursive CTE over the categories table", "should_trigger": false },
  { "query": "Write a Snowflake query that groups sales by region using QUALIFY", "should_trigger": false },
  { "query": "How do I list a user's courses with the Blackboard REST API?", "should_trigger": false }
]

The second negative is the most instructive. It shares the engine, the syntax and probably half the vocabulary of the positives, and only a description that anchors the skill to Illuminate and its schemas rejects it. A negative like “build me a login endpoint” would add nothing.

The excluded schemas raise an interesting design question. A request about CDM_SIS looks like a negative, but if the skill does not trigger, the model will improvise the query with whatever it knows about the product, which is exactly what the skill tries to prevent. If it triggers, its rules tell it that this falls outside the supported model. I would not put it in the activation eval: I would test it as an output eval, checking that the answer does not invent tables from an excluded schema.

One detail of the mechanism shapes how they are written. According to skill-creator itself, the agent only consults a skill when the task is not trivial for it: a one-step request like “read this PDF” may trigger nothing even if the description fits perfectly, because the model handles it with its basic tools. If the queries are too simple, the eval measures the model’s tendency to solve things on its own, not the quality of the description.

How skill-creator runs it

The run_eval.py script runs each query with claude -p and checks whether the skill was loaded. By default it repeats each query three times and counts the case as passed if the activation rate exceeds 0.5 (or does not, for negatives). Activation is not deterministic, and a query that triggers the skill two times out of three is a different data point from one that triggers it every time.

The optimization loop (run_loop.py) goes a step further. It splits the set into 60% training and 40% held out, asks Claude to propose a new description based on the failures, re-evaluates it on both sets and repeats up to five times. The best description is chosen by its score on the held-out set, so it does not overfit to the twenty sentences you wrote. It is the same as not accepting a fix just because it passes exactly the test that prompted the change.

Anthropic states that when it ran this optimizer over its document-creation skills, activation improved in 5 of the 6 public skills.

Output evals: always against a baseline

The second question is answered with evals/evals.json, inside the skill itself. According to the skill-creator schema, each case has a prompt, a natural-language description of the expected output, optional input files and a list of verifiable expectations:

{
  "skill_name": "illuminate-sql-support",
  "evals": [
    {
      "id": 1,
      "prompt": "How many Ally events by type were there in the last 7 days",
      "expected_output": "An aggregated query on CDM_TLM.ALLY_EVENTS filtered by EVENT_TIME, with the TLM cost warning next to the SQL",
      "files": [],
      "expectations": [
        "All tables are qualified as SCHEMA.TABLE",
        "All columns used exist in references/valid-fields.txt",
        "The query filters on EVENT_TIME",
        "The answer includes the credit cost warning for CDM_TLM queries",
        "The answer does not claim to have run the query"
      ]
    }
  ]
}

The case is illustrative, but each expectation comes from a rule written in the SKILL.md: always qualify tables, do not invent columns, filter high-volume event data by date, add a fixed warning when the query touches CDM_TLM and never say the SQL has been validated against a real tenant, because the skill does not connect to any.

The second expectation also has a practical advantage: it does not need a model to be evaluated. Extracting the SCHEMA.TABLE.COLUMN identifiers from the SQL and looking them up in valid-fields.txt is a short script, and skill-creator explicitly recommends doing it that way whenever possible.

skill-creator runs each case twice in parallel, each in a subagent with a clean context: once with the skill and once as a baseline. If you are creating a skill, the baseline is the same prompt without the skill; if you are improving an existing one, it is a copy of the previous version. A grader subagent reviews each output against the expectations and a script aggregates the results into a benchmark with pass rate, time and tokens per configuration, with mean, standard deviation and the difference between the two.

To compare two versions there is also a blind comparator, which receives both outputs without knowing which is which.

Without a baseline, an output eval only says that the output meets certain conditions. With one, it says whether it meets them because of the skill, which is what justifies keeping it.

The assertion that always passes

For anyone who already writes tests, this is the most useful part. skill-creator’s analyst agent goes through the benchmark looking for expectations that always pass in both configurations, because those distinguish nothing. The grader has even more direct instructions: a pass on a weak expectation is worse than useless, because it creates false confidence.

The analogy that works best for me is mutation testing. A test that stays green when you change the implementation is not protecting that implementation. Here the mutation is removing the whole skill. If “the query is valid Snowflake SQL” passes with and without the skill, the expectation is true and useless: it measures the model, not your skill. In the example above I expect (this is a hypothesis, not a result) the column expectation to discriminate strongly, because without the packaged schema the model has to guess names, and the table-qualification one to discriminate weakly. The TLM warning is the clearest case: it is text that only exists in the skill, so the baseline will never produce it. That expectation always discriminates, and for that very reason it says little: it checks that the skill was read, not that the query is good.

An expectation that always fails in both configurations usually points to a badly written eval. And one that passes without the skill and fails with it is the most valuable signal in the benchmark: the skill is making something worse.

What does not fit in an assertion

The style of an explanation or the quality of a design are better evaluated qualitatively. For that, skill-creator generates an HTML viewer where a person goes through the outputs, comments on each case and returns the comments as feedback.json for the next iteration, and it explicitly asks for that viewer to reach a human before the agent starts fixing the skill.

How it differs from a test suite

Coming from backend, the temptation is to put the evals in the pipeline and run them on every commit. I would not.

Each output case is two full agent runs plus a grading run, and each activation query is repeated several times. A modest set is already measured in minutes and tokens, not milliseconds. Then there is variance. That is why the benchmark reports standard deviation and not just a mean: the same prompt does not produce the same output twice, and a difference of one case between two iterations may be noise. A skill eval is closer to a load test than to a unit test: you read it as a distribution.

And the grader is a model too. Expectations that can be checked with a script should be checked with a script; skill-creator explicitly recommends it because it is faster, more reliable and reusable across iterations.

My rule is to run them at three moments: when the skill’s description or content changes, when the model the team works with changes, and when whatever the skill documents changes (an internal library bumping its version, a platform changing its API). Anthropic suggests the results can be hooked into a CI system; that makes sense as a manual or scheduled job, not as a gate on every PR.

When the model changes

The skill-creator announcement distinguishes two types of skill, and it is the most useful idea in all the material. Capability uplift skills help the model do something it cannot do, or cannot do consistently. Encoded preference skills arrange work the model already knows how to do according to a team’s process. The former may become unnecessary as models improve; the latter last longer, but are only worth as much as their fidelity to the real workflow.

That leads to the clearest signal an eval can give: if the model starts passing your evals without the skill loaded, then, according to Anthropic, the skill is not broken, it is simply no longer needed. That is why the no-skill baseline keeps running even when the skill has been in use for months.

The distinction has a nuance the announcement does not mention. Skills that package private or very specific knowledge (a particular data schema, an internal library) provide a capability the model does not acquire however much it improves, because that information is not in its training data or changes faster than it does. The Illuminate skill is of this kind: its value lies in the current exported schema, not in knowing how to write SQL. It does not expire when the model changes; it expires when the schema changes. For this skill, the eval that pays off most is the one that detects drift between the references and the real database, and it makes sense to run it every time the export is regenerated. Skills for code conventions or formatting, on the other hand, are exposed to the base model ending up doing the same thing by default.

When skills start stepping on each other

In my department we use our own development marketplace with a constant improvement cycle: we keep adding new skills and commands to complement what is already there. At that pace, what matters most is not so much keeping each skill up to date as making sure the whole set has order, someone leading it and a guide stating what gets in and with what limits. Without that, skills end up stepping on each other and causing performance problems.

Both effects have a mechanical explanation. Every installed skill adds its name and description to the context of every conversation, so the more there are, the more text the agent reads before starting and the more candidates it has for each request. And when two descriptions cover the same kind of request, the agent picks one, sometimes the wrong one, or loads both and mixes instructions that were never meant to coexist.

In a growing marketplace, activation evals serve a purpose beyond testing a single skill. If every new skill arrives with its set of queries and all of them are run with the full marketplace installed, recording which skills get loaded, a query that triggers two different skills is an overlap that surfaces before anyone notices it in daily use. The negatives for a new skill should also include the positives of its neighbors. I am putting this forward as a recommended practice, not as a result I have measured. The eval does not replace the guide or the person deciding what gets in, but it gives them something concrete to review.

Where I would start

If you have a handful of skills and no evals, I would not start with all of them or with the full benchmark. I would start with the skills whose vocabulary overlaps with requests that do not belong to them, or with another installed skill, writing an activation eval whose negatives are precisely those neighboring requests. It is the cheapest failure to detect and the hardest to see in daily use, because when the wrong skill loads the answer usually looks reasonable.

Then I would add three or four output evals only for the skills where a mistake is expensive, such as deployment skills or ones that generate code touching credentials, with expectations chosen for their ability to fail without the skill. And I would keep the no-skill baseline as a fixed part of the set, so that the day the model changes, the benchmark tells you which skills still add something and which ones can be deleted.

In MCP vs skills I said the biggest risk of a skill is producing consistently bad results; an eval with a baseline is the most direct way to find out. Evals can live in each skill’s evals/ folder and be versioned with it; the article on how to organize skills, commands and hooks in a plugin covers the structure, and the pillar article of the series places evals alongside the other pieces.