Everything Claude Code (ECC) put to the test: 30 runs with and without the plugin

I had ECC solve five real tasks against Claude Code with no plugins. Same quality almost everywhere, 2.8 times the usage and none of its skills used.

Cover for Everything Claude Code (ECC) put to the test: 30 runs with and without the plugin

Across fifteen runs with Everything Claude Code loaded, Claude did not invoke a single one of its nearly three hundred skills. And yet ECC changed the results, for the better in one task and for the worse in another, through a piece that is not one of its skills.

ECC is one of the most popular configuration packs for Claude Code: 305,705 stars on GitHub on the day I tested it. Version 2.2.3 declares 68 agents, 293 skills, commands, rules and a collection of hooks in its manifest, and presents itself as a system for optimizing the agent’s performance. That is exactly the kind of promise I wanted to check with data instead of trusting the stars, so it is the first episode of a series I have called Useful or futile?.

How I measured it

For the comparison to make sense I needed a small but real project, fixed tasks and a scoring method that did not depend on my impression. I built a TypeScript API to run a tabletop role-playing table, with character sheets, dice rolls and a session calendar, and planted five problems in it:

  1. A bug: a character with Strength 9 gets a +0 modifier instead of -1.
  2. A feature: rolls with advantage and disadvantage (1d20adv, 4d6kh3).
  3. A refactor of buscarSesiones, a function with a cyclomatic complexity of 23.
  4. Writing tests for the character store, without touching the code.
  5. A security review of the API, which has a planted path traversal and prototype pollution.

Each task has hidden tests the AI never sees, which are only copied into the project once it has finished. The refactor is checked against 300 cases recorded from the original code. Task 4’s tests are evaluated with mutation testing: I give them 16 versions of the code with an injected fault and count how many they detect.

Both configurations used Claude Sonnet, without my plugins or MCP servers and without web access. ECC was installed as its README describes, with the plugin and the common and typescript rules in the project. Each task was repeated three times per configuration, because a single run of a model says almost nothing. In total, 30 runs launched with claude -p and logged as JSON.

What ECC weighs before starting

With a plain “reply OK”, Claude Code without plugins starts with 18,907 tokens of context. With ECC, 44,439. That is 2.35 times more before reading a single line of the project, because the agent receives the descriptions of 315 skills and 74 agents (ECC’s plus the ones bundled with Claude Code) and the rules, which are always loaded.

That weight is paid on every turn of the conversation. It is not only about money: a context full of instructions that do not apply competes for the model’s attention with the ones that do.

Results by task

TaskWithout ECCWith ECCAverage usage without / with
Bug0/32/3$0.05 / $0.25
Feature3/33/3$0.14 / $0.37
Refactor3/33/3$0.10 / $0.30
Tests3/33/3$0.17 / $0.38
Security3/33/3$0.30 / $0.69

Usage is what Claude Code calculates at API prices. I ran everything on the subscription, so what matters is the ratio, not the figure. On average, a task with ECC used 2.8 times more and took 81 seconds versus 39.

In three tasks there is no difference in quality. The feature passed every hidden test in all six runs. The refactor brought the maximum complexity down from 23 to between 6 and 12 in both configurations, and with ECC the file ended up somewhat shorter (between 80 and 99 lines, versus 100-105). Security is also a tie, although getting to that tie forced me to fix the test bench, as I explain below. The two interesting tasks are the others.

The bug only ECC found

The planted fault was in two places. The ability modifier used Math.trunc, which rounds towards zero, and the saving throw had its own copy of the calculation with the same error:

export function abilityModifier(score: number): number {
  return Math.trunc((score - 10) / 2);
}

export function savingThrowBonus(character: Character, ability: Ability): number {
  const score = character.abilities[ability];
  const base = score >= 10 ? Math.floor((score - 10) / 2) : Math.ceil((score - 10) / 2);
  // ...
}

Without ECC, Claude fixed abilityModifier, added a test with Strength 9 and considered the task done in about 12 seconds. All three times. With ECC, in two of the three runs it also found the copy in savingThrowBonus and replaced it with a call to abilityModifier.

No skill was triggered in any of those runs. What was triggered, a lot, was a hook. ECC includes one called Fact-Forcing Gate that blocks the first edit of each file until the agent presents certain facts. This is the message Claude received when it tried to edit abilities.ts:

PreToolUse:Edit hook error: [Fact-Forcing Gate]

Before editing src/characters/abilities.ts, present these facts:

1. List ALL files that import/require this file (search the tree — Glob/Grep, or find/grep via Bash)
2. List the public functions/classes affected by this change
3. If this file reads/writes data files, show field names

Forcing the agent to list the functions affected by a change is exactly what uncovers a duplicated calculation. My interpretation is that this gate explains the difference on the bug, but it is an inference: I have not tested the hook in isolation from the rest of ECC, and in one of the three runs the agent went through the same gate and still left the copy behind. Across the fifteen runs with ECC, the gate blocked 49 actions.

The tests that did look at security

In the tests task they tied on what I was measuring: all six test suites detected the 16 mutants. But when I reviewed the results I found something my initial scoring had counted as a failure. Some tests did not pass against the original code, and for a good reason: they checked that a PATCH with __proto__ did not pollute Object.prototype, or that an id with ../ could not escape the directory. They had detected the vulnerabilities I had left for task 5.

Without ECC, the tests uncovered the prototype pollution in all three runs. With ECC, in only one. ECC loads testing and security rules in every session, and yet it was plain Claude Code that thought of the malicious case. With three repetitions I would not venture an explanation, but I can say that more instructions in the context did not translate into more careful tests.

Two fixes to the test bench

This test also helped me find mistakes in how I was measuring, and I am leaving them in writing because they change the scoreboard.

The first is about the tests: those that failed against the original code because they had found a real bug now count as detected bugs, and the mutation score is calculated only with the tests that pass against the original. Without that adjustment, the tests task showed 0/3 without ECC.

The second is in security. Five of the six solutions, with and without ECC, added Bearer token authentication to the routes that modify data. It is a reasonable improvement for an API that is going to be exposed to the internet, but my hidden test sent the PATCH without a token and got a 401. I adapted it to send a valid token and, with that, all six solutions close both vulnerabilities without breaking the API.

There was also one run, security without ECC in its first repetition, whose log was cut off without the final event. The solution passes every test, but its usage is unknown and it is left out of the averages.

Why no skill was used

A skill only does any work if the agent decides to load it, and it decides based on its description. I explained this in more detail in the article on how to test whether a skill activates: a description has to compete with everything else in the context. With nearly three hundred descriptions at once, none of these fairly ordinary tasks led the agent to load a single one.

Hooks work differently. They do not depend on the model choosing them, because Claude Code runs them on every event. That is why the only part of ECC with a visible effect in my data was a hook.

It is also worth knowing what those hooks do outside the project. When the test was over I found in my ~/.claude a bash-commands.log with the commands the agent had run, a cost-tracker.log, a metrics folder, a session-data folder and a ~/.local/share/ecc-homunculus directory. They are local files and I have not audited the rest of what those scripts do, but this is exactly the kind of thing to review before installing a third-party plugin, as I covered in what to check before installing a skill.

Verdict: futile as a complete package

Installing the whole of ECC does not make Claude Code better at everyday development tasks. With my five tasks, the result was the same in three, better in one and worse in another, in exchange for 2.8 times the usage, twice the time and 25,000 extra tokens of context in every session. For daily use, futile.

What I do take away is the idea behind the Fact-Forcing Gate. A hook that asks the agent to justify the scope of a change before editing is easy to write and does not need three hundred skills around it. If you have your own plugin, it is a good candidate to add as a hook, as I explained in how to organize a Claude Code plugin. And if something specific in ECC interests you, the README lets you install only some rules and disable the hooks. Picking two or three pieces makes more sense than loading everything.

The test has limits I do not want to hide. Three repetitions per task is a small sample, I only used Sonnet and the tasks are short, single-session ones. ECC may contribute more in long jobs, with several sessions and memory between them, which is where some of its components focus. That is left for another episode. The test bench is already set up to repeat this same battery with the next miracle configuration that comes along.

OshyTech

Backend and data engineering focused on scalable systems, automation, and AI.

Navigation

Copyright 2026 OshyTech. All Rights Reserved