Skip to content
BENIBIX Learn

claude plugin validate vs claude plugin eval

People think a plugin that passes claude plugin validate is a plugin that works. Validate only checks the shape of the files: the manifest, and the skills, agents and commands in the folder. claude plugin eval runs prompts through the plugin many times and scores the answers, with it and without it. So it shows if the plugin helps. A plugin can pass validate and still make Claude worse. Run validate on each change, and eval when what it does changes.

AI⌨️ Claude Code

claude plugin validate Command

It checks a plugin you are making for mistakes.

claude plugin validate checks a plugin you are making. It reads the plugin's manifest, a marketplace's manifest, or the skills, agents and commands in a folder. Then it lists what is wrong. It ends with code 0 when all is well and 1 when it finds a problem. So a CI job can stop a broken plugin from being shared.

Example
Jack's plugin repo runs claude plugin validate on each pull request. One day plugin.json is missing a field. The check fails before anyone installs it.
Why it matters
A plugin with a broken manifest fails for everyone who installs it. This catches it first.
Full entry
AI⌨️ Claude Code

claude plugin eval Command

It tests how much a plugin helps, with scored cases.

claude plugin eval tests whether a plugin does the job it was made for. Each test case is a prompt plus graders that score the answer. Claude Code runs each case several times in a clean session with only that plugin loaded. By default it also runs the case without the plugin, so the report shows how much the plugin helps.

Example
Lucy changes a skill in her plugin. claude plugin eval shows the new version passes more cases than the old one, and both beat running with no plugin.
Why it matters
A change to a skill can make it worse in ways you do not notice by hand. Scored runs show it.
Full entry

In one line

claude plugin validate Command
It checks a plugin you are making for mistakes.
claude plugin eval Command
It tests how much a plugin helps, with scored cases.