claude plugin validate vs claude plugin eval
People think a plugin that passes claude plugin validate is a plugin that works. Validate only checks the shape of the files: the manifest, and the skills, agents and commands in the folder. claude plugin eval runs prompts through the plugin many times and scores the answers, with it and without it. So it shows if the plugin helps. A plugin can pass validate and still make Claude worse. Run validate on each change, and eval when what it does changes.
claude plugin validate Command
It checks a plugin you are making for mistakes.
claude plugin validate checks a plugin you are making. It reads the plugin's manifest, a marketplace's manifest, or the skills, agents and commands in a folder. Then it lists what is wrong. It ends with code 0 when all is well and 1 when it finds a problem. So a CI job can stop a broken plugin from being shared.
claude plugin eval Command
It tests how much a plugin helps, with scored cases.
claude plugin eval tests whether a plugin does the job it was made for. Each test case is a prompt plus graders that score the answer. Claude Code runs each case several times in a clean session with only that plugin loaded. By default it also runs the case without the plugin, so the report shows how much the plugin helps.
In one line
- claude plugin validate Command
- It checks a plugin you are making for mistakes.
- claude plugin eval Command
- It tests how much a plugin helps, with scored cases.