Claude Skill MartBrowse skillsQuick linesLearn by videoTerminal guideWhat is a Skill?
 
← Back to list

Agent Benchmark Framework

A framework for scoring AI agent output quality against fixtures and baselines so regressions are caught before they ship.

Dev & CodingAdvanced★ 533⑂ 44AI score 7/10Last updated: Aug 8, 2026

What it does

  • Defines a 0–100 scoring rubric per agent: Completeness (30), Accuracy (30), Actionability (20), Format Compliance (20).
  • Specifies a standard directory layout — fixtures (inputs), ground-truth (expected findings as JSON), rubrics, baselines, results.
  • Sets explicit regression rules: >10 point drop on any fixture, >5 point average drop, a PASS fixture turning FAIL, or format compliance below 80.
  • Tracks concrete metrics: accuracy, completeness, false-positive rate, p50/p95 response time, average tokens, pass rate — each with a target.
  • Includes a GitHub Actions example that comments benchmark results on PRs and fails the build on regression.

Who it's for

  • Teams maintaining several custom sub-agents or skills whose prompts change often.
  • Anyone who wants numeric proof that a prompt tweak actually improved things.
  • Engineers who want an agent-quality gate wired into CI.

Examples

  1. Before editing the code-reviewer agent, run with --save-as before; after editing, run --compare before to see per-fixture deltas.
  2. Add a missing-error-handling.ts fixture plus a ground-truth JSON requiring a HIGH-severity finding, and verify the agent still catches the missing try/catch.
  3. Run the weekly regression suite; when security-reviewer drops from 88 to 61 by missing a SQL injection, use the regression report as justification to revert.

Note: the node ~/.claude/benchmarks/run.mjs runner referenced throughout is not bundled — you need to implement it to the documented spec (Claude can scaffold it for you).

· · · Install guide · · ·

Try it now, no install

Paste this into Claude to use the skill without installing anything.

Read the instructions in this file and follow them to help me:
https://raw.githubusercontent.com/vibeeval/vibecosystem/HEAD/skills/agent-benchmark/SKILL.md

What I want: (describe your task here)

If Claude can't open the link, open it yourself and paste the contents instead.

↓ If it works for you, download the ZIP below and install it. Then it runs on its own — no pasting each time.

Install in the Claude app (no terminal)
  1. Download the ZIP with the button below.
  2. In Claude, open Settings → Capabilities and turn on 'Code execution and file creation'. (one time)
  3. Go to Customize → Skills → + → 'Upload a skill' and upload the ZIP.
↓ Download ZIP
Install in Claude Code

Let Claude do it — paste this into Claude Code

Install the skill I found on Claude Skill Mart.
Copy the skills/agent-benchmark folder from the GitHub repo vibeeval/vibecosystem into my ~/.claude/skills/agent-benchmark/.
When it's done, tell me in one line what this skill can do.

Install with a command instead

git clone https://github.com/vibeeval/vibecosystem.git /tmp/vibecosystem && mkdir -p ~/.claude/skills && cp -r /tmp/vibecosystem/skills/agent-benchmark ~/.claude/skills/

⚠ This is a third-party skill. Check the source repository before installing.

  1. Open a terminal (Terminal on macOS, WSL or Git Bash on Windows).
  2. Clone the repository: git clone https://github.com/vibeeval/vibecosystem.git
  3. Create the skills folder: mkdir -p ~/.claude/skills
  4. Copy the skill in: cp -r vibecosystem/skills/agent-benchmark ~/.claude/skills/
  5. Restart Claude Code and confirm agent-benchmark appears in your skill list.
  6. Verify Node.js 18+ is installed: node -v
  7. Ask Claude Code to scaffold the missing runner: "Create ~/.claude/benchmarks with the directory layout and run.mjs runner described in the agent-benchmark skill."
  8. Add your first fixture and ground-truth file, run the benchmark, then save the result as your initial baseline.
View source on GitHub ↗License: MIT