Claude Skill MartBrowse skillsQuick linesLearn by videoTerminal guideWhat is a Skill?
 
← Back to list

Agent Eval Harness

Build a small, real-world eval set to prove an agent or prompt works — and catch regressions whenever the model or prompt changes.

Dev & CodingIntermediate★ 299⑂ 49AI score 7/10Last updated: Sep 20, 2026

What it does

  • Writes the pass condition first — valid JSON shape, correct figure present, tool called with the right id, refusal on specified inputs, under 120 words — instead of vague "quality" goals.
  • Builds a minimal ~20-case set sourced from real transcripts, mixed as 40% happy path, 30% edge cases, 20% adversarial, 10% frozen regressions.
  • Scores in three tiers: deterministic checks first, model-as-judge with a rubric and one-line reason for tone/faithfulness, plus human spot-checks on the cases that matter.
  • Runs it like a test suite: three runs per case at shipping temperature, reporting pass rate with variance and a diff against the baseline rather than an absolute score.
  • Tracks cost and latency (p50/p95) alongside quality so you don't buy two points for 4x the spend.

Who it's for

  • Engineers shipping LLM-backed features who edit prompts by vibe
  • Teams validating quality before and after a model version upgrade
  • Startups about to put an agent in front of customers
  • Anyone wiring LLM regression checks into CI

Examples

  1. Model swap check — "Will my prompt break on the new model?" Pull 20 cases from production logs, run both, report which cases regressed.
  2. Injection hardening — add "ignore previous instructions" cases; a leak in even 1 of 3 runs is flagged as a ship blocker.
  3. Bug-to-regression pipeline — every customer-reported failure is frozen as a permanent case once fixed, so it can never quietly return.

· · · Install guide · · ·

Try it now, no install

Paste this into Claude to use the skill without installing anything.

Read the instructions in this file and follow them to help me:
https://raw.githubusercontent.com/OneWave-AI/claude-skills/HEAD/agent-eval-harness/SKILL.md

What I want: (describe your task here)

If Claude can't open the link, open it yourself and paste the contents instead.

↓ If it works for you, download the ZIP below and install it. Then it runs on its own — no pasting each time.

Install in the Claude app (no terminal)
  1. Download the ZIP with the button below.
  2. In Claude, open Settings → Capabilities and turn on 'Code execution and file creation'. (one time)
  3. Go to Customize → Skills → + → 'Upload a skill' and upload the ZIP.
↓ Download ZIP
Install in Claude Code

Let Claude do it — paste this into Claude Code

Install the skill I found on Claude Skill Mart.
Copy the agent-eval-harness folder from the GitHub repo OneWave-AI/claude-skills into my ~/.claude/skills/agent-eval-harness/.
When it's done, tell me in one line what this skill can do.

Install with a command instead

git clone https://github.com/OneWave-AI/claude-skills.git /tmp/claude-skills && mkdir -p ~/.claude/skills && cp -r /tmp/claude-skills/agent-eval-harness ~/.claude/skills/

⚠ This is a third-party skill. Check the source repository before installing.

  1. Open a terminal.
  2. Clone the repo: git clone https://github.com/OneWave-AI/claude-skills.git
  3. Create the skills directory: mkdir -p ~/.claude/skills
  4. Copy this skill in: cp -r claude-skills/agent-eval-harness ~/.claude/skills/
  5. Restart Claude Code and try: "Build an eval set to check whether this prompt works."
  6. Note: the evals/run.mjs runner referenced in the doc is not bundled — ask Claude to scaffold one for your project (Node.js required).
View source on GitHub ↗License: MIT