Claude Skill MartBrowse skillsWhat is a Skill?
 
← Back to list

Promptfoo Evaluation

Sets up and runs Promptfoo-based LLM evaluations, from config files to custom Python and LLM-rubric assertions.

Dev & CodingIntermediate★ 1,323⑂ 212AI score 9/10Last updated: Aug 8, 2026

What it does

Guides Claude through configuring and running Promptfoo, the open-source CLI for testing and comparing LLM outputs.

  • Scaffolds promptfooconfig.yaml (prompts, providers, tests, defaultTest, outputPath)
  • Writes test-case YAML and explains file:// resolution rules (always relative to the config file)
  • Creates Python custom assertions returning pass / score / reason / named_scores
  • Designs llm-rubric LLM-as-judge criteria with thresholds and per-assertion graders
  • Uses the echo provider to preview rendered prompts with zero API cost
  • Multi-turn few-shot prompt patterns and long-text (10k+ char) evaluation recipes
  • Fixes common pitfalls: relay/proxy 401s, silently ignored maxConcurrency, truncated outputs, HTML tags skewing metrics

Who it's for

  • Developers who want measurable prompt regressions instead of vibes
  • Teams benchmarking Claude vs GPT vs others on a shared test set
  • AI engineers wiring prompt evals into CI pipelines
  • Anyone needing domain-specific metrics (compression ratio, length bounds, format checks)

Examples

  1. Model bake-off: "Build a promptfoo config comparing claude-sonnet and gpt-4.1 across my 20 cases" → full config with two providers, external tests file, and JSON output.
  2. Custom metric: "Assert my summaries compress the source by 70–90%" → generates metrics.py that strips HTML then scores the reduction ratio.
  3. Free debugging: "Verify my few-shot variables render correctly without burning tokens" → creates a preview config using the echo provider.

· · · Install guide · · ·

Install in the Claude app (no terminal)
  1. Download the ZIP with the button below.
  2. In Claude, open Settings → Capabilities and turn on 'Code execution and file creation'. (one time)
  3. Go to Customize → Skills → + → 'Upload a skill' and upload the ZIP.
↓ Download ZIP
Install in Claude Code

Let Claude do it — paste this into Claude Code

Install the skill I found on Claude Skill Mart.
Copy the promptfoo-evaluation folder from the GitHub repo daymade/claude-code-skills into my ~/.claude/skills/promptfoo-evaluation/.
When it's done, tell me in one line what this skill can do.

Install with a command instead

git clone https://github.com/daymade/claude-code-skills.git /tmp/claude-code-skills && mkdir -p ~/.claude/skills && cp -r /tmp/claude-code-skills/promptfoo-evaluation ~/.claude/skills/

⚠ This is a third-party skill. Check the source repository before installing.

  1. Open a terminal (Terminal on macOS, WSL or Git Bash on Windows).
  2. Create the skills folder: mkdir -p ~/.claude/skills
  3. Clone the repo: git clone https://github.com/daymade/claude-code-skills.git /tmp/claude-code-skills
  4. Copy just this skill: cp -r /tmp/claude-code-skills/promptfoo-evaluation ~/.claude/skills/
  5. Confirm Node.js is installed: node -v (install the LTS build from nodejs.org if missing).
  6. Smoke-test Promptfoo: npx promptfoo@latest --version
  7. Export the API key for the model you'll evaluate, e.g. export ANTHROPIC_API_KEY=sk-... (add export PROMPTFOO_PYTHON=python3 if you'll use Python assertions).
  8. Restart Claude Code and ask something like "set up a promptfoo eval for my prompt" — the skill triggers automatically.
View source on GitHub ↗License: MIT