AI Skill MartBrowse skillsPromptsProgramsLearnWhat is a Skill?
 
← Back to list

Promptfoo LLM Evaluation Setup

A skill that guides you through configuring and running prompt tests and model comparisons with Promptfoo.

Dev & CodingIntermediate★ 1,323⑂ 212AI score 8/10Last updated: Aug 8, 2026

What it does

  • Scaffolds a working promptfooconfig.yaml (prompts, providers, tests, defaultTest, outputPath).
  • Shows how to write Python custom assertions (get_assert and named functions), llm-rubric LLM-as-judge checks, and common assertion types like contains, regex, and latency.
  • Covers multi-turn few-shot prompt patterns in chat.json and the echo provider for previewing rendered prompts with zero API cost.
  • Documents real gotchas: apiBaseUrl belongs in providers[].config, maxConcurrency must sit under commandLineOptions, file:// paths resolve from the config root, and graders need their own provider behind a relay.

Who it's for

  • AI app developers who want regression tests for prompt quality instead of eyeballing outputs.
  • Teams comparing Claude, GPT, and other models on a shared test set.
  • Anyone calling LLM APIs through a corporate relay/proxy and fighting promptfoo config issues.

Examples

  1. "Set up a promptfoo project to evaluate my summarization prompt" → generates config, tests/cases.yaml, scripts/metrics.py.
  2. "Write a Python assertion that passes when the reduction ratio is 70–90%" → a check_length function that strips HTML then computes the ratio.
  3. "My llm-rubric returns 401 through our relay" → diagnosis that each rubric needs its own provider.config.apiBaseUrl.

· · · Install guide · · ·

Try it now, no install

Paste this into Claude to use the skill without installing anything.

Read the instructions in this file and follow them to help me:
https://raw.githubusercontent.com/daymade/claude-code-skills/HEAD/promptfoo-evaluation/SKILL.md

What I want: (describe your task here)

If Claude can't open the link, open it yourself and paste the contents instead.

↓ If it works for you, download the ZIP below and install it. Then it runs on its own — no pasting each time.

Install in the Claude app (no terminal)
  1. Download the ZIP with the button below.
  2. In Claude, open Settings → Capabilities and turn on 'Code execution and file creation'. (one time)
  3. Go to Customize → Skills → + → 'Upload a skill' and upload the ZIP.
↓ Download ZIP
Install in Claude Code

Let Claude do it — paste this into Claude Code

Install the skill I found on AI Skill Mart.
Copy the promptfoo-evaluation folder from the GitHub repo daymade/claude-code-skills into my ~/.claude/skills/promptfoo-evaluation/.
When it's done, tell me in one line what this skill can do.

Install with a command instead

git clone https://github.com/daymade/claude-code-skills.git && mkdir -p ~/.claude/skills && cp -r claude-code-skills/promptfoo-evaluation ~/.claude/skills/

⚠ This is a third-party skill. Check the source repository before installing.

  1. Open a terminal and go to your working directory.
  2. Clone the repo: git clone https://github.com/daymade/claude-code-skills.git
  3. Create the skills folder: mkdir -p ~/.claude/skills
  4. Copy the skill: cp -r claude-code-skills/promptfoo-evaluation ~/.claude/skills/
  5. Make sure Node.js (npx) and Python 3 are installed; set export PROMPTFOO_PYTHON=python3 if needed.
  6. Export the API keys you plan to use (ANTHROPIC_API_KEY, OPENAI_API_KEY, etc.).
  7. Restart Claude Code and ask something like "set up a promptfoo evaluation for my prompt" to trigger the skill.
View source on GitHub ↗License: MIT