AI Skill MartBrowse skillsPromptsProgramsLearnWhat is a Skill?
 
← Back to list

Add Runner Eval

Internal skill for adding or extending Paperclip Runner protocol eval definitions, rosters, assertions, and report fixtures with strict provenance and narrow validation.

Dev & CodingAdvanced★ 108,978⑂ 18,043AI score 6/10Last updated: Oct 30, 2026

What it does

Locks down the workflow for the Runner Evals family — a real Runner/provider session against a seeded mock control plane.

  • Checkout discovery rules (PAPERCLIP_ROOT, git rev-parse --show-toplevel, PAPERCLIP_EVALS_ROOT)
  • File layout: cases in cases/, company fixtures in fixtures/, runtime/model settings in configs/, selections in rosters/, campaign membership in campaigns/live-direct-full.json
  • Four provider-free validate commands to run before any live run
  • Publication boundary: sanitized fixture conversation and allowlisted tool outcomes may be public; credentials, secrets, raw trusted payloads, and hidden reasoning may not

Who it's for

  • Internal engineers working in the paperclip repo alongside the sibling paperclip-evals repo
  • Maintainers of Runner protocol graders and report fixtures
  • Not useful outside that setup — both repos are required

Examples

  1. Author a bounded case, then run eval_program.py validate --case ... --config live-codex-pinned.json with no provider calls.
  2. Add the case to rosters/live-mini.json and verify with run_live_roster.py validate --run-id validate-new-case.
  3. Add a negative case with malformed or missing evidence to confirm the grader fails closed.

· · · Install guide · · ·

Try it now, no install

Paste this into Claude to use the skill without installing anything.

Read the instructions in this file and follow them to help me:
https://raw.githubusercontent.com/paperclipai/paperclip/HEAD/.agents/skills/add-runner-eval/SKILL.md

What I want: (describe your task here)

If Claude can't open the link, open it yourself and paste the contents instead.

↓ If it works for you, download the ZIP below and install it. Then it runs on its own — no pasting each time.

Install in the Claude app (no terminal)
  1. Download the ZIP with the button below.
  2. In Claude, open Settings → Capabilities and turn on 'Code execution and file creation'. (one time)
  3. Go to Customize → Skills → + → 'Upload a skill' and upload the ZIP.
↓ Download ZIP
Install in Claude Code

Let Claude do it — paste this into Claude Code

Install the skill I found on AI Skill Mart.
Copy the .agents/skills/add-runner-eval folder from the GitHub repo paperclipai/paperclip into my ~/.claude/skills/add-runner-eval/.
When it's done, tell me in one line what this skill can do.

Install with a command instead

git clone https://github.com/paperclipai/paperclip.git /tmp/paperclip && mkdir -p ~/.claude/skills/add-runner-eval && cp -r /tmp/paperclip/.agents/skills/add-runner-eval/* ~/.claude/skills/add-runner-eval/

⚠ This is a third-party skill. Check the source repository before installing.

  1. Open a terminal.
  2. Clone the repo: git clone https://github.com/paperclipai/paperclip.git
  3. Create the skill folder: mkdir -p ~/.claude/skills/add-runner-eval
  4. Copy the files: cp -r paperclip/.agents/skills/add-runner-eval/* ~/.claude/skills/add-runner-eval/
  5. For real use, also check out the sibling paperclip-evals repo next to it and set PAPERCLIP_ROOT / PAPERCLIP_EVALS_ROOT if needed.
  6. Restart Claude Code and ask it to "add a Runner eval case" to trigger the skill.
View source on GitHub ↗License: MIT