LLM Eval Harness

What it does

Give it a base_url, a model ID, and an API key held in an environment variable, and it produces measured numbers instead of vendor marketing claims.

• Availability — which model IDs actually route, with three-state failure classification (no-channel, upstream-error, empty-content), keeping max_tokens at 8192 so reasoning models don't read as dead.
• Request fidelity — canary codes verify whether your system prompt, tools, and multi-turn history actually reach the model, or get silently dropped by a gateway that still returns a fluent 200.
• Speed — TTFT plus sustained decode tok/s, capturing reasoning_content so thinking tokens don't inflate throughput (a real 750 tok/s model once measured as 4700).
• Concurrency — ramps 10/20/40/60 to find where success rate or p50/p90 latency breaks, disabling ambient proxies and keep-alive so you measure the model, not the proxy.
• Protocol compliance — separately checks Anthropic thinking generation and history-replay acceptance, the latter being the cause of "session dies with 400 invalid part type: thinking" in agentic clients.
• Quality regression — runs your own use-case library, then has three independent blind judges score each answer, passing only on majority agreement and reporting precision per tag category.

Keys are always passed by env-var name (--key-env), never as a command-line value, so they stay out of ps, shell history, and saved reports.

Who it's for

• Developers and platform engineers vetting a new LLM provider, reseller, or gateway before adoption
• Owners of supported-model lists, deployment gates, or resident canary monitors
• Anyone debugging vague symptoms like "the model ignores the system prompt" or "it 400s after a few turns"
• Teams that need to know how much concurrency an endpoint survives before a workshop or demo

Example uses

1. Onboarding a new channel: run the availability probe to find which IDs route, then the fidelity probe with --repeat 10 to see how reliably the system prompt is delivered.
2. Verifying a tok/s claim: speed_probe.py --mode both reports real-task throughput and the sustained decode ceiling separately, so you compare apples to apples.
3. Tracking down broken sessions: protocol_probe.py --check history-replay reproduces a vendor rejecting replayed thinking blocks, then the bundled evidence protocol turns it into a filable bug report.

Read the instructions in this file and follow them to help me:
https://raw.githubusercontent.com/daymade/claude-code-skills/HEAD/llm-eval-harness/SKILL.md

What I want: (describe your task here)

Install the skill I found on Claude Skill Mart.
Copy the llm-eval-harness folder from the GitHub repo daymade/claude-code-skills into my ~/.claude/skills/llm-eval-harness/.
When it's done, tell me in one line what this skill can do.