LLM Eval Harness

What it does

Give it a base_url, a model, and an API key held in an environment variable, and it produces measured answers instead of vendor headline numbers.

• Availability — which model IDs actually route, with 3-state failure classification (no-channel, upstream-error, empty-content) and a deliberately high max_tokens so reasoning models don't read as dead.
• Request fidelity — does your system prompt, tool schema, and conversation history actually reach the model? A canary code catches the nastiest gateway bug: HTTP 200, fluent replies, and a system prompt that was silently dropped.
• Speed — TTFT plus sustained decode tok/s, thinking-aware. It captures reasoning_content separately so you don't report a ~750 tok/s model as 4700 tok/s.
• Concurrency — ramp 10/20/40/60 to find where success rate or p90 latency collapses; ambient proxies and keep-alive pooling are disabled so you measure the model, not the proxy.
• Protocol compliance — two orthogonal Anthropic checks: does thinking generation fire, and does the endpoint accept a thinking block replayed in history (the failure that kills agentic sessions on turn two)?
• Quality regression — run your own use-case library, then have three independent blind judges score in isolation; a case passes only on majority agreement, and silence never counts as a pass.

Keys are always passed by env-var name, never by value, so they stay out of ps, shell history, and saved reports. Your use-case library and model rosters live in ~/.llm-eval/, outside the public skill bundle.

Who it's for

• Backend/platform engineers vetting a new LLM gateway or reseller before adoption
• Agent developers chasing a "the model ignores my system prompt" bug with no error anywhere
• Operators who need to know a channel survives real concurrency before a workshop or launch
• Team leads who must prove with data that switching models won't regress quality

Examples

1. Onboarding a provider: "test this gateway before we integrate" → availability probe builds the supported-model list, then a 10-sample fidelity probe reports whether system-prompt delivery is delivered, intermittent 6/10, or not-delivered.
2. Sessions that die mid-conversation: a 400 "invalid part type: thinking" on the second turn → run protocol_probe.py --check history-replay, then cross-check the vendor's own native endpoint before blaming the reseller.
3. Verifying a tok/s claim: speed_probe.py --mode both reports real-task throughput alongside the sustained decode ceiling, so the write-up separates what was measured from what was inferred.

Install the skill I found on Claude Skill Mart.
Copy the llm-eval-harness folder from the GitHub repo daymade/claude-code-skills into my ~/.claude/skills/llm-eval-harness/.
When it's done, tell me in one line what this skill can do.