Claude Skill MartBrowse skillsQuick linesLearn by videoTerminal guideWhat is a Skill?
 
← Back to list

Paper Autoraters

Runs PaperOrchestra's four LLM-as-judge autoraters — Citation F1, 6-axis literature-review quality, and two side-by-side comparisons — on your paper drafts.

EducationAdvanced★ 681⑂ 94AI score 9/10Last updated: Oct 11, 2026

What it does

A faithful implementation of the four autoraters from PaperOrchestra (App. F.3):

  • Citation F1: partitions each reference list into P0 (must-cite) and P1 (good-to-cite), then computes P0/P1/overall Precision, Recall and F1 against a ground-truth paper via scripts/compute_f1.py.
  • Literature Review Quality: a 6-axis 0-100 rubric applied only to Introduction + Related Work, with anti-inflation caps (e.g. descriptive reviews cap Critical Analysis at 60) and an explicit penalty table.
  • SxS Overall Paper Quality / SxS Literature Review: side-by-side preference judgments run twice with the order swapped to neutralize positional bias, yielding win / tie / loss.

Who it's for

  • Researchers benchmarking automated paper-writing pipelines against baselines
  • Grad students auditing citation rigor and gap statements before submission
  • Teams that need a fair A/B verdict between two drafts or two model outputs

Examples

  1. "Score this paper draft against the benchmark" → JSON with axis scores, penalties, and an overall score.
  2. "Compute Citation F1 between my generated paper and the original" → P0/P1 partitions plus an f1_report.json.
  3. "Compare draft v2 and draft v3" → two order-swapped SxS judgments resolved into win/tie/loss.

· · · Install guide · · ·

Try it now, no install

Paste this into Claude to use the skill without installing anything.

Read the instructions in this file and follow them to help me:
https://raw.githubusercontent.com/Ar9av/PaperOrchestra/HEAD/skills/paper-autoraters/SKILL.md

What I want: (describe your task here)

If Claude can't open the link, open it yourself and paste the contents instead.

↓ If it works for you, download the ZIP below and install it. Then it runs on its own — no pasting each time.

Install in the Claude app (no terminal)
  1. Download the ZIP with the button below.
  2. In Claude, open Settings → Capabilities and turn on 'Code execution and file creation'. (one time)
  3. Go to Customize → Skills → + → 'Upload a skill' and upload the ZIP.
↓ Download ZIP
Install in Claude Code

Let Claude do it — paste this into Claude Code

Install the skill I found on Claude Skill Mart.
Copy the skills/paper-autoraters folder from the GitHub repo Ar9av/PaperOrchestra into my ~/.claude/skills/paper-autoraters/.
When it's done, tell me in one line what this skill can do.

Install with a command instead

git clone https://github.com/Ar9av/PaperOrchestra.git && mkdir -p ~/.claude/skills && cp -r PaperOrchestra/skills/paper-autoraters ~/.claude/skills/

⚠ This is a third-party skill. Check the source repository before installing.

  1. Open a terminal.
  2. Clone the repository: git clone https://github.com/Ar9av/PaperOrchestra.git
  3. Create the skills directory: mkdir -p ~/.claude/skills
  4. Copy the skill: cp -r PaperOrchestra/skills/paper-autoraters ~/.claude/skills/
  5. Confirm that references/ and scripts/compute_f1.py came along.
  6. Check Python 3 is available: python3 --version
  7. Restart Claude Code and ask something like "run the autoraters on this draft" to trigger the skill.
View source on GitHub ↗License: NOASSERTION