AI Skill MartBrowse skillsPromptsProgramsLearnWhat is a Skill?
 
← Back to list

Debug: Evidence-Before-Action Investigation

Forces ML training failures to be diagnosed by probing for evidence first, then hypothesis, smoke test, control, and only then a root-cause claim.

Dev & CodingIntermediate★ 448⑂ 38AI score 8/10Last updated: Nov 1, 2026

What it does

  • When a run is failing, diverging, OOMing, hanging, or producing weird metrics, it stops the assistant from asserting a plausible-sounding cause and makes it gather evidence first.
  • Enforces a five-step protocol: ① cheap probes (ps aux, dmesg, journalctl, nvidia-smi, df -h, log scrollback, checkpoint listing) → ② hypothesis explicitly labeled as a hypothesis with alternatives → ③ a 30-second smoke run to confirm or refute → ④ single-variable control experiments → ⑤ a cause claim that cites the exact tool output.
  • Ships per-hypothesis smoke recipes: batch_size=1 for OOM, lr=0 for optimizer suspicion, single-GPU for DDP/NCCL, synthetic dataset for data-path suspicion, eval-mode forward pass for model suspicion.
  • If evidence doesn't stack up, it must say "I don't yet know" and propose the next probe.

Who it's for

  • ML engineers and researchers running PyTorch / torchrun / accelerate jobs
  • Anyone babysitting long training runs on GPU servers or clusters
  • Teams tired of "restart-and-pray" debugging

Example uses

  1. "Loss went NaN, why?" — re-reads the last few hundred log lines and grad_norm pattern, then runs an lr=0 smoke to separate a broken loss from a broken optimizer.
  2. "GPU util has been 0 for three hours but the process is alive" — checks ps aux, nvidia-smi, and dmesg for dataloader blocking, then swaps in a synthetic dataset.
  3. "Last night's run just died" — correlates a oom-kill line in dmesg with the last checkpoint timestamp/size to distinguish a kernel OOM from an NFS write timeout.

· · · Install guide · · ·

Try it now, no install

Paste this into Claude to use the skill without installing anything.

Read the instructions in this file and follow them to help me:
https://raw.githubusercontent.com/fcakyon/labrat/HEAD/plugin/skills/debug/SKILL.md

What I want: (describe your task here)

If Claude can't open the link, open it yourself and paste the contents instead.

↓ If it works for you, download the ZIP below and install it. Then it runs on its own — no pasting each time.

Install in the Claude app (no terminal)
  1. Download the ZIP with the button below.
  2. In Claude, open Settings → Capabilities and turn on 'Code execution and file creation'. (one time)
  3. Go to Customize → Skills → + → 'Upload a skill' and upload the ZIP.
↓ Download ZIP
Install in Claude Code

Let Claude do it — paste this into Claude Code

Install the skill I found on AI Skill Mart.
Copy the plugin/skills/debug folder from the GitHub repo fcakyon/labrat into my ~/.claude/skills/fcakyon-debug-2/.
When it's done, tell me in one line what this skill can do.

Install with a command instead

git clone https://github.com/fcakyon/labrat.git /tmp/labrat && mkdir -p ~/.claude/skills && cp -r /tmp/labrat/plugin/skills/debug ~/.claude/skills/debug

⚠ This is a third-party skill. Check the source repository before installing.

  1. Open a terminal.
  2. Clone the repo: git clone https://github.com/fcakyon/labrat.git /tmp/labrat
  3. Create the skills directory: mkdir -p ~/.claude/skills
  4. Copy the skill: cp -r /tmp/labrat/plugin/skills/debug ~/.claude/skills/debug
  5. Verify it landed: ls ~/.claude/skills/debug/SKILL.md
  6. Restart Claude Code and ask something like "debug why this training run crashed" to trigger it.
  7. Note: the probes rely on dmesg, nvidia-smi, and journalctl, so it works best on a Linux box with NVIDIA GPUs and shell access.
View source on GitHub ↗License: MIT