Claude Skill MartBrowse skillsQuick linesLearn by videoTerminal guideWhat is a Skill?
 
← Back to list

Debug: Evidence-Before-Action ML Diagnosis

Forces a probe → hypothesis → smoke → control → claim discipline when an ML training run fails, diverges, OOMs, hangs, or looks slow.

Dev & CodingIntermediate★ 414⑂ 36AI score 8/10Last updated: Sep 16, 2026

What it does

Stops the assistant from asserting a plausible-sounding cause and "fixing" it. Instead it enforces five steps:

  1. Cheap probes: ps aux for live/zombie processes, dmesg and journalctl for oom-kills, hardware and NFS errors, nvidia-smi for real GPU utilization and memory, df -h/du -sh for disk, the last few hundred log lines, plus checkpoint timestamps and sizes.
  2. A labeled hypothesis, stated as a hypothesis with the alternatives not yet ruled out.
  3. A smoke run to confirm or refute cheaply: batch_size=1 for one step, a synthetic in-memory dataset, forward-only in eval(), lr=0, or single-GPU to implicate DDP/NCCL.
  4. Controls: change exactly one variable at a time (AMP, gradient checkpointing, torch.compile, worker count) to narrow the mechanism.
  5. Cause claim only once evidence stacks up, citing the exact tool output; otherwise say "I don't yet know" and propose the next probe.

Who it's for

  • ML researchers and engineers training with PyTorch, torchrun, or accelerate on Linux GPU boxes
  • Anyone whose overnight cluster jobs die and who ends up guessing from logs
  • Teams who want agents to justify diagnoses with real tool output instead of speculative patches

Examples

  • "My run vanished after 3 hours" → correlate dmesg oom-kill entries with the last checkpoint write time to separate a kernel OOM from an external kill.
  • "Loss went to NaN" → smoke with lr=0; if it still explodes, the loss function, not the optimizer, is broken.
  • "GPU util is 0 but it says training" → check process state and GPU stats for dataloader blocking, then run a control changing only worker count.

· · · Install guide · · ·

Try it now, no install

Paste this into Claude to use the skill without installing anything.

Read the instructions in this file and follow them to help me:
https://raw.githubusercontent.com/fcakyon/phd-skills/HEAD/plugin/skills/debug/SKILL.md

What I want: (describe your task here)

If Claude can't open the link, open it yourself and paste the contents instead.

↓ If it works for you, download the ZIP below and install it. Then it runs on its own — no pasting each time.

Install in the Claude app (no terminal)
  1. Download the ZIP with the button below.
  2. In Claude, open Settings → Capabilities and turn on 'Code execution and file creation'. (one time)
  3. Go to Customize → Skills → + → 'Upload a skill' and upload the ZIP.
↓ Download ZIP
Install in Claude Code

Let Claude do it — paste this into Claude Code

Install the skill I found on Claude Skill Mart.
Copy the plugin/skills/debug folder from the GitHub repo fcakyon/phd-skills into my ~/.claude/skills/fcakyon-debug/.
When it's done, tell me in one line what this skill can do.

Install with a command instead

git clone https://github.com/fcakyon/phd-skills.git && mkdir -p ~/.claude/skills && cp -r phd-skills/plugin/skills/debug ~/.claude/skills/

⚠ This is a third-party skill. Check the source repository before installing.

  1. Open a terminal.
  2. Clone the repo: git clone https://github.com/fcakyon/phd-skills.git
  3. Create the skills directory: mkdir -p ~/.claude/skills
  4. Copy just this skill: cp -r phd-skills/plugin/skills/debug ~/.claude/skills/
  5. Verify it landed: ls ~/.claude/skills/debug/SKILL.md
  6. Restart Claude Code and ask something like "debug why this training run is diverging" to trigger it.
  7. Note: probes such as nvidia-smi and dmesg only make sense on a Linux machine with NVIDIA GPUs, so run Claude Code on the training host for best results.
View source on GitHub ↗License: MIT