Claude Skill MartBrowse skillsQuick linesLearn by videoTerminal guideWhat is a Skill?
 
← Back to list

nanoGPT Minimal GPT Training Guide

A hands-on guide to nanoGPT: train a tiny Shakespeare model on CPU or reproduce GPT-2 (124M) on multi-GPU, step by step.

EducationIntermediate★ 13,448⑂ 948AI score 8/10Last updated: Oct 11, 2026

What it does

  • Walks through the full nanoGPT pipeline — data prep, training, sampling — with exact commands for Shakespeare, OpenWebText, fine-tuning and custom datasets.
  • Explains the key hyperparameters in train_shakespeare_char.py, train_gpt2.py and finetune_shakespeare.py (n_layer, n_head, block_size, learning rate, gradient accumulation).
  • Provides fixes for common failures: CUDA OOM, slow training, poor sample quality, failing GPT-2 weight loading.
  • Lists realistic runtime/VRAM expectations (CPU, T4, 1× and 8× A100) and when to switch to HuggingFace, Megatron-LM or LitGPT instead.

Who it's for

  • Learners who want to understand transformer/GPT internals at the code level.
  • Instructors and study groups needing a reproducible small-LLM training exercise.
  • ML engineers and researchers prototyping architecture variants quickly.

Examples

  1. "I only have a laptop CPU — can I train an LM in 5 minutes?" → three-command shakespeare_char prep/train/sample flow.
  2. "How do I reproduce GPT-2 124M on 8 GPUs?" → torchrun DDP command plus batch and gradient-accumulation settings.
  3. "Train on my own text file" → character-level tokenizer script that writes train.bin / val.bin.

· · · Install guide · · ·

Try it now, no install

Paste this into Claude to use the skill without installing anything.

Read the instructions in this file and follow them to help me:
https://raw.githubusercontent.com/Orchestra-Research/AI-Research-SKILLs/HEAD/01-model-architecture/nanogpt/SKILL.md

What I want: (describe your task here)

If Claude can't open the link, open it yourself and paste the contents instead.

↓ If it works for you, download the ZIP below and install it. Then it runs on its own — no pasting each time.

Install in the Claude app (no terminal)
  1. Download the ZIP with the button below.
  2. In Claude, open Settings → Capabilities and turn on 'Code execution and file creation'. (one time)
  3. Go to Customize → Skills → + → 'Upload a skill' and upload the ZIP.
↓ Download ZIP
Install in Claude Code

Let Claude do it — paste this into Claude Code

Install the skill I found on Claude Skill Mart.
Copy the 01-model-architecture/nanogpt folder from the GitHub repo Orchestra-Research/AI-Research-SKILLs into my ~/.claude/skills/nanogpt/.
When it's done, tell me in one line what this skill can do.

Install with a command instead

git clone https://github.com/Orchestra-Research/AI-Research-SKILLs.git /tmp/ai-research-skills && mkdir -p ~/.claude/skills/nanogpt && cp -r /tmp/ai-research-skills/01-model-architecture/nanogpt/* ~/.claude/skills/nanogpt/

⚠ This is a third-party skill. Check the source repository before installing.

  1. Open a terminal and clone the skill repo: git clone https://github.com/Orchestra-Research/AI-Research-SKILLs.git
  2. Create the skill directory: mkdir -p ~/.claude/skills/nanogpt
  3. Copy the files: cp -r AI-Research-SKILLs/01-model-architecture/nanogpt/* ~/.claude/skills/nanogpt/
  4. Clone the upstream code that the skill references: git clone https://github.com/karpathy/nanoGPT.git
  5. Install the Python dependencies: pip install torch numpy transformers datasets tiktoken wandb tqdm
  6. Restart Claude Code and ask something like "train a Shakespeare model with nanoGPT" to confirm the skill loads.
View source on GitHub ↗License: MIT