Courses / Course 03 · Ml
You end up withAn autograd engine you wrote, a GPT you trained from scratch with a tokenizer you wrote, and a from-scratch inference loop for a modern open-weights architecture. Enough to read a model's source and know what every line does.
Why build one when you already talk to one every day? Because the agent that writes your code is one of these. After this track, “context window”, “temperature”, “tokens” and “fine-tuning” are things you’ve implemented, not words you’ve heard. Hacker News has voted these tutorials up since 2013, and more than half the stories in this track’s pool are from 2023 or later. Every step below shows its HN score and quotes what HN said.
What this track won’t do for you, in HN’s words:
“Although in retrospect knowing how a transformer works wasn’t very useful at all in my day job applying LLMs, except as a sort of deep background for reassurance that I had some idea of how the big black box producing the tokens was put together, and to give me the mathematical basis for things like context size limitations etc.”
You’ll understand the machine. That isn’t the same as predicting what a frontier model will do.
How to read the receipts.
The visual intuition for what a network, a gradient and backpropagation are, before you write any code. The series has since grown chapters on transformers and attention.
“In my opinion this author produces the best math videos on youtube.”
A two-layer network with the gradients worked out by hand and trained with plain stochastic gradient descent. It’s small enough to finish in one sitting, and it makes Stage 2 less abstract.
“I see so many implementations of NeuralNets from Scratch on Repl.it (I’m a co-founder) but this is one of the simplest and clearest one.”
HN’s caveat (about the field as much as the post): “Why are deep learning researchers allergic to meaningful variable names?” (gambler, 2019 · comment)
The core of the track: Andrej Karpathy’s video series, where you write every piece yourself, in order. https://karpathy.ai/zero-to-hero.html · 1,450 points for the series page, submitted 7× (2023–2026) · Free · Python, then PyTorch
“This series of videos was by far the best, most “intuition building”, highest signal-to-noise ratio, and least “annoying” content to get through.”
“I have taken several masters-level courses in Machine Learning — and even with those credentials, I cannot recommend enough Andrej’s youtube series, “Neural Networks: Zero to Hero”.”
A note on the receipts: HN voted for the series as a whole and for the GPT video. The early episodes scored little on their own: micrograd 171 points, makemore 50. The step order below is the series order, not HN’s.
You write a scalar autograd engine and a tiny neural-net library on top of it: backpropagation from first principles, in about 150 lines.
“This was the first time I actually grokked backpropagation, just the first video alone is more lucid and valuable than any other resource about machine learning I had seen before…”
A character-level language model, rebuilt five times: bigram counts, an MLP, activations and batch norm, “Backprop Ninja” (you derive every gradient by hand and check it against PyTorch), and a WaveNet-style model.
“The “Backprop Ninja” video was hard work, you manually calculate the gradients and then compare your calculations against PyTorch.”
Self-attention, multi-head attention, residual connections and layer norm, building a decoder-only transformer trained on Shakespeare.
“Andrej’s entire series is by far one of the most useful resources I’ve found. Even as an instructor of these topics myself I learn something new about the material and about how to teach it!”
You write byte-pair encoding (the minbpe repo) and learn why tokenization explains so many odd LLM behaviours, from spelling to arithmetic.
“No metaphors trying explain “complex” ideas, making them scary and seem overly complex. Instead, hands on implementations with analogy explainers where you can actually understand the ideas and see how simple it is.”
Alternatives HN votes for:
GPT-2’s forward pass in plain NumPy, loading OpenAI’s real weights. After step 4 it’s a check: if you can read all 60 lines, you understand the architecture.
“This article is an absolutely fantastic introduction to GPT models - I think the clearest I’ve seen anywhere, at least for the first section that talks about generating text and sampling.”
Read alongside (pick one or two, not all):
The book-length version of Stages 2–3 done properly. It covers data loading, attention, a GPT model, pretraining, loading GPT-2 weights, then fine-tuning for classification and for following instructions. The README says the main chapters run on a conventional laptop.
Free companion: Writing an LLM from scratch · Giles Thomas · https://www.gilesthomas.com/llm-from-scratch · 1,741 points across 32 submissions of its parts (2025–2026). It’s one engineer’s long, honest log of working through Raschka’s book, ending with a base model trained from scratch on an RTX 3090. “I hadn’t thought that an RTX 3090 would be capable of generating a sort-of decent small LLM from scratch in a reasonable time, but he shows how in detail.” (RagnarD, 2025 · thread)
The smallest serious GPT training code: reproduce GPT-2 (124M) on OpenWebText, or fine-tune on your own text. The companion video Let’s reproduce GPT-2 (124M) (https://www.youtube.com/watch?v=l8pRSuU81PU; 9 points, a weak HN signal) builds it from an empty file.
Alternative: minGPT (Karpathy) · https://github.com/karpathy/minGPT · 597 points, submitted 2×. It’s nanoGPT’s older, more educational sibling.
Optional, for systems people: llm.c · Karpathy · https://github.com/karpathy/llm.c · 1,923 points, submitted 8× (2024–2025) · Free · C/CUDA. GPT-2 training in plain C and CUDA, with no PyTorch. “Kind of amazing that something that can be expressed in ~1000 lines of code has completely turned the world on its head.” (osigurdson, 2024 · thread)
Loads Meta’s real Llama 3 weights and runs them one matrix multiplication at a time: RoPE, grouped-query attention, SwiGLU, RMSNorm. It’s what changed between GPT-2 and a 2024 open-weights model.
“If you like this, it’s also worth looking at llama2.c[1], an implementation of the Llama 2 architecture in about 1000 lines of plain, dependency-free C, tokenizer and all.”
Alternative: Llama from scratch, or how to implement a paper without crying · Brian Kitano · https://blog.briankitano.com/llama-from-scratch/ · 672 points, submitted 2×. It’s as much about method as about Llama. “‘Use .shape religiously. assert and plt.imshow are your friends.’ Thank you.” (bravura, 2023). HN found a bug: “There is a bug : While in SwiGLU beta is a learnable parameter, in the reference paper the feed forward network set beta as a constant” (GistNoesis, 2023 · comment).
An inference engine written without libraries, then optimised step by step: why decoding is memory-bandwidth bound, KV caches, and kernel by kernel speed-ups.
Read first (one evening): Making Deep Learning Go Brrrr From First Principles · Horace He · https://horace.io/brrr_intro.html · 368 points, submitted 8× (2022–2026). Compute, memory bandwidth and overhead: the three things that decide how fast a model runs. “This post is a classic!” (ollin, 2026 · thread)
On a Mac instead: tiny-llm · Alex Chi Z · https://github.com/skyzh/tiny-llm · 299 points, submitted 2×. A course in LLM serving on Apple Silicon with MLX.
Percy Liang and Tatsunori Hashimoto’s course, with minimal scaffolding. You build a tokenizer, a transformer and an optimiser, then write a FlashAttention2 kernel in Triton, fit scaling laws, filter Common Crawl into training data, and do alignment. It’s the whole stack, at research depth.
“Despite having a decent foundation in deep learning, it took me several months to finish it using bits of my after-work hours and weekends.”
| Book | HN signal | Cost | Read it during |
|---|---|---|---|
| Understanding Deep Learning · Simon Prince · https://udlbook.github.io/udlbook/ | 433 points over 9 submissions | Free PDF; print is paid | Stages 2–4 |
| Dive into Deep Learning · Zhang, Lipton, Li, Smola · https://d2l.ai/ | 1,028 points over 26 submissions | Free online; print (Cambridge UP) is paid | Stages 2–4 |
“The field is bifurcating into two different skill sets: ML engineer and ML scientist (or researcher).”
How neural-net training fails silently, and the habits that catch it.
“This blog post is full of the kind of real-world knowledge and how-to details that are not taught in books and often take endless hours to learn the hard way.”
“Just look at your damned data to sanity check it and understand what’s going on.”
Checked against HN on 2026-11-08 (candidates_receipts.json), not used:
Not yet checked: Hugging Face’s courses, Stanford CS224N, Karpathy’s Intro to Large Language Models talk, Sebastian Raschka’s newer Reasoning LLMs from scratch.
phase1-stories/data/ml_candidates.json). Bare “transformer”, “tokenizer” and “LLM” were avoided as news noise.phase1-stories/scripts/check_track.py.tracks/ml/receipts.json (resources used) and candidates_receipts.json (all 33 checked).