All talks
Meetup talk LLM inferenceLive coding

NanoGPT-inference: How to build LLM inference from scratch

KV caching, sampling kernels and CUDA graphs, one step at a time

February 11, 2026 · AI Tinkerers Ghent AI Tinkerers Ghent Aikido Security, Ghent ·Event page

Abstract

Most of us call an inference server and never look inside it. In this talk I started from Karpathy's nanoGPT, which recomputes everything for every new token, and added the optimizations that real inference engines rely on, one at a time. After each step we looked at what happened to throughput, time to first token and memory.

AI Tinkerers has a strict format: fifteen minutes, no slides, live code only. So this was a walk through nanoGPT-inference, a small fork of nanoGPT with a few hundred lines of extra inference code. Everything runs on GPT-2 weights, so you can reproduce the benchmarks on a single GPU.

Technical talk, in English. Assumes you roughly know how a transformer works.

Walkthrough

What we built, step by step

  1. 01 Baseline. The original nanoGPT forward pass, which recomputes attention over the whole sequence for every new token. Even with FlashAttention, decoding is compute-bound almost immediately.
  2. 02 KV caching. Keep the key and value tensors of every layer around and only run attention for the new token. Decoding stops being compute-bound, in exchange for memory that grows with sequence length and batch size.
  3. 03 Faster sampling. Naive top-p sampling sorts the full vocabulary distribution at every step. FlashInfer's rejection-sampling kernels skip the sort with inverse transform sampling and a pivot, all in a single fused kernel.
  4. 04 CUDA graphs. Capture the decode step once and replay it, which removes the overhead of launching every kernel separately. This needs a pre-allocated KV cache and masking instead of dynamic shapes, and that extra bookkeeping shows up at larger batch sizes.
  5. 05 Measuring it. Decode throughput in tokens per second, time to first token in milliseconds and peak VRAM, benchmarked per engine and batch size on an L40S.

From the room

Photos

Walking through the nanoGPT-inference code at AI Tinkerers Ghent.
Walking through the nanoGPT-inference code at AI Tinkerers Ghent. · Photo by Robin Wijnant
A full room at Aikido Security for the Ghent meetup.
A full room at Aikido Security for the Ghent meetup. · Photo by Robin Wijnant

Resources

Further reading

Code

nanoGPT-inference on GitHub

KV caching, sampling kernels and CUDA graphs on top of nanoGPT, with a benchmark script

 Related

nanoGPT by Andrej Karpathy

The model implementation this repo forks

 Related

FlashInfer: sorting-free GPU kernels for LLM sampling

The rejection-sampling kernels used in the sampling step

 Related

llmq

My library for distributed LLM inference