Micrograd

Neural networks from scratch — following Andrej Karpathy's Neural Networks: Zero to Hero series.

What's Inside

Seven notebooks building up from a scalar autograd engine to a full GPT, each one implementing everything manually in PyTorch:

#	Notebook	What It Covers
01	micrograd	Custom Value class with automatic differentiation, computational graph visualization with Graphviz, training a small MLP from scratch
02	makemore	Character-level bigram language model, one-hot encoding, softmax + cross-entropy, name generation from 32K baby names
03	makemore_part2_mlp	Character embeddings (27→10-dim), 3-char context window, MLP with hidden layer (200 units), train/val/test splits
04	makemore_part3	Deeper networks (5 hidden layers), batch normalization from scratch, Kaiming init, activation distribution analysis
05	makemore_part4_backprop	Manual backpropagation through every operation — gradients computed by hand and verified against loss.backward()
06	makemore_part5_cnn	Hierarchical architecture with FlattenConsecutive layers, 8-char context, ~76K parameters, progressively compressing context
07	gpt_dev	Transformer with multi-head self-attention, causal masking, residual connections, layer norm — trained on Tiny Shakespeare (~210K params)

Project Structure

micrograd/
├── notebooks/
│   ├── 01_micrograd.ipynb
│   ├── 02_makemore.ipynb
│   ├── 03_makemore_part2_mlp.ipynb
│   ├── 04_makemore_part3.ipynb
│   ├── 05_makemore_part4_backprop_.ipynb
│   ├── 06_makemore_part5_cnn1.ipynb
│   └── 07_gpt_dev.ipynb
├── data/
│   └── names.txt              # 32K baby names dataset
└── requirements.txt

Setup

git clone https://github.com/aynursusuz/micrograd.git
cd micrograd
pip install -r requirements.txt
jupyter notebook

Requires Python 3.8+, PyTorch 2.0, NumPy, Matplotlib, Jupyter.

License

MIT