DIY Transformer

A pedogical implementation of a decoder-only transformer from scratch.

---

Install dependencies:

python -m venv diyt
source diyt/bin/activate
pip install -r requirements.txt
pip install -e .

To be able to run jupyter notebooks:

pip install ipykernel
python -m ipykernel install --user --name diyt --display-name "Python (diyt)"

---

Experiment: Overfit to Harry Potter Corpus

Find data in data/harry_potter.txt

To train:

python src/diyt/pre_train.py

See notebook notebooks/playground.ipynb for generation.

TODO:

• Implement KV caching