This post does a great job at showing that the core idea behind GPT is relatively simple.

To do something useful it needs tons of data and then everything starts becoming more and more complex to deal with

This is so cool. Andrej has been single-handedly making the world of llms more interesting and democratized

You can even strip out the autodiff and write an explicit backwards pass, training time went down from 40s to 5s for me.