Good article. As I reflect about it, I genuinely wonder (not in jest) whether pangram software uses AI to generate code? Secondly, what patterns do they look for in the text? Genuinely curious.
As an aside, for someone interested and who's an absolute beginner, can someone please recommend good resources on how to build LLMs from scratch? Thank you in advance.
I second 1. I'm a newbie in neural networks and I think it's an excellent book! One of my barriers in ML is the resources, I find them overcomplicated or too simplistic without a mid term. It's not the case of this book, everything is well-explained. Neural Networks aren't fun for me, but this book makes it very interesting.
I think this is the best structured class out there that teaches how to scale LLMs . Hope the 17 year old knows linear algebra. Building an intuition for the shape of the matrices is important. A lot of understanding the 'building from scratch' means understanding choices like why RoPE instead of the original frequency based positional encoding. Start with Karpathy and then go to CS 336
For last few months, whenever I am trying to understand a difficult problem or a concept or addressing a complex issue at work or in relationships, I try to write one page by hand. Often it brings clarity or surfaces the core confusion/misunderstanding. I don't know why that works but it has been really useful.
I feel LLMs are indeed getting better at writing models. But, in my experience, they struggle to come up with correct safety and liveness properties unless you closely work with them. And of these two, they struggle the most with correct liveness properties.
Also for some problems I observe that models produced by LLMs often cause state space explosion. For simpler models they can fix this when you guide them though.
I’m sure LLMs will get even better.
That said, I take slightly different approach. Lamport said “If you're thinking without writing, you only think you're thinking.” So taking that advice I always try to write the first draft with hand and once I have the final shape in place I then turn to an LLM for further exploration and experimentation if I have to.
> And even at the end, when asked what he would like to do, he said: "I would like to learn something."
Don't have an exact word to describe how I feel after reading above. Find it beautiful that such an accomplished person wanted to learn something even towards the end of his life.
Inspiring and well written. It resonated with me for I find myself in a similar position. I wonder how much time did the author commit on weekly basis. Nonetheless, I wanted to signup on Math Academy immediately but doesn't look cheap.
Are there any other recommended websites for learning math (apart from Khan Academy, Math Academy)?
Hey! This is my blog post thanks for reading! At my peak I spent roughly 4h a day on math academy because I wanted to get 100+ XP. I've brought it down to about 2h a day since I'm also teaching myself python for my goal of being a MLE in the future.
Thank you! I initially included the illustration and explanation in the first draft but decided to remove them in favor of the full trail. Looking back, I think that was a mistake. I’ve now added back the illustration along with some explanation.
You're right, I could have phrased it better to something like:
"If we run P and Q concurrently with ‘n’ initialized to zero, what extreme interleaving could result in the lowest value of 'n' when the two processes finish executing their statements on a model checker?"
tbh I think zero is a completely reasonable result for a general thought-exercise like this. sleep-based "memory fixes" are quite common in practice, and there's no synchronization at all in the sample.
though there's a lot of fun in here if you allow for optimizing compilers and lack of synchronization. e.g. without explicit synchronization between P/Q and the observing thread (assuming it's the main thread), it's reasonable for a compiler to simply delete P and Q entirely and replace the whole program with `print(0)`
reply