You made me order The Cuckoo's Egg. Luis Alvarez is my scientific hero since I read his memoir last year. Truly underappreciated in the pop-sci community.
I’m not convinced that useful mathematical foundations will be found anytime soon. Neural nets exist because we want to make decisions in a world that is so noisy, complex and chaotic that we can’t satisfy the requirements of more rigorous analytical frameworks. It seems to me that the irreducible complexity is in the real world, not the neural networks.
Never underestimate the power of mathematics. On the other hand, a lot of mathematical breakthroughs in the history are not found by people doing mathematics but in physics and engineering. Calculus, statistics, information theory, etc.
Even for computer science, take a look at Turing Award from 1966 [0], we will see how short sighted we are if we only follow the trend. Time will tell and smart people will find new path.
I have a background in mathematics, I believe in mathematics, but I don't believe in blind faith. Physics gave us statistical mechanics precisely because it's impossible to measure, model and predict the behavior of every individual particle in real-world systems. My gut feeling is that a mathematical theory of LLMs is more likely to look like statistical mechanics than something that tames chaos. That certainly doesn't mean that theory wont't open new doors though that we haven't currently thought of.
Reading this was a good reminder not to be intimidated by assumptions about complexity. (Without giving it much thought) I would have assumed that it would be hard to replace malloc for such fundamental applications as ls, but it's surprisingly simple.
There's usually an easy-ish way to override malloc/calloc/realloc/free on Unix, as it's very useful to do when debugging issues or just to collect allocation metrics.
In ELF objects (i.e. on Linux) this is usually done with the "Weak" symbol binding. This is an optional flag for symbols in ELF format that let you override a symbol by providing a competing non-weak symbol, which the linker will prefer when there is a conflict. https://en.wikipedia.org/wiki/Weak_symbol
You can see the list of Weak symbols by looking for a 'W' in the output of `nm` on linux hosts.
Unfortunately, a lot of system level knowledge like this is not found in a single place but spread over many articles/manuals/books/etc.
However, the book Advanced C and C++ Compiling: An Engineering Guide to Compiling, Linking and Libraries using C and C++ by Milan Stevanovic brings together a lot of information which you might find interesting.
That might be true for this particular thing, but there’ll still be some other perceived barrier of complexity, e.g. maybe it’s hardware, maybe it’s math, maybe it’s some higher level application like graphics. My point was that I was reminded to not just assume something would be hard without looking into it.
This applies to a lot of things unfortunately. There is a cult of just being afraid and scaring other people.
"You can't do it, just use a library.". "Just use this library, everyone uses it.". "Even google uses this library, do you think you are better." etc.
To add another example to this, you will read that memcpy is super mega optimized on libc and you shouldn't do it yourself etc. etc. etc.
But if you just check clickhouse [1] as an example. They implemented it, it is pretty basic and they say it works well in the comments of the code.
Also you can check musl libc code etc. and it is fairly simple.
People still would argue that you used some intrinsic so it isn't portable or you just benchmarked on one case so it won't work well overall.
Well you CAN benchmark as wide as you want inside your project and have a different memcpy code per project. This kind of thing isn't as bad as people make it out to be in my opinion.
Ofc memcpy is just an example here and it applies similarly to memory allocation, io etc.
As a negative note, imo this is one of the major reasons why most software is super crappy now. Everything uses some library -> those libraries change all the time -> more breaking -> more maintenance. Similar chain happens in terms of performance because the person that wrote that library probably doesn't even know how I am using the library.
This is also why people have endless arguments about what library/tool to use while they can be learning more and more things every day.
100% - the number of times you will need to use a super optimized memcpy() in real life versus the benefit you can get from looking at and writing basic versions of it for different CPU's is very slim.
Then you'll have a much better idea of when to _really_ use one that depends on intrinsics, is optimized etc, and how to benchmark them ... those are the real skills.
The killer feature of DTs is how fast they can be. I worked very hard on a project to try and replace DT based classifiers with small NNs in a low latency application. NNs could achieve non-trivial gains in classification accuracy but remained two orders of magnitude higher latency at inference time.
Also, decision trees (but not their boosted or bagged variants) are easy (well, easy-ish) to port manually to an edge device that needs to run inference. Small vanilla NNs are as well, but many other popular "classical" ML algorithms are not.
> It's always funny how many people think that the only font of altruism is taking care of children who have your DNA, like that's some kind of selfless act
This is a strawman position in my opinion. I don't think there's that many people who think they're carrying out some selfless act by having children. It's simply biologically true that the children you'll probably have the easiest time raising are your own and, assuming we want to continue as a species, we do need people to have children. It's fine to have them, fine to not, neither side has some moral high ground.
Personally I want to have my cake and eat it here. Tech has amazing potential to make the world a better place to live in and genuinely bring people together. The crowning achievement of AI so far to me is not Claude Code, it’s AlphaFold. I find the documentary DM released about developing it inspiring both as a technology story but also a team achieving things together that make the world better. I want to see more of that and hope I can steer my career in that direction.
Yeah, hopefully an outgrowth of this will be new amazing applications like that, that we never could've dreamed of before. I imagine "distributed services" will be "solved" by EOY, and the days of glorified CRUD app coders making 200K straight out of college are over for good.
But I think there will be new opportunities for people who are willing and able to learn. Entirely new fields will pop up and somebody will have to work on them. Most likely, the CS grads who are out of a job, or just frustrated and want to do something else.
So I don't think the opportunity to do innovative things and make a difference in the world is gone. But the opportunity to do so by typing code into a text editor may have breathed its last.
reply