It is lz4. While all that you say about disk being slow makes sense, measurements show that in this specific situation disabling compression improves performance by a little. My hand-wavy explanation is that freeing CPU to doing compilation rather than [de]compression is a better use of it, with all the uncompressed disk writes probably ending up in the system's file cache.
[build2 author here] Thanks for the feedback! Some additional details:
> The first one is a criticism of CMake, not Ninja (?), so I don't think it can be why.
Fair enough. The point I was making is that if you want to compete with Ninja, you cannot leave any potential performance gains on the table.
> The second reason given is doing some work like header dependencies in multiple threads. This is the most plausible reason to me but it still feels unlikely.
We are talking about ~2% performance difference here. Parallelizing even a small amount of work across 24 threads rather that doing it serially saving a percent or two feels plausible to me.
> There is some hand waving about file access patterns but I am skeptical; if the end-to-end build time is 3 seconds then the project is small enough to all fit in kernel caches.
It fits into the system's file cache unless there is memory pressure, like one would expect from having 24 C++ compiler jobs running in parallel. We actually measured this in isolation (with more detailed results in the linked article) and it has a measurable effect.
> They also mention doing other things like invoking the compiler to get version information. This seems like it would dwarf any performance gain from number 2.
I measured this, it costs 70ms or ~2% of the overall time.
> It fits into the system's file cache unless there is memory pressure, like one would expect from having 24 C++ compiler jobs running in parallel. We actually measured this in isolation (with more detailed results in the linked article) and it has a measurable effect.
300 TUs is not much. If they build in 3 seconds then they are trivial (small). If the machine is 24-thread (I assume some sort of heterogeneous 12-core), how little RAM does the machine have for the kernel to start evicting page cache during the build?
> > They also mention doing other things like invoking the compiler to get version information. This seems like it would dwarf any performance gain from number 2.
I measured this, it costs 70ms or ~2% of the overall time.
Couldn’t you amortize this to 0 by just caching the result and only changing it if the binary timestamp changed?
> Couldn’t you amortize this to 0 by just caching the result and only changing it if the binary timestamp changed?
Yes, that would be nice, but the tricky question is can any of this information change without the compiler binary mtime changing? First off, GCC's gcc/g++ binaries are drivers and are not what does the actual compilation, it's private cc1/cc1plus binaries that do the job. Can one of these change but not the driver? I think it's plausible (some package manager optimization where the file is not touched if it hasn't changed). So at a minimum we would need to discover where those are located (probably by invoking gcc/g++) and checking them as well. Could there be something else? Who knows. We value speed very much but we value correctness even more.
I think a more fruitful direction to explore is to improve GCC itself to dump all this information in a single invocation and in a machine-readable format (JSON). I think if we go from 70ms to 14ms (and perhaps even lower because this special GCC mode could conceivably do things faster than how we do it now), it would be good enough.
I don't know, someone who solves their borrow checker errors by adding more dynamic allocations (which seems to be common/recommended), will be taught how to program in Java, not C++.
That's only recommended for beginners while they get tuo to speed (to try and help them avoid ahve to learn too much at once). It's absolutely not recommended in general. The whole point of the borrow checker is to enable memory safety without heap allocating and garbage collecting everything.
> Commands in a CLI … I think what makes for good product design is having a very small number of primitives.
Small but not too small. Case in point: shell scripts (POSIX shell, bash) where the scripting part was decided to be modelled as commands thus not introducing another bunch of concepts. We all know what the result is (hot, slow mess).
I know it's in vogue to bash Bash but I feel that criticism is unfair.
Shell scripting is a victim of its own success: it is _so easy_ to get started that most users get value out of knowing the first one percent and never bother to actually learn the rest.
There aren't many who have read the Bash manual, or know what zsh can do that Bash cannot, etc.
"Shell scripting is a hot, slow mess" is the same hot slow mess that you get wherever the barrier to entry is extremely low (e.g. early PHP, early JavaScript/frontend development, game development with a game engine where you can just click around in the editor, etc).
There’s also the fact that shell scripting is for automation of what you may do interactively. It’s not for stuff where you want data structures to manipulate in memory. Trying to use it like python is an exercise in frustration.
I remember a friend saying they had a university assignment to implement something like a spreadsheet program in bash - I suppose it was to teach them the intricacies of bash but the big takeaway was meant to be - don't use bash for anything too complicated.
GCC (libstdc++) as all other major C++ runtimes (libc++, MSVC) implements the small object optimization for std::function where a small enough callable is stored directly in std::function's state instead of on the heap. Across these implementations, you can reply on being able to capture two pointers without a dynamic allocation.
You would think so, but it actually doesn't. last time I checked, libstdc++ could only optimize std::bind closures. A trivial test with a stateless lambda shows this is still the case in GCC14 and 15. In fact I can't even seem to trigger the library optimization with bind.
Differently from GCC14, GCC15 itself does seem to be able to optimize the allocation (and the whole std::function) in trivial cases though (independently of what the library does).
The strangest thing about Flow is that its compiler is implemented in C#. So if you decide to use it in your C++ codebase, you now have a C#/.Net dependency, at least at build time.
It’s also funny because it’s a small, incomplete, incompatible subset of c++… seems like a perfect LLVM / clang rewriter case too, it would be easy to convert and be pure c++. Hell even a clang plugin to put the compile time into one process wouldn’t be awful. But i wonder looking at the rewrites if there’s not a terribly janky way to not need a compiler, if at some runtime cost of contextual control flow info.
I believe the point is if something is UB, like NULL pointer dereference, then the compiler can assume it can't happen and eliminate some other code paths based on that. And that, in turn, could be exploitable.
Yes, that part was clear. The certainty of a vulnerability is worse than the possibility of a vulnerability, and most UB does not in fact produce vulnerabilities.
Most UB results in miscompilation of intended code by definition. Whether or not they produce vulnerabilities is really hard to say given the difficulty in finding them and that you’d have to read the machine code carefully to spot the issue and in c/c++ that’s basically anywhere in the codebase.
You stated explicitly it isn’t but the compiler optimizing away null pointer checks or otherwise exploiting accidental UB literally is a thing that’s come up several times for known security vulnerabilities. It’s probability of incidence is less than just crashing in your experience but that doesn’t necessarily mean it’s not exploitable either - could just mean it takes a more targeted attack to exploit and thus your Baysian prior for exploitability is incorrectly trained.
But not in reality. For example a signed overflow is most likely (but not always) compiled in a way that wraps, which is expected. A null pointer dereference is most likely (but not always) compiled in a way that segfaults, which is expected. A slightly less usual thing is that a loop is turned into an infinite one or an overflow check is elided. An extremely unusual thing and unexpected is that signed overflow directly causes your x64 program to crash. A thing that never happens is that your demons fly out of your nose.
You can say "that's not expected because by definition you can't expect anything from undefined behaviour" but then you're merely playing a semantic game. You're also wrong, because I do expect that. You're also wrong, because undefined behaviour is still defined to not shoot demons out of your nose - that is a common misconception.
Undefined behaviour means the language specification makes no promises, but there are still other layers involved, which can make relevant promises. For example, my computer manufacturer promised not to put demon-nose hardware in my computer, therefore the compiler simply can't do that. And the x64 architecture does not trap on overflow, and while a compiler could add overflow traps, compiler writers are lazy like the rest of us and usually don't. And Linux forbids mapping the zero page.
What would be the SQLite's equivalent to indexing starting from 1, not 0? Off the top of my head I can't think of anything that would go so much against the grain.
