Isn't Fable intentionally trained and system prompted to act maliciously and attempt to sabotage third party attempts to use it to train or improve other LLMs?
The only (still in prototype stage!) "competitor" for those GB10/Ryzen Al Max+ 395 (in my region, borderline unobtainable) systems seems to be the Xiaomi AI Cube.
- waitpid for all process in the other process chain?
- propagate/handle signals, like for example SIGINT/SIGTSTP/SIGPIPE/SIGHUP forward and back signals?
- change stdio buffering mode?
- remember to count how many bytes actually were written by write, and blocking if not, before clobbing the 64kb of the pipe buffer size with another write?
- flush, then close all file descriptors left behind by the pipes when it ends?
It's for reasons like that, that I don't trust anything non-trivial, not-shell to use pipelines correctly.
Duckdb supports loading and saving to zstd for all it's base loading/saving formats csv/tsv/json/jsonlines, but, for good or bad, those are solid compression.
Under most r/w workloads, using parquet/lance/vortex/native-duckdb, with their built-in columnar compression will result in more performance AND space savings. Non-solid compression. Then, the query engine can push down your query predicate to a column row group level, instead of forcing it to decompress the entire dataset to operate.
Practical example: duckdb has syntax - https://duckdb.org/docs/lts/data/multiple_files/overview - to glob multiple files at once, but that really only works if you're applying push down query predicates instead of re-decompressing your entire data set per SELECT. I would say for most dataset, even 20%+ size is worth not having to decompress (or even download!) the entire dataset, to figure out if something fits the predicate.
After all, if you have to download and decompress the dataset back again to operate, then the "space savings" are gone.
Using macos on low memory regimes will make it use disk-based swap.
For example, a Macbook Neo (so in theory, something with around 4GiB of free RAM lying around) might eat around 900GB of writes a day while not doing much at all, because it's basically on low on RAM and swapping all the time.
A NAT implementation could broadcast any "WAN" side incoming packets to all link local clients (aka: put everyone in the DMZ). The only thing preventing that is a stateful firewall.
If you're using specifically zstd, on the sender side, instead of tweaking the whole tunnel once, you can use --adapt to dynamically adjust to i/o conditions.
reply