"As the code base improved, kdb+ releases became faster – and smaller."
This is what I always strive for. Although I've only met Arthur a couple of times, he has been a huge influence on me as a developer. Besides some shared personal interests, his development philosophy that simplicity and consistency produces performance and malleability (to be able to constant add, subtract, or write).
He's on the top 3 developers I try to model myself after. While he's a god among men, his products (A+, K, KDB, Q, and now Sakti) make others mortal also gods among men by just learning to think and use his tools. My co-workers I'm sure get annoyed at my code-reviews ("delete this", "rthisewrite fir first with that then delete the second", "delete all this and return this",. ...) where I consistenly call for deleting a remodeling code to be as minimal as needed. I'm brutal about it - as I learn from K and it is partially Arthur's fault (and another dev I look up to). Maybe now they know where I picked part of it up.
https://shakti.com/ is his new distributed database project that is still somewhat hush hush, (and he not the kind of person that plays buzzword bingo - so I assume he has something special planned). I cannot wait to see how that turns out. I don't know if he's ever produced a bad product.
Remember that as a reviewer, you always have an advantage over the coder: as the coder, you need to crack the problem and find an algorithmic solution. It's easy to have a couple of false starts with this. These do stick around through iterations and changes. Eventually, you submit something that works, but you're overlooking that the code could probably be less convoluted in places, because you know the solution and it is perfectly reasonable.
The reviewer who gets to read the code gets to walk the shortcut to the solution after the trail was blazed for them: they get to think purely about the working solution without the intermediate steps. And they naturally will find places where the original developer was thinking in terms of his failed earlier attempts.
I bet that if I gave you a non-trivial problem to solve in a reasonable amount of time I could do the same kind of review on your code, forcing you to shorten it. And I will find something. Guaranteed. That is just how these iterations work.
I bet that [..] I could do the same [..] And I will find something. Guaranteed.
Why are you framing this as though it's adversarial and the parent comment is stating some kind of superiority, and you're here to turn it around and put them in their place?
Isn't teamwork all about doing better than one person working alone?
My take on the parent comment is that it has some adversity and feeling of superiority to it. And I don't like that. I agree that in the end it is not a competition, I didn't think that I would have to point it out.
In fact I am in the middle of fixing up code that I wrote as novel research because the powers that be want it in a production system. The code review I got was detailed and - frankly - embarassing at times because all the uncertainties and meanderings of research had me change pieces of code over and over and over again to a point where remnants of old approaches didn't stand out to me anymore and where I had at times built on top of bad decisions because they were meant as a temporary hack and experiment, but worked well enough and stayed in. And by the time I submitted the code for review I had become blind to these mistakes. I know how to write good and clean code and I get through code reviews easily when I know what I am doing. Not this time, though.
APL and the culture surrounding it is what you get if you assume that programming languages are, like human languages, meant to be learned and fully exploited, unlike the lowest-common-denominator dumbing down that's more common in mainstream programming languages. I find it very interesting how much people can learn if they try.
I don't like the fact that it is closed source but not having source code does not stop me from utilising it on a personal basis. I think it makes sense to use it in this capacity and learn the language as best one can in the event that an open source option besides kona does someday become available. k is one of the very few closed source programs I use.
It is interesting how the closed source issue is consistently raised on HN with respect to k when there are numerous other examples of closed source software used by HN readers where I never see this issue raised. In those cases it does not seem to be important. For some reason however, it seems availablity of source code is important to some folks in the case of k.
I am curious; What would you do with the source code? Is it an issue of verifying what the software is doing, or perhaps being able to modify the software or make "fixes"? Is it something else? Maybe you simply want a free option for commercial use. If the later, please disregard my curiosity.
For me the issue I have with k being closed source has always been ports, i.e., limited OS support. They used to have a FreeBSD port many years ago, but today there is no support for OS other than Windows/Solaris/Linux, e.g., NetBSD.
I'd never use a closed source programming language because it makes me dependent on the vendor. With FOSS, if the original company goes away I can pay someone else to keep the software running on a modern computer. With closed source software I don't have that option.
Not disagreening with you, but in addition to the more recent versions, I run an old version of k from around 2000, and it keeps running even with the latest kernels and hardware, thanks to availablity of legacy Debian libraries.
At some point in my career I was forced to use a closed source compiler from the early nineties. With each new version of Windows it got harder to keep it running. We also discovered bugs in the compiler and had to rewrite our code to avoid them. I quit that job eventually, but I imagine that in one or two more Windows iterations the developers will be forced to compile inside a VM.
Having code be free and open is for all the reasons you mentioned. Additionally, my thinking is that if you want wider adoption you will need to be open. Building a community around a language or platform starts with the "hackers" who are curious about the code and want to contribute.
The guys who license kdb/q probably make more money today from service/consulting agreements and not the software licenses. So it's to their benefit to go open (a la MongoDB)
Also, if you're building a tech stack from scratch - you want it to be "clean". You don't want to muddy it with expensive/onerous software licenses.
> The guys who license kdb/q probably make more money today from service/consulting agreements and not the software licenses. So it's to their benefit to go open (a la MongoDB)
k (specifically k3) runs on FreeBSD if you use linux emulation. The same is true of q. IIRC, there are no limits on the 32 bit version of q (unlike the old k3). kona is more portable but needs work.
I have an old license for a 2.x version which removes the workspace limits.
/emul/linux is how I use k on NetBSD. However I hesitate to rely on Linux emulation because it does not seem to be something that is important to current BSD developers. If I am not mistaken, both OpenBSD and Dragonfly have removed it.
Since K is supposed to be very small, the mystery is why no one has looked at the disassembled code of the free version and copied the optimizations to elsewhere.
I too wish we could get rid of all the bloat and use the marvelous machines at our fingertips to their full potential. But I seriously doubt that you can write a bloat-free OS that runs on multiple brands of laptops with a bloat-free webbrowser that allows me to read most websites with a small team in reasonable time, no matter how brilliant the people involved are.
As others have mentioned, a bloat-free web browser is impossible currently because the web itself is far too bloated.
However, I'm not convinced that our OSs can't be much, much simpler. You can, almost reasonably, use DOS as a daily driver still. I have very fond memories of DOS because it was simple. Even as late as the early 2000s, if you wanted something like a MAME cabinet you'd often use DOS. You just plop some application and data folders onto a disk with like 4 system files and you were pretty much done.
Where is today's DOS? I suppose it's Linux, which is orders of magnitude more complicated (especially once you start talking about a GUI) and even Linus admits it has become quite bloated. I personally think we can do a lot better and I've been toying with the idea of putting something together because I'm just so sick of modern computing's bullshit.
PS: If anyone is aware of an organized effort towards this goal that already exists, please let me know.
Goal: a system where you can fit the whole stack in your head, yet participate in a networked world.
Approach A. Outline a reference hardware platform. Port NetBSD or minix3 or plan9. Then simplify. e.g if unix, get rid of users and groups, get rid of x, get rid of package mgmt, get rid of nfs. You could bootstrap this in qemu.
Approach B. Establish an alternate browser. Something like gopher, but with async events between the 'page' and the user. This allows for chat and form manipulation. You can send non textual content through this async link - video, audio. Codecs are a trap, they lead to pkg management and complexity. How to stop proliferation?
I think Casey hit the nail on the head here. We used to have hardware with standardized ABIs, some of which had extensions (like Tandy graphics compared to CGA). That made it relatively trivial to make OSs, which is what PC booter games did really. DOS itself barely abstracted anything at all, because you didn't need drivers for standardized hardware ABIs.
Unfortunately the only way to deal with that is to give up on the idea of running on a majority of existing PC hardware and target only a small subset of commodity hardware. Ideally it would all be open, to ensure it isn't going anywhere for a while, but sadly there is no such system.
I'm glad to see I'm not the only one who thinks Users and Groups are the wrong abstraction for systems like this.
I've been playing around with this idea myself. My solution is to isolate the "stack of shit" that is still useful into VMs or separate processors entirely. The USB stack is already implemented on microcontrollers. If you only want mass storage and HID keyboards you can implement your own simple protocol. TCP/IP, again, plop it onto a microcontroller and create some simple file interface for it. The browser, um, well, put it in a VM running linux... or really want I would rather do, put it on a VM in the cloud and make the big computer in the cloud do all of the hard work and just send down a simple description of the page, maybe using a protocol like, well, I don't know, Postscript?
1. Package managers need not be complicated. I think a simple sandboxing app mgmt is possible.
2. about web: too bad that design-addict webdev community bloated it. But wish someone make a simpler, faster & straightforward web as portable app platform with few nice commercial qualities...
3. About OS: OS need not be bloated if it isn't running behind popularity by satisfying everyone's remote needs. ( source: plan9, suckless, busybox )...
1. Package managers only exist to manage complexity. If you want one, the system is already too complicated. You can sandbox applications without them.
2. No disagreement here.
3. Unfortunately it still kinda does, because drivers. Two of the ones you listed sit on top of the Linux kernel, which even Linus thinks has become bloated.
Unfortunately I have to agree with this precisely for the reasons outlined by Casey in the parent's linked video. The shift from standardized ABIs to driver-abstracted hardware has ballooned the code required to work on "a PC" geometrically. It's basically impossible to catch up to established OSs now.
An alternative would be targeting a single set of hardware that's likely to be around a really long time. I'd be ok with that and something like a Raspberry Pi, but only if all the hardware was open. Of course, then you're in more of a 68k Mac space where hardware choices are extremely limited.
As someone who had to develop a DOS application for a job within the last ten years, let me say emphatically, no, DOS is not even remotely suitable as a daily driver today.
If you want simple nowadays, boot a Linux kernel into busybox. It'll boot almost as fast as DOS but you'll have all the crazy modern amenities like 64 bit support and network drivers.
You don't even need a package manager. You could just unpack a GCC binary and compile all the junk you want yourself.
I'm curious what kind of use case you have in mind for this OS you're envisioning. I'll admit I can't really figure out either what you're trying to describe.
I actually like AppImage, even if it is slightly over-engineered compared to simple appdirs. It's a shame more developers don't actually publish them, and that not a single file manager that exists supports displaying their icons.
Still, AppImage is an attempt to bring sanity to the garbage fire of the Linux Desktop. I'd rather have a system that wasn't a garbage fire to begin with.
How does an AppImage handle shared library dependencies? Do multiple independent AppImages share the same mmap'ed library or do they each occupy their own in RAM? If there's no way to dedup the same dependency code between apps then memory usage can grow out of control pretty easily. Package-based distributions concern themselves with this a lot...
I don't disagree with you at all on principle here. What I'm genuinely curious about is if this approach is akin to "compile everything statically?" I'm assuming that, if so, the issue of memory use has obviously been raised and the response is akin to "memory is cheap now" so it's a bygone concern. That may all be true and totally worth it. I just feel compelled to ask having been accustomed to the shared memory mapping model for decades...
I think a bloat-free web browser that works with most existing websites is a contradiction in terms. You could have something fairly simple that did the _job_ (jobs?) of a web browser, but it would not be broadly compatible. You could write something broadly compatible, but it would be intrinsically, inescapably very complex.
Maybe if we all thought like Arthur we'd just use gopher and be able to write a new client or server from scratch in an afternoon.
Let's add a couple more constraints. The OS and the applications that run on it should be accessible to people with various disabilities and should support a variety of human languages for both input and output. And there are probably more things I'm not thinking of that any widely used platform must have.
To be sure, software does have some bloat. But some of the complexity is necessary for software that can be used by the whole world.
Google are (only) replacing the kernel. Which I think is the right approach - identify the bits that are holding you back (in this case holding you from taking full control of the os) and replace only those with ones that are as compatible as possible with the existing software.
I agree that's important, but I don't think that a great OS should become just a good OS in pursuit of that goal. After the OS is made, it seems like there would absolutely be ways to make the system accessible as-needed.
As a handicapped person: The internet is my accessibility.
Most of the standard conviences available to "normal" people because smartphones are a thing and yadda is how I make my life work.
Among other things, I also co-own a tiny list for blind developers. So I have some familiarity with the fact that there are people who actually need accessible design in a way that I don't.
But I think any internet that works well dramatically improves life for the vast majority of the 60 percent of people who have varying degrees of impairment, two-thirds of whom do not self identify as disabled or handicapped.
Cars were invented for people with sufficient use of their limbs and eyesight. After their invention, the control systems were adapted for people who, for instance, can't use their legs. I don't know of any way they have been adapted for e.g. blind people. And yet, even though they are not accessible to a large fraction of disabled people, the world is better off with cars, designed for not-disabled people and then adapted as far as possible, than without cars.
That's all I'm getting at. Its too much of a constraint on invention, but likely to be adaptable later.
I think you and I may have different definitions of a great OS. To me, a great OS should use the machine efficiently, yes, but it's more important for it to be usable by all people regardless of factors beyond their control.
Really? All? And if not all, are we back to the ADA reasonable accommodation standard? What bugs me about the "accessible or bust" people is that they will deprive 99% of the population of something because 1% can't have it. There's some on-going case about university courses in California of some sort being publicly released but "not being accessible". So now, no one can access it because they got taken to court. It costs zero dollars (or very few) to publish existing material to the world; it costs quite a bit more to add stuff to it.
Maybe a great OS should be available which can run quickly without accessibility features, but slower if they are explicitly installed by an end user. Like it or not, it's a balancing act between how much we're willing to give up and for how few people.
This has been feasible and practical several times in the recent past. Serial terminals, Thin clients, X server thin clients, PC Anywhere, VNC, Windows Terminal Server, Citrix whatever it's called, Virtual Desktop Infrastructure, steam link, etc.
Office productivity applications have a lot looser latency requirements, and tend to change the screen less frequently than games too. But current solutions are capable of playing videos and some gaming.
Let's not forget that there are hacks like old mainframe apps that get accessed through web browsers where the web server is literally just a screen scraping wrapper around the legacy application. Inelegant? Fragile? Crappy to maintain? Yes. But it works.
You don't need to involve streaming, though that is an option. You could do like we used to do with some personal computers and just have an add-on board that ran some other arch and OS. Between that, virtualization, emulation, and WINE-like compat layers you should be able to run just about anything.
Well, we already have a database like that. So why not a browser or an OS.
Unfortunately, there's only one Arthur Whitney. Other people can emulate his style and genius, but I don't know anybody else who brings this kind of ruthless minimalism to everything they do -- and make it work so well.
What, precisely, do you find unappealing? Is it unappealing to have to write so little code to accomplish the task? Is it aesthetically unappealing to you? How much of that comes from it looking unfamiliar? If you spent a few days working with code like this, would it seem less strange? A month?
I'm a python guy myself and I love the APL languages. In my mind I put python and APL in the same category of: I spend more of my time thinking about what to write and less time actually banging out text in the editor.
Is he saying k is a lisp? I'm sure there are important differences here, but I don't know enough to say what they are. (There is a question implied there.)
the new project. Arthur has a tendency to turn mid stream sometimes. I'm sure much of that code was used in this new DB he's working on (db's and os's are quite similar).
Besides Whitney being being very good at finding and sidestepping bottlenecks (like writing his own memory management because the OS one is too slow), the entire interpreter, db engine and 3rd party software fits into L1 or at least L2 cache. This is a marketing blurb saying that as well https://kx.com/discover/in-memory-computing/ . But when you read about and try out Whitney software you will see that terseness in every way does makes things faster.
I've heard about the cache thing, but that doesn't explain much. How is it different from machine code or VMs? Does it remove the need to read program's data? How much time does a non-K program usually spend moving the code from memory? Why wouldn't generations of programmers make the same optimizations―or just copy them from K? Or what are the tradeoffs, considering that K programs are essentially bytecode?
Also, there are plenty of memory allocators to choose from, and every runtime that doesn't use the libc has to ship its own (if it does memory management, of course). The OS only gives you big chunks to use as you please, and you can't sidestep the syscall. So that's not a super novel thought.
It's very optimized for in memory use, so assuming the db is all in working memory and the code operating on the data is in L1/L2, and for the type of data in the column store (Kdb+), you usually pull related chunks into the Lx data cache on which you filter. I would say k is fast for the use case where column memory based storage is a good fit, so relatively small data sets that live in memory and where the operations on the data fit column stores well; I would bet almost all k programmers (including myself) mostly have experience with those datasets or have shoehorned less suitable cases into that case because it did not perform otherwise.
There is obviously no magic, besides, for many 'new' programmers and for current standards, the incredibly small binaries for so much functionality. And that helps with always staying in cache vs almost every other software you will use these days where there is continues chunks of instruction data going to and from cache. That's why C (and maybe Rust some time in the future but the binaries were too large when I last checked) is still important if you really want to get to that point.
Ofcourse in that 200something kb k/kdb+/q binary there is no room for much optimizing, that is why it absolutely falls down for cases it was not optimized for. And when you use a lot of k you know how (and automatically do) to shoehorn basically anything in those cases.
It is not magic by any means, but it comes closer to magic for many as it is alien like assembly compared to JS or C# or something like that I guess. And the speed comes from keeping to your niches.
> Also, there are plenty of memory allocators
As I understand it, he did it because he did not find the Windows memory allocator efficient enough specifically; not sure why he didn't check for other allocators; I guess because he was optimizing a runtime, he just rolled his own instead.
This is what I always strive for. Although I've only met Arthur a couple of times, he has been a huge influence on me as a developer. Besides some shared personal interests, his development philosophy that simplicity and consistency produces performance and malleability (to be able to constant add, subtract, or write).
He's on the top 3 developers I try to model myself after. While he's a god among men, his products (A+, K, KDB, Q, and now Sakti) make others mortal also gods among men by just learning to think and use his tools. My co-workers I'm sure get annoyed at my code-reviews ("delete this", "rthisewrite fir first with that then delete the second", "delete all this and return this",. ...) where I consistenly call for deleting a remodeling code to be as minimal as needed. I'm brutal about it - as I learn from K and it is partially Arthur's fault (and another dev I look up to). Maybe now they know where I picked part of it up.
https://shakti.com/ is his new distributed database project that is still somewhat hush hush, (and he not the kind of person that plays buzzword bingo - so I assume he has something special planned). I cannot wait to see how that turns out. I don't know if he's ever produced a bad product.