"The solution is probably to be more selective what you're open sourcing if you're planning to build a business."
Fully agree, FOSS maintainers are - in the grand economic scheme of things - beggars high on altruism. The whole model of FOSS from the position of a non-corporate sponsored developer was already broken and with the advent of AI is now most probably broken beyond-repair. Buy-me-a-coffee with a <0.01% hit rate compared to open feature requests/gh issues on some of the more popular projects out there, all while companies like atlassian - using FOSS to run their business on for free - can charge 50k EUR for a single 1h migration call. Time is the most valuable resource one can waste, something you get to realize once all the mental gymnastics wear off(either by itself or due to less fortunate life situations)
..a strategy valid in the good ol' days when a complete rewrite of the currently non-licenseable non-protectable underlying desings and ideas would not cost you 200$ on tokens to reimplement in a different or even in the same language.. maybe SW patents will be a thing soon .. and inadvertently become another nail in the coffin of FOSS and SW development in general, who knows
I would be really grateful if someone could battle-test my frankendb in a full-fledged RAG setup(lmdb + roaring bitmaps + to-be-removed lance with a bitmap-based virtual fs-like tree on top of your data) outside of its original narrow use-case (index for user data + workflows)
Singlefile based web pack+download is exactly the next feature I need besides indexing URLs / browser tabs with[0], have it on my to-research for a looong time. I do have a server-side script that can be triggered on doc ingestion as a hook "if data/schema/tab is indexed and path is /ml/* then run script "fetch-url.sh" and file the result into {{doc.path}}" but being able to do that in-browser is more user-friendly.
I'm hacking on a in-process DB on top of LMDB+Lance(for now, hilbert space kung-fu with a custom matryoshka embedding setup with separate spatial + temporal + internal and content derived anchors will replace that ~last-century~ last-year tech) + roaring bitmaps as the primary indexing engine with the same or similar goal[0] and will definitely deep-dive into yours.
I'm also trying to index users unstructured documents and workflows(tabs, emails, files, notes, identities etc)
- Organize them into semantically meaningful user or agent created context or directory-like virtual trees (the same photo of a nice kitchen may be surfaced under `/travel/barcelona` and `/arch/interieour/kitches`)
- ..where tree nodes are mapped to bitmaps - `/travel/barcelona` does a fast and cheap `travel` AND `barcelona`, want to "zoom-out" you just go one directory up to `/travel` and see all documents tagged with travel)
- You can use multiple timelines - extract that fancy md-converted en-wiki hf dataset into a wikipedia db dataset + timeline, tag your personal timeline as "personal" - wanna know the zeitgeist of your grandmothers birth date - search for it with timelines personal + wikipedia in layered mode and you'll get everything that happened or was happening during that time.
- You can have long-running stateful query sessions and refine your searches dynamically - search for "winter" and get all documents with a winter scenery or mentioning winter - refine with "nice view" then "laptop" - citing a recent example[1]
- Documents have relations that would be cumbersome to map in a virtual tree structure(worth an experiment due to the zoom-in/out you get with context bitmap trees though) - hence on top of the initial structure you can use graph edges(also powered by bitmaps - as most indexes are)
- All vector queries always run on top of a candidate set you get by the bitmap/bitmap-based filter algebra hence searching through 100k+ docs is usually pretty fast
This world of cheap (vc) money, huge private equity sells and high revenue early in the products development phase is so distant to me(geographically and even more so, mentally), I need some help deciding what to do with my project and I'd really appreciate if you could DM me
I am building my "dream tool" too, a cross-device roaming profile-experience context-focused UI. Many, many iterations that go back maybe 15y, even built a custom linux distro (iolinux) back in the day(ro root, containerized user-spac with composer-like app handling). Always worked on side-projects, consulting gigs or one-off implementations, often times across multiple customer-devices. Switching context is damn expensive, hence tried to build a tool to help with that. Always failed on the UI parts - native linux tooling was just awful. What do you specifically mean by "context" you ask? Well, you get a call, switch to work on `work://customer-foo/project-bar/task-baz`, every bound device / tool / application loads data related to that task (your filesystem mount shows relevant files, browser loads relevant tabs, obsidian relevant notes etc). Building my own small db engine to accommodate that use-case, getting more fun working on it every day
I divide my unstructured universe into workspaces, each workspace is a self-contained shareable/moveable soon-to-be-containerized runtime running an in-process LMDB with roaring bitmaps and (for now, S2-like hilbert space projections will replace it soon) lancedb for some basic cossim search. You configure your data sources per Workspace - imap, files, s3, soon slack(bc I use it), on top you create a classically-looking virtual tree. That tree is made of bitmaps - /work/customers/acme org/projects does a logical AND on all bitmaps in the path (work AND customers AND..) - if you zoom out to /work/customers you see all data for all customers, if you'd create /work/projects - you'd see all data tagged with any project. On top of that tree sit contexts - think moveable db views. You create a context "acmeorg" and bind your browsers, cli(for dynamic dotfiles), notes apps etc to it. You can even mount a file representation of all indexed objects via a fuse driver. When you point acmeorg at /work/customers/acme org/projects/foo everything moves with you - your browser stashes current tabs and opens relevant ones, your fs mount shows only relevant content etc - moving to projects/bar does the same, if you do that via commandline ctx set ../projects/foo --update-dotfiles - you get your dotfiles
You can setup actions - if I send a browser tab thats a yt url into home/music download it and place it into my roaming profile workspace Home, if I index a arxiv website, hit the library agent to properly assign it to existing tree paths and write a summary
There is also timeline support - you can have a personal and wikipedia aaand historian foo timeline, wanna know the zeitgeist of when your grandma was born - tick extra timelines and query in layered mode
Ou, and there is real-time streaming support, your db can resurface documents based on GPS or based on your camera feed (or, will implement today - show related documents based on your desktop view) - primitive vector sim search now, will be something much better riding the internal representations of gemma4 (for now) - ok let me stop here, you'll have a pinned task list per workspace on the left and a set of a2ui canvases and old-school boring and rigid apps that will either be directly contextualized or just ride the contextualized filesystem(as obsidian does). Ou, did I mention agent usecase? Agent switches to /work/task1 and his local *.md files are only related to that task, want more zoom, switch to /work. ok, oook, let me really stop here
This is awesome. I always have imagined my computer being well, my desk. So far I have to live with a bunch of i3 screens I name after whatever I'm working on. I'd love to be able to page through the bench to flip to a task. Unfortunately with apps going so mono-windowed (slack, etc) it's harder to do. If I could just say have my slack window for this client, with their docs, proposals, database, whatever up, that'd be sick. I love the idea thank you. good luckl
I want to believe I work on something creative in this space (even though, the current UI was never "meant to be" and (d?)evolved from a substrate of cheaply available UI components and laziness). THen again, much of the unorthodoxy actually comes from all the small(and big) usability nightmares that I can not bring myself to tolerate while working in a desktop environment(so, created new ones - like bitmap-based context trees ;)
reply