I used to think about abandoning ghidra-delinker-extension.
It was a project that started innocently enough, but its domain is unbelievably complex. Recovering MIPS relocation spots from a Ghidra database sounds like an easy enough task, until you're confronted with behemoth functions that span thousands of instructions and undocumented psABI extensions that produces edge cases from Hell.
But then, someone contributed a PoC COFF exporter to go along with the PoC x86 ISA analyzer and after that the Windows video game decompilation picked it up, spreading by word of mouth. I've spent a ridiculously long time fixing bugs and learning about MSVC on-the-fly (quipping "there are lies, damned lies and the Microsoft Portable Executable and Common Object File Format Specification." on the decomp.me Discord server at one point). Then other architectures started creeping up in PRs, first x86_64 and later PowerPC. It's a bottomless pit of toolchains and platforms minutiae that demand perfection to pull off and would drive anyone stark raving mad.
It was bad enough that I let it sit for months at a time, only for someone to message me and fall back into it, then discover it got even more popular while I was away. I also somehow got invited to present a poster about it at ACM CCS 2025 in Taiwan, an absolutely insane story (how many hobbyists are invited to present something at a world-class academic conference on cyber-security?) that while very enlightening also physically wiped me out.
Copilot saved this project and I really mean it. Preparing artifacts, writing tests, performing investigations and large-scale refactorings: hours of grueling, soul-crushing menial work that I no longer have to subject myself to. Features that looked impossible like generating debugging symbols became within reach. The ironclad regression test suite happened to provide the perfect feedback loop. I still review the code and design, but I no longer burn myself out on this madness.
Recently I've been air-dropped into such a legacy project at work in order to save a cybersecurity-focused release date. Millions of lines of open-source code checked in a decade ago prior to the Subversion-to-Git migration, then patched everywhere to the point where diffs for the CVEs don't apply and we're not even sure what upstream versions best describe the forks.
By the end, the project manager begged me to turn off my flamethrower, as I was ripping it all out for a clean west manifest to tagged versions and stacks of patches. "Take it home OSS" is like take-out food: if you don't do your chores and leave it out for months or years on the kitchen counter, the next person to enter the apartment is going to puke.
But our modern embedded firmware projects all use Zephyr and west, so I just created a west manifest, stole parts of the scripts/ folder from the Zephyr repository to have a working "west patch" command and went to town. If I had more time to work on it, I'd have gotten "west build", "west flash" and "west debug" working too (probably with bespoke implementations) and removed the cargo cult shell scripts.
You can use west without Zephyr, it's just that by itself it only provides "west init" and "west update".
I have mixed feelings about west in general, but I like it enough that I'd probably look at doing something like that in the future too for harmony-sake with our existing Zephyr projects.
As the maintainer of ghidra-delinker-extension, whenever I get a non-trivial PR (like adding an object file format or ISA analyzer) I'm happy that it happens. It also means that I get to install a toolchain, maybe learn how to use it (MSVC...), figure out all of the nonsense and undocumented bullshit in it (COFF...), write byte-perfect roundtrip parser/serializer plus tests inside binary-file-toolkit if necessary, prepare golden Ghidra databases, write the unit tests for them, make sure that the delinked stuff when relinked actually works, have it pass my standards quality plus the linter and have a clean Git history.
I usually find it easier to take their branch, do all of that work myself (attributing authorship to commits whenever appropriate), push it to the master branch and close the PR than puppeteering someone halfway across the globe through GitHub comments into doing all of that for me.
Conversely, at work I implemented support for PKCS#7 certificate chains inside of Mbed-TLS and diligently submitted PRs upstream. They were correct, commented, documented, tested, everything was spotless to the implicit admission of one of the developers. It's still open today (with merge conflicts naturally) and there are like five open PRs for the exact same feature.
When I see this, I'm not going to insist, I'll move on to my next Jira task.
> push it to the master branch and close the PR than puppeteering someone halfway across the globe through GitHub comments into doing all of that for me
While I understand the sentiment I am glad I got into open source more than fifteen years ago, because it was maintainers “puppeteering” me that taught me a lot of the different processes involved in each project that would be hard to learn by myself later.
This is the Eternal September problem in disguise. That is, the personal interactions we treasure so much in small communities simply do not scale when the communities grow. If a community (or a project) grows too large then the maintainers / moderators can no longer spend this amount of time to help a beginner get up to speed.
There's a balance though. Some people want to end up with a perfect PR that gets applied, some just want the change upstream.
Most of my PRs are things where I'm not really interested in learning the ins and outs of the project I'm submitting to; I've just run into an issue that I needed fixed, so I fixed it[1], but it's better if it's fixed upstream. I'll submit my fix as a PR, and if it needs formatting/styling/whatevs, it'll probably be less hassle if upstream tweaks it to fit. I'm not looking to be a regular contributor, I just want to fix things. Personally, I don't even care about having my name on the fix.
Now, if I start pushing a lot of PRs, maybe it's worth the effort to get me onto the page stylewise. But, I don't usually end up pushing a lot of PRs on any one project.
[1] Usually, I can have my problem fixed much sooner if I fix it, than if I open a bug report and hope the maintainer will fix it.
> I usually find it easier to take their branch, do all of that work myself (attributing authorship to commits whenever appropriate), push it to the master branch and close the PR than puppeteering someone halfway across the globe through GitHub comments into doing all of that for me.
My most negative experiences with free and open source contributions have been like that. The one maintainer who engaged with me until my patches got into master was the best experience I ever had contributing to open source software to this day.
Pretty sad that people see engaging with other developers as "puppeteering someone halfway across the globe through GitHub comments"...
Ordinarily, I would agree. I don't do that with my other GitHub repositories nor at work.
But ghidra-delinker-extension's domain is unusually exacting. Mistakes or shortcuts there will cause undefined behavior of the exotic kind that can be exceedingly difficult to troubleshoot. The only way I've found to keep things from imploding is through unusually exacting quality standards that a drive-by contributor (who usually only cares about a particular ISA/object file combination on a given platform) can't be expected to match.
To be clear, I don't go silent when I receive a PR for this project. In those cases, I do engage with the contributor and give high-level feedback. But my spare time is finite and so is the other person's willingness to jump through all of my hoops.
> I usually find it easier to take their branch, do all of that work myself (attributing authorship to commits whenever appropriate), push it to the master branch and close the PR than puppeteering someone halfway across the globe through GitHub comments into doing all of that for me.
The PR system is great for reviewing changes within a dev team.
But, indeed, when its an external contribution, it kind of falls apart. It's unrealistic to expect from an external contributor, often a one time one, to know the ins and outs of a project (code standards, naming tastes, documentation, tests, ...)
I really like your workflow, and often did something similar in my projects (or merge in main, with subsequent fixes/realignments).
But I'm wondering if it could be smoothed out and normalized by Github/Gitlab/Forgeo, or maybe at the version control software level.
AI is like a genie: be careful what you wish for or you'll get what you asked for.
Lately at work I've done C++ optimization tricks like inplace_map, inplace_string, placement new to inline map-like iterators inside a view adapter's iterators and putting that byte buffer as the first member of the class to not incur std::max_align_t padding with the other members. At a higher architecture level, I wrote a data model binding library that can serialize JSON, YAML and CBOR documents to an output iterator one byte at a time without incurring heap allocation in most cases.
This is because I work on an embedded system with 640 KiB of SRAM and given the sheer amount of run-time data it will have to handle and produce, I'm wary not only about heap usage, but also heap fragmentation.
AI will readily identify such tricks, it can even help implement them, but unless constrained otherwise AI will pick the most expedient solution that answers the question (note that I didn't say answers the problem).
All my old software before AI was self documenting and didn't need comments -- it just was obvious. Today my prompts never make slop. I'm a really good driver.
At work, I wrote a C++20 data binding library. It works by running visitors over a data model that binds to the application state. My comment comes from a different set of trade-offs driven by memory constraints.
I've implemented a bunch of serialization visitors. For the structured formats, most (JSON, YAML, CBOR with indefinite lengths) use an output iterator and can stream out one character/byte at a time, which is useful when your target is a MCU with 640 KiB of SRAM and you need to reply large REST API responses.
And there's the BSON serializer, which writes to a byte buffer because it uses tag-length-value and I need to backtrack in order to patch in the lengths after serializing the values. This means that the entire document needs to be written upfront before I can do something with it. It also has some annoying quirks, like array indices being strings in base 10.
There are also other trade-offs when dealing with JSON vs. its binary encodings. Strings in JSON may have escape characters that require parsing, if it has them then you can't return a view into the document, you need to allocate a string to hold the decoded value. Whereas in BSON or CBOR (excluding indefinite-length strings) the strings are not escaped and you can return a std::string_view straight from the document (and even a const char* for BSON, as it embeds a NUL character).
Some encodings like CBOR are also more expressive than JSON, allowing for example any value type to be used for map keys and not just strings.
Parquet file format writes its metadata including length info after all data, at the footer. It was counterintuitive when I first look at it, but smart thinking about it now. I haven't had to trade off for memory constraints, but being able to stream output is definitely easier!
Interesting point about the difference in escape characters, I stored length and the decoded value so it's ready for string view. But when I need them back as JSON string, I need to encode them again :)
I'm working on ghidra-delinker-extension [1], a relocatable object file exporter for Ghidra. Or in other words, a delinker.
Delinking is the art of stripping program for parts, essentially. The tricky part is recovering and resynthesizing relocation spots through analysis. It is a punishingly hard technique to get right because it requires exacting precision to pull off, as mistakes will corrupt the resulting object files in ways that can be difficult to detect and grueling to debug. Still, I've managed to make it work on multiple architectures and object file formats; a user community built up through word of mouth and it's now actively used in several Windows video game decompilation projects.
Recently I've experimented with Copilot and GPT-5.3 to implement support for multiple major features, like OMF object file format and DWARF debugging symbols generation. The results have been very promising, to the point where I can delegate the brunt of the work to it and stick to architecture design and code review. I've previously learned the hard way that the only way to keep this extension from imploding on itself was with an exhaustive regression test suite and it appears to guardrail the AI very effectively.
Given that I work alone on this in my spare time, I have a finite amount of endurance and context and I was reaching the limits of what I could manage on my own. There's only so much esoterica about ISAs/object file formats/toolchains/platforms that can fit at once in one brain and some features (debugging symbols generation) were simply out of reach. Now, it seems that I can finally avoid burning out on this project, albeit at a fairly high rate of premium requests consumption.
Interestingly enough, I've also experimented with local AI (mostly oss-gpt-20b) and it suffers from complete neural collapse when trying to work on this, probably because it's a genuinely difficult topic even for humans.
I'm working on an embedded project where I'm actually thinking about using ECS on a STM32H5. It's not about cache-friendly locality (waitstates on internal SRAM for MCUs is basically a rounding error compared to the DRAM latency seen on desktop or server-class hardware), but the business is so complex that the traditional inheritance/composition is getting quite messy. When you end up using virtual and diamond inheritance in C++, you know you're in trouble...
It's too bad that ECS isn't more widely known outside of gamedev. It's not just good for performance, it's also legitimately a useful architecture design on its own to solve problems.
I'm working on ghidra-delinker-extension [1], which is a relocatable object file exporter for Ghidra.
The algorithms needed to slice up a Ghidra database into relocatable sections, and especially to recover relocations through analysis are really tricky to get right. My MIPS analyzer in particular is an eldritch horror due to several factors combining into a huge mess (branch delay slots, split HI16/LO16 relocations, code flow analysis, register graph dependency...).
The entire endeavor requires an unusual level of exacting precision to work and will produce some really exotic undefined behavior when it fails, but when it works you feel like a mechanic in a Mad Max universe, stripping programs for parts and building unholy chimeras from them, some examples I've linked in the readme. It has also led to a poster presentation to the SURE workshop at ACM CCS 2025 in Taiwan as a hobbyist, an absolutely insane story.
Mad respect. I tried extracting a clean .o file out of a statically linked ELF once, and it's an absolute nightmare. How are you handling switch tables and indirect jumps? Without dynamic analysis, it's sometimes physically impossible to figure out what a register is actually pointing to
I have analyzers that resynthesize relocations from the contents of the Ghidra database, no custom annotations required. They evaluate relocation candidate spots through primary references and pointers/instructions and emit warnings if the math doesn't check out.
It does require a reasonably accurate Ghidra database to work properly, but I've had users delink megabytes of code and data from a program successfully (as in, relinking it at a different address results in a functionally identical executable) once they've cleaned it up. The accuracy warning in the readme is mostly because it's really complicated to describe exactly what inaccuracies you can get away with, there's a fair amount of wiggle room in reality as long as you know what you're doing.
I'm curious how your analyzers handle those tricky MIPS edge cases like split address loads via lui and addiu where you have interleaved instructions in between. If Ghidra fails to collapse those into a single reference, does your math check just flag it as an error and bail, or does it actually try to reconstruct the instruction chain itself?
It uses register dependency graph traversal and code block flow analysis to try and find which two instructions are the likely targets of a HI16/LO16 for a given reference. It also uses an unhealthy amount of recursion and has been rewritten multiple times in order to deal with all these edge cases.
Unfortunately, my extension is only set up for one HI16 relocation having one or many LO16 child relocations. There's an undocumented GNU extension that allows multiple HI16 relocations to be shared by a single LO16 relocation, and I have a really ugly hack that shifts branch targets to "normalize" this into the standard pattern. I don't know why this hack works (I didn't even know about the GNU extension, I thought it was an assembler peephole optimization), but with it my extension can delink the whole of Tenchu: Stealth Assassins without any manual annotations, just the regular references from the cleaned-up Ghidra database.
I'm working on stuff in that market, it's still largely is. DC Power System Design For Telecommunications is still a must read and it doesn't even cover the last 15 years or so of development, notably lithium batteries and high efficiency rectifiers.
I will say that this is a surprisingly deep and complex domain. The amount of flexibility, variety and scalability you see in DC architectures is mind-boogling. They can span from a 3kW system that fits in 2U all the way to multiples of 100kWs that span entire buildings and be powered through any combination of grid, solar and/or gas.
This appears to be organized top-down, with categories as the entry point. How do we reverse-lookup a story? That is, given a story, how can I find it (if it's there) and walk it back up into its category?
I have stories (assuming that they've made the cut) that are either in my favorites or that I've submitted, that I'd like to know how they were classified and alongside what other stories. So instead of browsing this from the outside in, I'd browse it from the inside out.
It was a project that started innocently enough, but its domain is unbelievably complex. Recovering MIPS relocation spots from a Ghidra database sounds like an easy enough task, until you're confronted with behemoth functions that span thousands of instructions and undocumented psABI extensions that produces edge cases from Hell.
But then, someone contributed a PoC COFF exporter to go along with the PoC x86 ISA analyzer and after that the Windows video game decompilation picked it up, spreading by word of mouth. I've spent a ridiculously long time fixing bugs and learning about MSVC on-the-fly (quipping "there are lies, damned lies and the Microsoft Portable Executable and Common Object File Format Specification." on the decomp.me Discord server at one point). Then other architectures started creeping up in PRs, first x86_64 and later PowerPC. It's a bottomless pit of toolchains and platforms minutiae that demand perfection to pull off and would drive anyone stark raving mad.
It was bad enough that I let it sit for months at a time, only for someone to message me and fall back into it, then discover it got even more popular while I was away. I also somehow got invited to present a poster about it at ACM CCS 2025 in Taiwan, an absolutely insane story (how many hobbyists are invited to present something at a world-class academic conference on cyber-security?) that while very enlightening also physically wiped me out.
Copilot saved this project and I really mean it. Preparing artifacts, writing tests, performing investigations and large-scale refactorings: hours of grueling, soul-crushing menial work that I no longer have to subject myself to. Features that looked impossible like generating debugging symbols became within reach. The ironclad regression test suite happened to provide the perfect feedback loop. I still review the code and design, but I no longer burn myself out on this madness.
reply