I strongly recommend checking out 010 Editor [1].

It's very powerful, lets you display in multiple different formats (not just 1/2/4/8 bytes, but interlaced formats and byte-arrays) and has the most amazing templating / scripting engine I've seen for this type of tool.

The only caveat is that it isn't free, but if this is something you do for a living (as I do) it's an indispensable tool for exploring file formats and other binary data sources.

[1]: https://www.sweetscape.com/010editor/

Yes! The 010 Editor's templating language is a nearly unique (in my experience) hybrid between C-like declarations and imperative flow that's exactly what I want for parsing arbitrary binary files.

The template executes like a script (with conditionals and looping) and a line like "int32 myNum;" is actually just syntactic sugar for "read 4 bytes from the current file pointer and label it myNum." Their example probably does a better job explaining it than I can:

  struct FILE {
      struct HEADER {
   char    type[4];
          int     version;
          ushort  numRecords;
      } header <bgcolor=cLtGray>;
  
      struct RECORD {
          int     employeeId;
          char    name[40];
          float salary;
          if( file.header.version > 1 )
              int      numChildren;
          if( file.header.version > 2 )
              time_t   birthDate;
      } record[ file.header.numRecords ];
  
  } file;

( From: https://www.sweetscape.com/010editor/templates.html )

Yep, same!

010 Editor is one of the few commercial tools that I just have to have a license for - I don't use it to make a living, but I've participated in some CTFs and challenges, and just yesterday I noticed that a WAV file had metadata which I didn't know was a thing, so I just opened it in 010 Editor with the WAV template and I could see how the metadata was stored, and all other fields!

If you’re on the Mac, Synalyze It! is somewhat similar.

https://www.synalysis.net/

I used this for viewing mysql binary logs recently to understand the format. I was upset when I realized the $10 version was severely nerfed. With lack of scripting I couldn't write even a basic grammar to parse out the binlog file. At that point I was too annoyed to shell out another $40 for the pro version.

If you are considering this, get the Pro version. I would recommend to the owner to just make the non-pro version free.

What's the lowest level of granularity? Byte level?

Because I'd love templating feature like that works at the bit level.

IOW: you can specify some a bunch of fields to be, say, 3 bits each, and that when all the bits of a byte are used, it simply spills over to the next byte.

Serious question, is it really worth it for $130?

The next time you want to dwell into a file format, download it and do the 30 day trial - maybe there's already an existing template for the file you're looking for - if not, try making your own - it's super easy! Then you can decide if it's worth it. If such a situation never arises, then no, it's not worth it in your case.

I used it a lot when datamining in World of Warcraft - I would find reverse engineered documentation from previous versions of the game, write it into a 010 editor template struct, run it on the DBC file and then I'd usually get a result where the header and the first few fields would be fine, but the rest would be completely wrong. All I did was add a field in the template before the first obviously wrong field, with different, random sizes, to add random padding until the field started looking correct again.

It's a pretty specific case, but it almost always worked!

Okteta is part of the KDE project, GPL, and has similar templating functionality

it's also my hex editor of choice, has been for years

Something didn't seem quite right about those hexdumps at first glance; and then I saw the odd numbers in the left and realised it was showing 28 bytes per line instead of the usual 16 or 32. The lack of spaces between bytes also tends to make one want to read the values as BE dords, which they often are not.

(Long-time reverse-engineer. My go-to hex editor is still HIEW.)

FYI both the byte grouping and bytes per line are are configurable. It fits to the window width by default.

I love hiew as well for Windows. Do you know of a tool that is comparable for *nix?

On other hex editors: HxD[1] recently received some updates after years of being dormant

HxD is a bit sentimental to me as it's the hex editor I used over a decade ago when figuring out that you could flip one bit at a deterministic position in Game Maker executables to enable the debugger for release builds. A small find, since soon after someone else figured out how to extract the whole source from release builds

1: https://mh-nexus.de/en/hxd/

I've been using HxD for years - would love to see a comparison vs Rehex.

So Kaitai Struct (http://kaitai.io/ - they have a web IDE) isn't a hex editor, but their IDE is what I'd jump to if I had to pull apart a blob of binary data. You define the schema and it automatically turns the data into a labeled tree you can click through without needing to remember offsets or byte patterns or whatever.

You can use the schema in a number of programming languages then, and if you wanted to compare files it might be a few more steps but you could export your data as json and get a semantic diff vs "4 bytes changed at offset 1294".

I tried Kaitai struct, won't recommend it.

The yaml based language is awful to use. It doesn't support bit level reading, so you can't do unaligned bit reading, used often in networks packets.

macOS users should take a look at Hex Fiend for reverse engineering: https://github.com/ridiculousfish/HexFiend

We used to use that in 'production' to clean up CSVs that got exported with a weird byte at the beginning

Perhaps the weird byte(s?) were a BOM?

Are you sure that weird byte at the begining is not a Unicode BOM (Byte-Order-Mark)? I've seen a few programs do that (or expect it) and other programs ignore it or go crazy on it.

Why would you need anything more than a sed oneliner in your shell rc?

utf8-debom() {local SED=gsed; $SED -i '1s/^\xEF\xBB\xBF//' $@}

Hex Fiend is a good hex editor but it’s really not the same type of tool as this is.

On a related topic, there's this program, not an editor but a binary explorer, which looks somehow relevant to reverse engineering: https://hachoir.readthedocs.io/

The principe seems very good, like a wireshark for binary files. Not sure how powerful it is in practice.

It can browse the hierarchy behind a binary format, down to individual fields (even bits).

API offers edition capabilities.

It has graphical (wx) and textmode (python-urwid) interfaces.

Hachoir is able to open invalid / truncated files. Here are the available commands:

    hachoir-grep
    hachoir-metadata
    hachoir-metadata-csv
    hachoir-metadata-gtk
    hachoir-metadata-qt
    hachoir-strip
    hachoir-subfile
    hachoir-urwid
    hachoir-wx

A tool I released years back is in the similar vein. It is mostly intended to be a binary file template specification and analysis tool: https://github.com/binspector/binspector

If anyone is interested, it is using the C++ library wxWidgets[0] for it's cross platform UI.

[0] https://www.wxwidgets.org/

Which doesn't support dark mode (my eyes bleed).

light themes are actually easier on the eyes but you do you

You do you doesn't apply here, cause I can't.

In UI, a lack of consistency hurts regardless of whatever preference. Which is why all these toolkits was a mess back in the days. Which is why applications have the ability to follow whatever preference the user set in UI (light, dark, solarized, etc). Ignoring following default choice, a lack of choice for dark mode therefore always hurts dark mode users.

While dark mode during day is perfectly fine, light mode during night does not cut it for me. I usually go for dark mode full brightness in day, and half brightness during night (after sundown).

This is neat, though it doesn't really go deep enough for me. Not being able to specify variable length fields is unfortunate.

What do you mean by variable-length fields? Feedback on how to improve it is always welcome!

Let's say I have a data definition that is some length value and then some number of bytes based on the length value. If I want to load up the same definition for another file (something I guess you also don't support) the length will be wrong if I just manually go forward that many bytes. Sorry, this is a really bad explanation.

Yeah, there isn't support for anything like that yet. I'm planning on having support for Kaitai Struct (and/or other definition languages) in addition to a tool for automating repeating comments/highlights/etc, which might end up being able to do what you describe.

Yeah that would probably do it

Impressive! Really neat work!

I've played a little bit with hexl-mode in Emacs; does anyone know of any extensions/built-in functions that can do some of what's going on here with Rehex?

Thank you! This is awesome. I've been using Hex Fiend on Mac along with Ghidra and I could use a leveled up editor.

Hi, you do you use the feature of Disassembly ?

Do any of these hex editors handle RADIX50 (RAD50) ?

This would have been so useful when learning ASN.1..