HDF5 isn't perfect but it does this kind of job pretty well. The C, C++, and HDF5 APIs are definitely not fun to use, but there are wonderful and intuitive APIs available in some languages---I'm thinking of Python's h5py here.
Let me add that the OP's experience that HDF5 files were less space efficient than comparable CSV files suggest that something was grossly amiss in his use of HDF5.
(I'm the author of the library) I'm almost certain you're correct - we (thought) we had compression enabled on our feature builder and never found the root cause, but regardless we're happy with how BTables ended up for the other reasons detailed. For future use cases we'll definitely be re-evaluating HDF5!
Let me add that the OP's experience that HDF5 files were less space efficient than comparable CSV files suggest that something was grossly amiss in his use of HDF5.