[pytables-users] Writing PyTables Format From C/C++

[Shaklee3]

Hi, I have a high-speed network packet capturing application written in C++
that I need to save binary files out that will eventually get loaded into
MATLAB (HDF5). There is a library called MATIO that can save to the format
MATLAB requires for their HDF5 files, and that seems to work well. However,
I'd also like the ability to query this large dataset (> 10TB eventually),
which got me interested in PyTables. I noticed on a stackoverflow post I
found that for the most part PyTables just adds a couple extra attributes
to HDF5 files, but it wasn't very specific. My question is: if I wanted to
save files from my C++ application that can later be read into PyTables, is
there a document I can follow to use that format? I think this part would
be harder than loading it into MATLAB, so if I can solve that issue first I
can probably figure out how to convert it to MATLAB later. Thanks!

[Antonio Valentino]

Hi Shaklee3,

Post by Shaklee3
Hi, I have a high-speed network packet capturing application written in C++
that I need to save binary files out that will eventually get loaded into
MATLAB (HDF5). There is a library called MATIO that can save to the format
MATLAB requires for their HDF5 files, and that seems to work well. However,
I'd also like the ability to query this large dataset (> 10TB eventually),
which got me interested in PyTables. I noticed on a stackoverflow post I
found that for the most part PyTables just adds a couple extra attributes
to HDF5 files, but it wasn't very specific. My question is: if I wanted to
save files from my C++ application that can later be read into PyTables, is
there a document I can follow to use that format? I think this part would
be harder than loading it into MATLAB, so if I can solve that issue first I
can probably figure out how to convert it to MATLAB later. Thanks!

PyTables format is described in [1].

Anyway PyTables should be able to open any plain HDF5 file.
Once open the data file you can add indexes and perform queries.


[1] https://pytables.github.io/usersguide/file_format.html


cheers