[pytables-users] Blosc decompression error when reading table

[David Menéndez Hurtado]

Hi

We are encountering some random errors reading from a table (~200 GB).
The reading is sequential and deterministic, but it never seem to
crash on the same place. Here is one traceback:


Exception! HDF5 error back trace

File "H5Dio.c", line 173, in H5Dread
can't read data
File "H5Dio.c", line 554, in H5D__read
can't read data
File "H5Dchunk.c", line 1875, in H5D__chunk_read
unable to read raw data chunk
File "H5Dchunk.c", line 2905, in H5D__chunk_lock
data pipeline read failed
File "H5Z.c", line 1372, in H5Z_pipeline
filter returned failure during read
File "c-blosc/hdf5/blosc_filter.c", line 294, in blosc_filter
Blosc decompression error

End of HDF5 error back trace

Problems reading records.
Exception in thread Thread-10:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.
py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File
"/home/miljaaho/.venv/local/lib/python2.7/site-packages/keras/engine/training.py",
line 427, in data_generator_task
generator_output = next(generator)
File "neural_pdbcull.py", line 125, in test_generator
features.extend([getattr(table.cols, k)[i0:i1] for k in keys_h5_float])
File
"/home/miljaaho/.venv/local/lib/python2.7/site-packages/tables/table.py",
line 3582, in __getitem__
return table.read(start, stop, step, self.pathname)
File
"/home/miljaaho/.venv/local/lib/python2.7/site-packages/tables/table.py",
line 2008, in read
arr = self._read(start, stop, step, field, out)
File
"/home/miljaaho/.venv/local/lib/python2.7/site-packages/tables/table.py",
line 1930, in _read
self.row._fill_col(result, start, stop, step, field)
File "tables/tableextension.pyx", line 1232, in
tables.tableextension.Row._fill_col (tables/tableextension.c:14485)
File "tables/tableextension.pyx", line 578, in
tables.tableextension.Table._read_records
(tables/tableextension.c:7491)
HDF5ExtError: HDF5 error back trace

File "H5Dio.c", line 173, in H5Dread
can't read data
File "H5Dio.c", line 554, in H5D__read
can't read data
File "H5Dchunk.c", line 1875, in H5D__chunk_read
unable to read raw data chunk
File "H5Dchunk.c", line 2905, in H5D__chunk_lock
data pipeline read failed
File "H5Z.c", line 1372, in H5Z_pipeline
filter returned failure during read
File "c-blosc/hdf5/blosc_filter.c", line 294, in blosc_filter
Blosc decompression error

End of HDF5 error back trace

Problems reading records.


Sometimes, only one the HDF5 traceback is shown. The table was created
by PyTables with Filters(9, 'blosc:snappy'), doesn't have an index and
it is being read as:

h5.root.table.cols.column_name[i0:i1]



More information on the system:

tables.__version__
'3.2.2'

tables.get_hdf5_version()
'1.8.17'

tables.blosc_get_complib_info()
{'blosclz': ('BloscLZ', '1.0.2.1'),
'lz4': ('LZ4', '1.2.0'),
'lz4hc': ('LZ4', '1.2.0'),
'snappy': ('Snappy', '1.1.1'),
'zlib': ('Zlib', '1.2.8')}

The OS is Linux.

I realise this is probably not enough information to diagnose. How can
we investigate this further?


Thanks
/David.

[Francesc Alted]

Hmm, I am in a hurry now, but can you put a minimal example (or file)
reproducing the crash so that I can have a look at it? Also, is snappy the
only codec creating the issue? Can you reproduce that with any other codec
(blosclz, lz4, lz4hc or zlib)?

Thanks for the report!

Post by David Menéndez Hurtado
Hi
We are encountering some random errors reading from a table (~200 GB).
The reading is sequential and deterministic, but it never seem to
Exception! HDF5 error back trace
File "H5Dio.c", line 173, in H5Dread
can't read data
File "H5Dio.c", line 554, in H5D__read
can't read data
File "H5Dchunk.c", line 1875, in H5D__chunk_read
unable to read raw data chunk
File "H5Dchunk.c", line 2905, in H5D__chunk_lock
data pipeline read failed
File "H5Z.c", line 1372, in H5Z_pipeline
filter returned failure during read
File "c-blosc/hdf5/blosc_filter.c", line 294, in blosc_filter
Blosc decompression error
End of HDF5 error back trace
Problems reading records.
File "/usr/lib/python2.7/threading.
py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File
"/home/miljaaho/.venv/local/lib/python2.7/site-packages/keras/engine/training.py",
line 427, in data_generator_task
generator_output = next(generator)
File "neural_pdbcull.py", line 125, in test_generator
features.extend([getattr(table.cols, k)[i0:i1] for k in keys_h5_float])
File
"/home/miljaaho/.venv/local/lib/python2.7/site-packages/tables/table.py",
line 3582, in __getitem__
return table.read(start, stop, step, self.pathname)
File
"/home/miljaaho/.venv/local/lib/python2.7/site-packages/tables/table.py",
line 2008, in read
arr = self._read(start, stop, step, field, out)
File
"/home/miljaaho/.venv/local/lib/python2.7/site-packages/tables/table.py",
line 1930, in _read
self.row._fill_col(result, start, stop, step, field)
File "tables/tableextension.pyx", line 1232, in
tables.tableextension.Row._fill_col (tables/tableextension.c:14485)
File "tables/tableextension.pyx", line 578, in
tables.tableextension.Table._read_records
(tables/tableextension.c:7491)
HDF5ExtError: HDF5 error back trace
File "H5Dio.c", line 173, in H5Dread
can't read data
File "H5Dio.c", line 554, in H5D__read
can't read data
File "H5Dchunk.c", line 1875, in H5D__chunk_read
unable to read raw data chunk
File "H5Dchunk.c", line 2905, in H5D__chunk_lock
data pipeline read failed
File "H5Z.c", line 1372, in H5Z_pipeline
filter returned failure during read
File "c-blosc/hdf5/blosc_filter.c", line 294, in blosc_filter
Blosc decompression error
End of HDF5 error back trace
Problems reading records.
Sometimes, only one the HDF5 traceback is shown. The table was created
by PyTables with Filters(9, 'blosc:snappy'), doesn't have an index and
h5.root.table.cols.column_name[i0:i1]
tables.__version__
'3.2.2'
tables.get_hdf5_version()
'1.8.17'
tables.blosc_get_complib_info()
{'blosclz': ('BloscLZ', '1.0.2.1'),
'lz4': ('LZ4', '1.2.0'),
'lz4hc': ('LZ4', '1.2.0'),
'snappy': ('Snappy', '1.1.1'),
'zlib': ('Zlib', '1.2.8')}
The OS is Linux.
I realise this is probably not enough information to diagnose. How can
we investigate this further?
Thanks
/David.
--
You received this message because you are subscribed to the Google Groups
"pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[Daπid]

I have been bitten by this in production for months, but all the
simplified cases were working fine. I have finally realised: the crash
happens when several threads try to access the same file, but using
multiprocessing instead works fine. The compressor used (or lack of)
is irrelevant. Attached is an example: just set P_SAFE to True/False
to switch behaviour and see it crash immediately, or run for hours.

For my case, the workaround is trivial, but I think we either need a
few locks, or document this (HDF5 is supposed to be readable in
parallel).


The tracebacks weren't even consistent: the crashes came from
different files every time. I was starting to imagine some weird
interaction with the GPU or some other mystical things. On hindsight,
I should have suspected something related with parallelism.


/David.

[Daπid]

Post by Daπid
document this (HDF5 is supposed to be readable in
parallel).

I see it has been done, but I missed it:

http://www.pytables.org/latest/cookbook/threading.html