Anthony Scopatz
2013-07-12 18:40:30 UTC
Hi Robert,
Glad these materials can be helpful. (Note: these questions really should
be asked on the pytables-users mailing list -- CC'd here -- so please join
that list: https://lists.sourceforge.net/lists/listinfo/pytables-users)
On Fri, Jul 12, 2013 at 12:48 PM, Robert Nelson <
recollection) did not open an issue and so no changes have been made to
PyTables to handle this.
Also in this strategy, you should only be loading in the indices to start
with. I doubt (though I could be wrong) that you have 1 Gb worth of index
data alone. The whole idea here is to do a unique (set) and a sort
operation on the much smaller index data AND THEN use fancy indexing to
pull the actual data back out.
As always some sample code and a sample file would be extremely helpful. I
don't think I can do much more for you without these.
Be Well
Anthony
Glad these materials can be helpful. (Note: these questions really should
be asked on the pytables-users mailing list -- CC'd here -- so please join
that list: https://lists.sourceforge.net/lists/listinfo/pytables-users)
On Fri, Jul 12, 2013 at 12:48 PM, Robert Nelson <
Dr. Scopatz,
I came across your SciPy 2012 "HDF5 is for lovers" video and thought you
might be able to help me.
I'm trying to read large (>1GB) HDF files and do multidimensional indexing
convert it to a NumPy array but this takes too long.
I think that the strategy is the same as before. Ask (to the best of myI came across your SciPy 2012 "HDF5 is for lovers" video and thought you
might be able to help me.
I'm trying to read large (>1GB) HDF files and do multidimensional indexing
convert it to a NumPy array but this takes too long.
recollection) did not open an issue and so no changes have been made to
PyTables to handle this.
Also in this strategy, you should only be loading in the indices to start
with. I doubt (though I could be wrong) that you have 1 Gb worth of index
data alone. The whole idea here is to do a unique (set) and a sort
operation on the much smaller index data AND THEN use fancy indexing to
pull the actual data back out.
As always some sample code and a sample file would be extremely helpful. I
don't think I can do much more for you without these.
Be Well
Anthony
Have there been any updates in PyTables that would make this possible?
Thank you!
Robert Nelson
Colorado State University
763-354-8411
Thank you!
Robert Nelson
Colorado State University
763-354-8411