should I use pytables?

Discussion:

Chao YUE

2013-08-07 12:44:26 UTC

Dear all,

I have a hierachical nested python dictionaries with the end of the branch
as either pandas dataframe, or np.ndarray or list or plain scalars.

let's say the different levels of keys are:

1st level: ['top1', 'top2', 'top3']
2nd level: ['mid1','mid2','mid3']
3rd level: ['bot1','bot2','bot3','bot4']

I think I am looking for some data strucuture that allow easy retrieving of
the data at different levels as dictionaries (I cannot think out something
better yet).

for example: data.ix['top1',:,'bot1'] will have keys only at the middle
levels.

I have a quick look of pytables document but not very sure, should I use
pytables for this purpose?

thanks a lot for any idea.

cheers,

Chao

--
***********************************************************************************
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
************************************************************************************

Anthony Scopatz

2013-08-07 16:44:35 UTC

Permalink

Post by Chao YUE
Dear all,
I have a hierachical nested python dictionaries with the end of the branch
as either pandas dataframe, or np.ndarray or list or plain scalars.
1st level: ['top1', 'top2', 'top3']
2nd level: ['mid1','mid2','mid3']
3rd level: ['bot1','bot2','bot3','bot4']
I think I am looking for some data strucuture that allow easy retrieving
of the data at different levels as dictionaries (I cannot think out
something better yet).
for example: data.ix['top1',:,'bot1'] will have keys only at the middle
levels.
I have a quick look of pytables document but not very sure, should I use
pytables for this purpose?

Hello Chao,

If you are only ever going to use this data structure in memory, you
shouldn't use pytables. If you are going to persist this information to
disk than pytables is a great choice! Every dictionary will become a group
and every leaf data structure will become an Array or a Table.

Be Well
Anthony

Post by Chao YUE
thanks a lot for any idea.
cheers,
Chao
--
***********************************************************************************
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
************************************************************************************
------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
Pytables-users mailing list
https://lists.sourceforge.net/lists/listinfo/pytables-users

Chao YUE

2013-08-07 19:44:09 UTC

Permalink

Thanks Anthony, I think I will give a try, apprently at some stage I would
like to flush the data into disk :p

cheers,

Chao

Post by Anthony Scopatz

Post by Chao YUE
Dear all,
I have a hierachical nested python dictionaries with the end of the
branch as either pandas dataframe, or np.ndarray or list or plain scalars.
1st level: ['top1', 'top2', 'top3']
2nd level: ['mid1','mid2','mid3']
3rd level: ['bot1','bot2','bot3','bot4']
I think I am looking for some data strucuture that allow easy retrieving
of the data at different levels as dictionaries (I cannot think out
something better yet).
for example: data.ix['top1',:,'bot1'] will have keys only at the middle
levels.
I have a quick look of pytables document but not very sure, should I use
pytables for this purpose?

Hello Chao,
If you are only ever going to use this data structure in memory, you
shouldn't use pytables. If you are going to persist this information to
disk than pytables is a great choice! Every dictionary will become a group
and every leaf data structure will become an Array or a Table.
Be Well
Anthony

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
Pytables-users mailing list
https://lists.sourceforge.net/lists/listinfo/pytables-users