Discussion:
[pytables-users] multi-thread with numexpr
Jap Pyck
2017-11-17 15:31:05 UTC
Permalink
Hello,

I am new at pytables and particularly interested in the multi-thread
feature available through numexpr. I ran a few tests on a tables with 20
millions rows and 2 columns 'colA', 'colB' with MAX_NUMEXPR_THREADS set to
1, 2 and 4. The test was run on a MacPro 8cores.



Although I see the number of threads increases during the execution of the
query, the performance does not change.

It looks very similar to the limitation imposed by the GIL on
multi-threaded functions in python. I though that HDF5 could get around
this limitation.

Question: is there anything I missed or do wrong? Do I need to do
additional configuration changes to see a performance increase?

Thank you for your help
-JPK
--
You received this message because you are subscribed to the Google Groups "pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pytables-users+***@googlegroups.com.
To post to this group, send an email to pytables-***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Francesc Alted
2017-11-17 19:25:18 UTC
Permalink
Hi Jap,

​I think you need to use NUMEXPR_MUM_THREADS instead of
MAX_NUMEXPR_THREADS. See
http://numexpr.readthedocs.io/en/latest/user_guide.html for more info about
the former.
​
​Francesc​
Post by Jap Pyck
Hello,
I am new at pytables and particularly interested in the multi-thread
feature available through numexpr. I ran a few tests on a tables with 20
millions rows and 2 columns 'colA', 'colB' with MAX_NUMEXPR_THREADS set to
1, 2 and 4. The test was run on a MacPro 8cores.
Although I see the number of threads increases during the execution of the
query, the performance does not change.
It looks very similar to the limitation imposed by the GIL on
multi-threaded functions in python. I though that HDF5 could get around
this limitation.
Question: is there anything I missed or do wrong? Do I need to do
additional configuration changes to see a performance increase?
Thank you for your help
-JPK
--
You received this message because you are subscribed to the Google Groups
"pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.
--
Francesc Alted
--
You received this message because you are subscribed to the Google Groups "pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pytables-users+***@googlegroups.com.
To post to this group, send an email to pytables-***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Jap Pyck
2017-11-17 22:48:24 UTC
Permalink
Hi Francesc,
Thanks for your quick response. I tested NUMEXPR_MUM_THREADS but didn't get
any improvement. Here is a short example the runtimes are:

9.650904 sec

9.616117 sec

9.511723 sec
Thank you again for your help,
-JPK

#--------------------------------------------------------
from tables import *
import tables.parameters as para
import numpy as np
import time

nItems =100000000
f = open_file('test.hdf5', mode = 'w')
Table = np.zeros(nItems,dtype=([('A','f4'),('B','f4')]))
Table['A']= np.random.random_sample(nItems)
Table['B']= np.random.random_sample(nItems)
f.create_table(f.root,'Table',description=Table,expectedrows=nItems)
f.flush()
f.close()
#
#
def test():
f = open_file('test.hdf5', mode = 'r')
Table = f.root.Table
Table.get_where_list('((A > 0.2) & (A < 0.3)) | ((B > 0.5) & (B <
0.6))')
f.close()

t = time.clock()
para.NUMEXPR_MUM_THREADS = 1
test()
print time.clock() -t; t = time.clock()
para.NUMEXPR_MUM_THREADS = 2
test()
print time.clock() -t; t = time.clock()
para.NUMEXPR_MUM_THREADS = 3
test()
print time.clock() -t; t = time.clock()
#--------------------------------------------------------
Post by Jap Pyck
Hello,
I am new at pytables and particularly interested in the multi-thread
feature available through numexpr. I ran a few tests on a tables with 20
millions rows and 2 columns 'colA', 'colB' with MAX_NUMEXPR_THREADS set to
1, 2 and 4. The test was run on a MacPro 8cores.
Although I see the number of threads increases during the execution of the
query, the performance does not change.
It looks very similar to the limitation imposed by the GIL on
multi-threaded functions in python. I though that HDF5 could get around
this limitation.
Question: is there anything I missed or do wrong? Do I need to do
additional configuration changes to see a performance increase?
Thank you for your help
-JPK
--
You received this message because you are subscribed to the Google Groups "pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pytables-users+***@googlegroups.com.
To post to this group, send an email to pytables-***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Francesc Alted
2017-11-18 09:08:40 UTC
Permalink
Hmm, you should rather be using NUMEXPR_MUM_THREADS as an *environment*
variable in the shell. But I have been revising the docs and apparently
`tables.parameters.MAX_NUMEXPR_THREADS` should work too; however, I am not
sure if you can change this parameter dynamically. A good test would be to
try the very same expression but using numpy arrays with numexpr instead of
HDF5 datasets and see if you can see acceleration by using more threads for
this case.

Francesc
Post by Jap Pyck
Hi Francesc,
Thanks for your quick response. I tested NUMEXPR_MUM_THREADS but didn't
9.650904 sec
9.616117 sec
9.511723 sec
Thank you again for your help,
-JPK
#--------------------------------------------------------
from tables import *
import tables.parameters as para
import numpy as np
import time
nItems =100000000
f = open_file('test.hdf5', mode = 'w')
Table = np.zeros(nItems,dtype=([('A','f4'),('B','f4')]))
Table['A']= np.random.random_sample(nItems)
Table['B']= np.random.random_sample(nItems)
f.create_table(f.root,'Table',description=Table,expectedrows=nItems)
f.flush()
f.close()
#
#
f = open_file('test.hdf5', mode = 'r')
Table = f.root.Table
Table.get_where_list('((A > 0.2) & (A < 0.3)) | ((B > 0.5) & (B <
0.6))')
f.close()
t = time.clock()
para.NUMEXPR_MUM_THREADS = 1
test()
print time.clock() -t; t = time.clock()
para.NUMEXPR_MUM_THREADS = 2
test()
print time.clock() -t; t = time.clock()
para.NUMEXPR_MUM_THREADS = 3
test()
print time.clock() -t; t = time.clock()
#--------------------------------------------------------
Post by Jap Pyck
Hello,
I am new at pytables and particularly interested in the multi-thread
feature available through numexpr. I ran a few tests on a tables with 20
millions rows and 2 columns 'colA', 'colB' with MAX_NUMEXPR_THREADS set to
1, 2 and 4. The test was run on a MacPro 8cores.
Although I see the number of threads increases during the execution of
the query, the performance does not change.
It looks very similar to the limitation imposed by the GIL on
multi-threaded functions in python. I though that HDF5 could get around
this limitation.
Question: is there anything I missed or do wrong? Do I need to do
additional configuration changes to see a performance increase?
Thank you for your help
-JPK
--
You received this message because you are subscribed to the Google Groups
"pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.
--
Francesc Alted
--
You received this message because you are subscribed to the Google Groups "pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pytables-users+***@googlegroups.com.
To post to this group, send an email to pytables-***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Jap Pyck
2017-11-19 17:13:40 UTC
Permalink
Thanks for the suggestion. I will run that test and get back to you
Post by Francesc Alted
Hmm, you should rather be using NUMEXPR_MUM_THREADS as an *environment*
variable in the shell. But I have been revising the docs and apparently
`tables.parameters.MAX_NUMEXPR_THREADS` should work too; however, I am not
sure if you can change this parameter dynamically. A good test would be to
try the very same expression but using numpy arrays with numexpr instead of
HDF5 datasets and see if you can see acceleration by using more threads for
this case.
Francesc
Post by Jap Pyck
Hi Francesc,
Thanks for your quick response. I tested NUMEXPR_MUM_THREADS but didn't
9.650904 sec
9.616117 sec
9.511723 sec
Thank you again for your help,
-JPK
#--------------------------------------------------------
from tables import *
import tables.parameters as para
import numpy as np
import time
nItems =100000000
f = open_file('test.hdf5', mode = 'w')
Table = np.zeros(nItems,dtype=([('A','f4'),('B','f4')]))
Table['A']= np.random.random_sample(nItems)
Table['B']= np.random.random_sample(nItems)
f.create_table(f.root,'Table',description=Table,expectedrows=nItems)
f.flush()
f.close()
#
#
f = open_file('test.hdf5', mode = 'r')
Table = f.root.Table
Table.get_where_list('((A > 0.2) & (A < 0.3)) | ((B > 0.5) & (B <
0.6))')
f.close()
t = time.clock()
para.NUMEXPR_MUM_THREADS = 1
test()
print time.clock() -t; t = time.clock()
para.NUMEXPR_MUM_THREADS = 2
test()
print time.clock() -t; t = time.clock()
para.NUMEXPR_MUM_THREADS = 3
test()
print time.clock() -t; t = time.clock()
#--------------------------------------------------------
Post by Jap Pyck
Hello,
I am new at pytables and particularly interested in the multi-thread
feature available through numexpr. I ran a few tests on a tables with
20 millions rows and 2 columns 'colA', 'colB' with MAX_NUMEXPR_THREADS set
to 1, 2 and 4. The test was run on a MacPro 8cores.
Although I see the number of threads increases during the execution of
the query, the performance does not change.
It looks very similar to the limitation imposed by the GIL on
multi-threaded functions in python. I though that HDF5 could get around
this limitation.
Question: is there anything I missed or do wrong? Do I need to do
additional configuration changes to see a performance increase?
Thank you for your help
-JPK
--
You received this message because you are subscribed to the Google Groups
"pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
For more options, visit https://groups.google.com/d/optout.
--
Francesc Alted
--
You received this message because you are subscribed to the Google Groups "pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pytables-users+***@googlegroups.com.
To post to this group, send an email to pytables-***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...