[pytables-users] [IMPORTANT] How to contribute to the future of PyTables?

[Francesc Alted]

Hi there,

We have been recently discussing about how to put more resources in the
maintenance of PyTables:

https://groups.google.com/forum/#!topic/pytables-dev/6fCsVzO6Ayg

Although Antonio Valentino and Anthony Scopatz have been doing a terrific
work maintaining the library, it seems like nowadays they are too busy to
continue putting time in PyTables anymore. We know that there are many
projects out there using the library, and we are in a point now where we
should decide what to do with the beast.

My first thought is to ask for current users that have benefited from using
PyTables in the past or are planning to continue using it, to contribute
economically to the sustainability of the project. For this, perhaps a
nice initiative would be to setup a site where people can do contributions
to the project (in the style of https://www.kickstarter.com or
http://www.verkami.com/), mainly for maintenance or even for new
developments. That would be a good way to measure the interest in keeping
PyTables alive, IMO. If you think that is a good idea, we would like to
hear about you. If you think there are better ways for doing the
maintenance, we are interested too.

As you know, PyTables is software that allows you not only to store and
retrieve data in HDF5, but implements a series of features that are not in
HDF5, like indexing or out-of-core expression evaluation, that set it apart
from other solutions. I personally think this is worth to be kept alive,
but it is ultimately you who has to decide if the project must go on.

Thanks,

[faltet]

Ok, so I suppose this 2 week-long silence should be interpreted as there is
not that much interest in keeping PyTables going on. Although I must
recognize that this is a bit sad for me as the original creator and
long-time maintainer of the project, one should accept when it is time to
let a project die as such is the fate of open source (and life in general),
and just acknowledge that the trip has been worth it.

Personally, what I am going to do is to take the time to release 3.2.0 as
soon as possible (will release a RC2 probably today and a final release
next week), and then basically stop the maintenance of it. Just for not
letting people down immediately, those out there that still are using
PyTables and want some basic maintenance I can still be contacted privately
to do some work on it as a freelance.

Thanks to everybody, it has been a good ride, and please continue enjoying
data! :)
Francesc

Post by Francesc Alted
Hi there,
We have been recently discussing about how to put more resources in the
https://groups.google.com/forum/#!topic/pytables-dev/6fCsVzO6Ayg
Although Antonio Valentino and Anthony Scopatz have been doing a terrific
work maintaining the library, it seems like nowadays they are too busy to
continue putting time in PyTables anymore. We know that there are many
projects out there using the library, and we are in a point now where we
should decide what to do with the beast.
My first thought is to ask for current users that have benefited from
using PyTables in the past or are planning to continue using it, to
contribute economically to the sustainability of the project. For this,
perhaps a nice initiative would be to setup a site where people can do
contributions to the project (in the style of https://www.kickstarter.com
or http://www.verkami.com/), mainly for maintenance or even for new
developments. That would be a good way to measure the interest in keeping
PyTables alive, IMO. If you think that is a good idea, we would like to
hear about you. If you think there are better ways for doing the
maintenance, we are interested too.
As you know, PyTables is software that allows you not only to store and
retrieve data in HDF5, but implements a series of features that are not in
HDF5, like indexing or out-of-core expression evaluation, that set it apart
from other solutions. I personally think this is worth to be kept alive,
but it is ultimately you who has to decide if the project must go on.
Thanks,
--
Francesc Alted

[Andrew Collette]

Hi Francesc,

I suspect there's still a great deal of interest in PyTables, but it may
have a different form than in years past. PyTables and h5py are now mature
projects that are substantially feature-complete, with less "green field"
available for people to contribute totally new things. So the average
NumPy/Python developer may be less engaged these days. That raises
problems for Kickstarter-style solutions, which depend on broad mass-market
appeal.

However, PyTables is a critical dependency for a number of high-profile
projects, including pandas. You (or Antonio) might try contacting a few
key people on those projects directly, since they have a vested interest in
seeing PyTables in good shape. I'm almost certain someone would be willing
to take over at least as a nominal maintenance contact. There are lots of
people who don't want a PyTables-shaped hole in their projects, and given
the compelling feature set PyTables adds to HDF5, porting to e.g. h5py is
not going to be an option for most of them.

Andrew

Post by faltet
Ok, so I suppose this 2 week-long silence should be interpreted as there
is not that much interest in keeping PyTables going on. Although I must
recognize that this is a bit sad for me as the original creator and
long-time maintainer of the project, one should accept when it is time to
let a project die as such is the fate of open source (and life in general),
and just acknowledge that the trip has been worth it.
Personally, what I am going to do is to take the time to release 3.2.0 as
soon as possible (will release a RC2 probably today and a final release
next week), and then basically stop the maintenance of it. Just for not
letting people down immediately, those out there that still are using
PyTables and want some basic maintenance I can still be contacted privately
to do some work on it as a freelance.
Thanks to everybody, it has been a good ride, and please continue enjoying
data! :)
Francesc

Post by Francesc Alted
Hi there,
We have been recently discussing about how to put more resources in the
https://groups.google.com/forum/#!topic/pytables-dev/6fCsVzO6Ayg
Although Antonio Valentino and Anthony Scopatz have been doing a terrific
work maintaining the library, it seems like nowadays they are too busy to
continue putting time in PyTables anymore. We know that there are many
projects out there using the library, and we are in a point now where we
should decide what to do with the beast.
My first thought is to ask for current users that have benefited from
using PyTables in the past or are planning to continue using it, to
contribute economically to the sustainability of the project. For this,
perhaps a nice initiative would be to setup a site where people can do
contributions to the project (in the style of https://www.kickstarter.com
or http://www.verkami.com/), mainly for maintenance or even for new
developments. That would be a good way to measure the interest in keeping
PyTables alive, IMO. If you think that is a good idea, we would like to
hear about you. If you think there are better ways for doing the
maintenance, we are interested too.
As you know, PyTables is software that allows you not only to store and
retrieve data in HDF5, but implements a series of features that are not in
HDF5, like indexing or out-of-core expression evaluation, that set it apart
from other solutions. I personally think this is worth to be kept alive,
but it is ultimately you who has to decide if the project must go on.
Thanks,
--
Francesc Alted

[Anthony Scopatz]

Hi Francesc,

I agree with basically everything that Andrew has said. Also, let me say
that I regret having become so busy lately. I wish I had the opportunity
to work on PyTables more directly.

I have had a couple big picture thoughts about PyTables lately. The first
is that it might be possible to get monetary support from Numfocus or
elsewhere. SciPy is going through something similar, I believe. I think
that these maintenance problems are something of a challenge problem that
could be posed to the new data science centers in the US that rely on this
software stack.

Another thought that would solve the long term maintainability would be a
possible merger with h5py. The details of this would have to be worked out,
but they seem much better supported, though with a distinct enough feature
set that a consolidation could be good for everyone. Also I think that a
proposal to do this could get funding from a wider variety of different
sources. I don't really know how anyone feels about this, though... I'd be
happy to lead effort if there was interest on the PyTables side.

Be Well
Anthony

Post by Andrew Collette
Hi Francesc,
I suspect there's still a great deal of interest in PyTables, but it may
have a different form than in years past. PyTables and h5py are now mature
projects that are substantially feature-complete, with less "green field"
available for people to contribute totally new things. So the average
NumPy/Python developer may be less engaged these days. That raises
problems for Kickstarter-style solutions, which depend on broad mass-market
appeal.
However, PyTables is a critical dependency for a number of high-profile
projects, including pandas. You (or Antonio) might try contacting a few
key people on those projects directly, since they have a vested interest in
seeing PyTables in good shape. I'm almost certain someone would be willing
to take over at least as a nominal maintenance contact. There are lots of
people who don't want a PyTables-shaped hole in their projects, and given
the compelling feature set PyTables adds to HDF5, porting to e.g. h5py is
not going to be an option for most of them.
Andrew

Post by faltet
Ok, so I suppose this 2 week-long silence should be interpreted as there
is not that much interest in keeping PyTables going on. Although I must
recognize that this is a bit sad for me as the original creator and
long-time maintainer of the project, one should accept when it is time to
let a project die as such is the fate of open source (and life in general),
and just acknowledge that the trip has been worth it.
Personally, what I am going to do is to take the time to release 3.2.0 as
soon as possible (will release a RC2 probably today and a final release
next week), and then basically stop the maintenance of it. Just for not
letting people down immediately, those out there that still are using
PyTables and want some basic maintenance I can still be contacted privately
to do some work on it as a freelance.
Thanks to everybody, it has been a good ride, and please continue
enjoying data! :)
Francesc

Post by Francesc Alted
Hi there,
We have been recently discussing about how to put more resources in the
https://groups.google.com/forum/#!topic/pytables-dev/6fCsVzO6Ayg
Although Antonio Valentino and Anthony Scopatz have been doing a
terrific work maintaining the library, it seems like nowadays they are too
busy to continue putting time in PyTables anymore. We know that there are
many projects out there using the library, and we are in a point now where
we should decide what to do with the beast.
My first thought is to ask for current users that have benefited from
using PyTables in the past or are planning to continue using it, to
contribute economically to the sustainability of the project. For this,
perhaps a nice initiative would be to setup a site where people can do
contributions to the project (in the style of
https://www.kickstarter.com or http://www.verkami.com/), mainly for
maintenance or even for new developments. That would be a good way to
measure the interest in keeping PyTables alive, IMO. If you think that
is a good idea, we would like to hear about you. If you think there are
better ways for doing the maintenance, we are interested too.
As you know, PyTables is software that allows you not only to store and
retrieve data in HDF5, but implements a series of features that are not in
HDF5, like indexing or out-of-core expression evaluation, that set it apart
from other solutions. I personally think this is worth to be kept alive,
but it is ultimately you who has to decide if the project must go on.
Thanks,
--
Francesc Alted

--

You received this message because you are subscribed to the Google Groups
"pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[Francesc Alted]

Hi Anthony,

Post by Andrew Collette
Hi Francesc,
I agree with basically everything that Andrew has said. Also, let me say
that I regret having become so busy lately. I wish I had the opportunity
to work on PyTables more directly.

No worries. We know that you had bigger fishes to cope with :)

Post by Andrew Collette
Another thought that would solve the long term maintainability would be a
possible merger with h5py. The details of this would have to be worked out,
but they seem much better supported, though with a distinct enough feature
set that a consolidation could be good for everyone. Also I think that a
proposal to do this could get funding from a wider variety of different
sources. I don't really know how anyone feels about this, though... I'd be
happy to lead effort if there was interest on the PyTables side.

That's an interesting thought and something that was needed to be said
publicly. If I would be asked on how to tackle this, I don't think that a
proper merge between PyTables and h5py would be that good, mainly because
the result would be a heavy Godzilla with a lot of restrictions due to the
inherited technological debt (e.g. different ways to store strings).

But I can certainly see a new PyTables made on top of h5py implementing
just the searching capabilities (and maybe the out-of-core computational
capabilities, but these are not that important IMO). This would lead to
better separation of concerns, bringing the opportunity to rewrite the
aging 'where*' iterators in PyTables (they were coded more than 10 years
ago), reduce the codebase of PyTables immensely and ultimately would not
enforce h5py to adopt PyTables conventions for storage.

That could be a good start, but even in case we reach an agreement then
again, resources would be needed for this, and I am not sure where these
could be obtained from.

Francesc

Post by Andrew Collette
Be Well
Anthony
On Mon, Apr 27, 2015 at 10:08 AM Andrew Collette <

Post by Andrew Collette
Hi Francesc,
I suspect there's still a great deal of interest in PyTables, but it may
have a different form than in years past. PyTables and h5py are now mature
projects that are substantially feature-complete, with less "green field"
available for people to contribute totally new things. So the average
NumPy/Python developer may be less engaged these days. That raises
problems for Kickstarter-style solutions, which depend on broad mass-market
appeal.
However, PyTables is a critical dependency for a number of high-profile
projects, including pandas. You (or Antonio) might try contacting a few
key people on those projects directly, since they have a vested interest in
seeing PyTables in good shape. I'm almost certain someone would be willing
to take over at least as a nominal maintenance contact. There are lots of
people who don't want a PyTables-shaped hole in their projects, and given
the compelling feature set PyTables adds to HDF5, porting to e.g. h5py is
not going to be an option for most of them.
Andrew

Post by faltet
Ok, so I suppose this 2 week-long silence should be interpreted as there
is not that much interest in keeping PyTables going on. Although I must
recognize that this is a bit sad for me as the original creator and
long-time maintainer of the project, one should accept when it is time to
let a project die as such is the fate of open source (and life in general),
and just acknowledge that the trip has been worth it.
Personally, what I am going to do is to take the time to release 3.2.0
as soon as possible (will release a RC2 probably today and a final release
next week), and then basically stop the maintenance of it. Just for not
letting people down immediately, those out there that still are using
PyTables and want some basic maintenance I can still be contacted privately
to do some work on it as a freelance.
Thanks to everybody, it has been a good ride, and please continue
enjoying data! :)
Francesc

Post by Francesc Alted
Hi there,
We have been recently discussing about how to put more resources in the
https://groups.google.com/forum/#!topic/pytables-dev/6fCsVzO6Ayg
Although Antonio Valentino and Anthony Scopatz have been doing a
terrific work maintaining the library, it seems like nowadays they are too
busy to continue putting time in PyTables anymore. We know that there are
many projects out there using the library, and we are in a point now where
we should decide what to do with the beast.
My first thought is to ask for current users that have benefited from
using PyTables in the past or are planning to continue using it, to
contribute economically to the sustainability of the project. For this,
perhaps a nice initiative would be to setup a site where people can do
contributions to the project (in the style of
https://www.kickstarter.com or http://www.verkami.com/), mainly for
maintenance or even for new developments. That would be a good way to
measure the interest in keeping PyTables alive, IMO. If you think
that is a good idea, we would like to hear about you. If you think there
are better ways for doing the maintenance, we are interested too.
As you know, PyTables is software that allows you not only to store and
retrieve data in HDF5, but implements a series of features that are not in
HDF5, like indexing or out-of-core expression evaluation, that set it apart
from other solutions. I personally think this is worth to be kept alive,
but it is ultimately you who has to decide if the project must go on.
Thanks,
--
Francesc Alted

--

You received this message because you are subscribed to the Google Groups
"pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[Jeff Reback]

Hey all,

I just wanted to have a big thank you all of the contributors to PyTables
over the years, esp Antonio, Anthony and Francesc.

I originally wrote pandas HDFStore on top of PyTables. It turned out to be
very easy to implement.

Over the years, pandas users have only reported a very small number of bugs
(which were quickly fixed). I have actually viewed PyTables as a pretty
mature / stable base for quite a while (even with the shifts in 3.0.0). It
is also IMHO pretty feature complete, and bug-minimal. I know everyone would
like real support of variable length strings. But this hasn't stopped
innovation on top of PyTables.

Unfortunately, I am lacking in any real solution to the maintenance burden.
I will forward this thread to NumFocus and see if anything bears fruit.

thanks

Jeff Reback

Post by Francesc Alted
Hi Anthony,

Post by Andrew Collette
Hi Francesc,
I agree with basically everything that Andrew has said. Also, let me say
that I regret having become so busy lately. I wish I had the opportunity
to work on PyTables more directly.

No worries. We know that you had bigger fishes to cope with :)

Post by Andrew Collette
Another thought that would solve the long term maintainability would be a
possible merger with h5py. The details of this would have to be worked out,
but they seem much better supported, though with a distinct enough feature
set that a consolidation could be good for everyone. Also I think that a
proposal to do this could get funding from a wider variety of different
sources. I don't really know how anyone feels about this, though... I'd be
happy to lead effort if there was interest on the PyTables side.

That's an interesting thought and something that was needed to be said
publicly. If I would be asked on how to tackle this, I don't think that a
proper merge between PyTables and h5py would be that good, mainly because
the result would be a heavy Godzilla with a lot of restrictions due to the
inherited technological debt (e.g. different ways to store strings).
But I can certainly see a new PyTables made on top of h5py implementing
just the searching capabilities (and maybe the out-of-core computational
capabilities, but these are not that important IMO). This would lead to
better separation of concerns, bringing the opportunity to rewrite the
aging 'where*' iterators in PyTables (they were coded more than 10 years
ago), reduce the codebase of PyTables immensely and ultimately would not
enforce h5py to adopt PyTables conventions for storage.
That could be a good start, but even in case we reach an agreement then
again, resources would be needed for this, and I am not sure where these
could be obtained from.
Francesc

Post by Andrew Collette
Be Well
Anthony

Post by Andrew Collette
Hi Francesc,
I suspect there's still a great deal of interest in PyTables, but it may
have a different form than in years past. PyTables and h5py are now mature
projects that are substantially feature-complete, with less "green field"
available for people to contribute totally new things. So the average
NumPy/Python developer may be less engaged these days. That raises
problems for Kickstarter-style solutions, which depend on broad mass-market
appeal.
However, PyTables is a critical dependency for a number of high-profile
projects, including pandas. You (or Antonio) might try contacting a few
key people on those projects directly, since they have a vested interest in
seeing PyTables in good shape. I'm almost certain someone would be willing
to take over at least as a nominal maintenance contact. There are lots of
people who don't want a PyTables-shaped hole in their projects, and given
the compelling feature set PyTables adds to HDF5, porting to e.g. h5py is
not going to be an option for most of them.
Andrew

Post by faltet
Ok, so I suppose this 2 week-long silence should be interpreted as
there is not that much interest in keeping PyTables going on. Although I
must recognize that this is a bit sad for me as the original creator and
long-time maintainer of the project, one should accept when it is time to
let a project die as such is the fate of open source (and life in general),
and just acknowledge that the trip has been worth it.
Personally, what I am going to do is to take the time to release 3.2.0
as soon as possible (will release a RC2 probably today and a final release
next week), and then basically stop the maintenance of it. Just for not
letting people down immediately, those out there that still are using
PyTables and want some basic maintenance I can still be contacted privately
to do some work on it as a freelance.
Thanks to everybody, it has been a good ride, and please continue
enjoying data! :)
Francesc

Post by Francesc Alted
Hi there,
We have been recently discussing about how to put more resources in
https://groups.google.com/forum/#!topic/pytables-dev/6fCsVzO6Ayg
Although Antonio Valentino and Anthony Scopatz have been doing a
terrific work maintaining the library, it seems like nowadays they are too
busy to continue putting time in PyTables anymore. We know that there are
many projects out there using the library, and we are in a point now where
we should decide what to do with the beast.
My first thought is to ask for current users that have benefited from
using PyTables in the past or are planning to continue using it, to
contribute economically to the sustainability of the project. For this,
perhaps a nice initiative would be to setup a site where people can
do contributions to the project (in the style of
https://www.kickstarter.com or http://www.verkami.com/), mainly for
maintenance or even for new developments. That would be a good way to
measure the interest in keeping PyTables alive, IMO. If you think
that is a good idea, we would like to hear about you. If you think there
are better ways for doing the maintenance, we are interested too.
As you know, PyTables is software that allows you not only to store
and retrieve data in HDF5, but implements a series of features that are not
in HDF5, like indexing or out-of-core expression evaluation, that set it
apart from other solutions. I personally think this is worth to be kept
alive, but it is ultimately you who has to decide if the project must go on.
Thanks,
--
Francesc Alted

--

You received this message because you are subscribed to the Google
Groups "pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send
<javascript:>.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
For more options, visit https://groups.google.com/d/optout.

--
Francesc Alted

[Andrew Collette]

Hi Francesc,

Post by Francesc Alted
That could be a good start, but even in case we reach an agreement then
again, resources would be needed for this, and I am not sure where these
could be obtained from.

I think this is an excellent point. Whatever solution people converge
on, at the end of the pipeline there need to be one or two people
writing PyTables code or at the very least responding on the bug
tracker and approving pull requests. But the first step is to
identify the interested parties... people with something to lose if
PyTables becomes unmaintained.

Btw, I will be at SciPy in July, and so will a couple of people from
the HDF Group.

Andrew

[Anthony Scopatz]

Hi All,

I want to second some of what Jeff has said. I do consider PyTables to be
stable and not in need of a lot of constant love and attention. However,
it does need *someone* occasionally, to Francesc's point.

Andrew, I think there are a number of interested parties that rely or use
on PyTables (pandas, banks, pyne [a project I do a lot of work on]) etc.
However, I think it is also useful to consider projects that might benefit
from pytables-like features in h5py. yt is the one that jumps out at me
here, though there are likely others.

And for the record, I always thought that a potential merger would look
similar to what Francesc described - whether or not the PyTables side lived
on as its own project or not. My argument for a full merger rests on the
easier-to-maintain and easier-to-find-funding statements.

I was sort of on the fence about coming to SciPy 2015, this has tipped it
over. If we set aside a healthy chunk of time to talk about this issue one
way or the other. Francesc, are you planning on going? If not, I am sure
we can remote you and Antonio and anyone else in....

Again, I think we need a plan or a set of plans that we can propose to
potential sponsors.

Be Well
Anthony

Post by Andrew Collette
Hi Francesc,

Post by Francesc Alted
That could be a good start, but even in case we reach an agreement then
again, resources would be needed for this, and I am not sure where these
could be obtained from.

I think this is an excellent point. Whatever solution people converge
on, at the end of the pipeline there need to be one or two people
writing PyTables code or at the very least responding on the bug
tracker and approving pull requests. But the first step is to
identify the interested parties... people with something to lose if
PyTables becomes unmaintained.
Btw, I will be at SciPy in July, and so will a couple of people from
the HDF Group.
Andrew

[Francesc Alted]

I don't think I completely agree in that PyTables does need someone just
occasionally. My experience with releasing PyTables 3.2 is that there have
been a good bunch of important bugs and improvement suggestions sleeping
and accumulating during the last years (just have a look at the recent
tickets that I closed, and how many there remains that still require a good
deal of love), and that is quite unfortunate. Also, the amount of time
that Antonio put in implementing Python 3 support (not only for PyTables,
but *also* for numexpr) should have been pretty heroic. Let me re-enforce
the opinion that, in order to provide good feedback to users, and hence
maintaining the project alive and in good shape, we do need more than just
someone occasionally putting time on that. I would say that having at
least a half-time person dedicated to this would be more realistic. I know
that this may sound a bit strong to some ears, but that's my opinion.

Regarding a possible meeting with Andrew, SciPy being held in the US makes
European guys more difficult to attend. But I am +1 for a video-conference
which, not being the same, is pretty inexpensive and can be equally
productive.

Francesc

Post by Anthony Scopatz
Hi All,
I want to second some of what Jeff has said. I do consider PyTables to be
stable and not in need of a lot of constant love and attention. However,
it does need *someone* occasionally, to Francesc's point.
Andrew, I think there are a number of interested parties that rely or use
on PyTables (pandas, banks, pyne [a project I do a lot of work on]) etc.
However, I think it is also useful to consider projects that might benefit
from pytables-like features in h5py. yt is the one that jumps out at me
here, though there are likely others.
And for the record, I always thought that a potential merger would look
similar to what Francesc described - whether or not the PyTables side lived
on as its own project or not. My argument for a full merger rests on the
easier-to-maintain and easier-to-find-funding statements.
I was sort of on the fence about coming to SciPy 2015, this has tipped it
over. If we set aside a healthy chunk of time to talk about this issue one
way or the other. Francesc, are you planning on going? If not, I am sure
we can remote you and Antonio and anyone else in....
Again, I think we need a plan or a set of plans that we can propose to
potential sponsors.
Be Well
Anthony
On Tue, Apr 28, 2015 at 11:25 AM Andrew Collette <

Post by Andrew Collette
Hi Francesc,

Post by Francesc Alted
That could be a good start, but even in case we reach an agreement then
again, resources would be needed for this, and I am not sure where these
could be obtained from.

I think this is an excellent point. Whatever solution people converge
on, at the end of the pipeline there need to be one or two people
writing PyTables code or at the very least responding on the bug
tracker and approving pull requests. But the first step is to
identify the interested parties... people with something to lose if
PyTables becomes unmaintained.
Btw, I will be at SciPy in July, and so will a couple of people from
the HDF Group.
Andrew

[Andrew Collette]

Hi Francesc,

Post by Francesc Alted
Regarding a possible meeting with Andrew, SciPy being held in the US makes
European guys more difficult to attend. But I am +1 for a video-conference
which, not being the same, is pretty inexpensive and can be equally
productive.

Sounds like a good solution. I'll be there all three days of the main
conference session.

Andrew

[Anthony Scopatz]

Sounds good to me.

Post by Andrew Collette
Hi Francesc,

Post by Francesc Alted
Regarding a possible meeting with Andrew, SciPy being held in the US

makes

Post by Francesc Alted
European guys more difficult to attend. But I am +1 for a

video-conference

Post by Francesc Alted
which, not being the same, is pretty inexpensive and can be equally
productive.

Sounds like a good solution. I'll be there all three days of the main
conference session.
Andrew

[Matthew Turk]

Hi Anthony,

Post by Anthony Scopatz
Hi All,
I want to second some of what Jeff has said. I do consider PyTables to be
stable and not in need of a lot of constant love and attention. However,
it does need *someone* occasionally, to Francesc's point.
Andrew, I think there are a number of interested parties that rely or use
on PyTables (pandas, banks, pyne [a project I do a lot of work on]) etc.
However, I think it is also useful to consider projects that might benefit
from pytables-like features in h5py. yt is the one that jumps out at me
here, though there are likely others.

I agree with your assessment that yt may benefit from functions in
PyTables; in fact, our trajectory originally went PyTables -> hand-coded-C
+ some h5py -> pure h5py. We've succeeded in pushing a great deal of data
selection back into the IO layer in recent months, and PyTables has been on
at least my mind quite a lot recently.

What would the best way to explore this type of project be? Would it be to
engage with groups like the HDF Group, to try to find interested project
members that could participate more in PyTables, to try to explore a joint
project with h5py, PyTables, etc? What I'm getting at is, what's the best
way to help, from the perspective of concerned non-developers?

Thanks,

Matt

Post by Anthony Scopatz
And for the record, I always thought that a potential merger would look
similar to what Francesc described - whether or not the PyTables side lived
on as its own project or not. My argument for a full merger rests on the
easier-to-maintain and easier-to-find-funding statements.
I was sort of on the fence about coming to SciPy 2015, this has tipped it
over. If we set aside a healthy chunk of time to talk about this issue one
way or the other. Francesc, are you planning on going? If not, I am sure
we can remote you and Antonio and anyone else in....
Again, I think we need a plan or a set of plans that we can propose to
potential sponsors.
Be Well
Anthony

Post by Andrew Collette
Hi Francesc,

Post by Francesc Alted
That could be a good start, but even in case we reach an agreement then
again, resources would be needed for this, and I am not sure where these
could be obtained from.

I think this is an excellent point. Whatever solution people converge
on, at the end of the pipeline there need to be one or two people
writing PyTables code or at the very least responding on the bug
tracker and approving pull requests. But the first step is to
identify the interested parties... people with something to lose if
PyTables becomes unmaintained.
Btw, I will be at SciPy in July, and so will a couple of people from
the HDF Group.
Andrew

[m***@hdfgroup.org]

On behalf of The HDF Group, I would like to say that countless people have
benefited from PyTables, but I nobody more than the HDF Group. There is a
good chance our company would not still exist were not for Fransesc and his
colleagues’ creation.

Please count us among those who want very much to keep PyTables alive and
well. We want to be part of the conversation, and we have some resources
we can apply to the effort.

As Andrew mentioned we will have someone at SciPy 2015. They will come
ready to discuss ways we can be involved.

Mike Folk

Post by Francesc Alted
Hi Anthony,

Post by Anthony Scopatz
Hi All,
I want to second some of what Jeff has said. I do consider PyTables to
be stable and not in need of a lot of constant love and attention.
However, it does need *someone* occasionally, to Francesc's point.
Andrew, I think there are a number of interested parties that rely or use
on PyTables (pandas, banks, pyne [a project I do a lot of work on]) etc.
However, I think it is also useful to consider projects that might benefit
from pytables-like features in h5py. yt is the one that jumps out at me
here, though there are likely others.

I agree with your assessment that yt may benefit from functions in
PyTables; in fact, our trajectory originally went PyTables -> hand-coded-C
+ some h5py -> pure h5py. We've succeeded in pushing a great deal of data
selection back into the IO layer in recent months, and PyTables has been on
at least my mind quite a lot recently.
What would the best way to explore this type of project be? Would it be
to engage with groups like the HDF Group, to try to find interested project
members that could participate more in PyTables, to try to explore a joint
project with h5py, PyTables, etc? What I'm getting at is, what's the best
way to help, from the perspective of concerned non-developers?
Thanks,
Matt

Post by Anthony Scopatz
And for the record, I always thought that a potential merger would look
similar to what Francesc described - whether or not the PyTables side lived
on as its own project or not. My argument for a full merger rests on the
easier-to-maintain and easier-to-find-funding statements.
I was sort of on the fence about coming to SciPy 2015, this has tipped it
over. If we set aside a healthy chunk of time to talk about this issue one
way or the other. Francesc, are you planning on going? If not, I am sure
we can remote you and Antonio and anyone else in....
Again, I think we need a plan or a set of plans that we can propose to
potential sponsors.
Be Well
Anthony

Post by Andrew Collette
Hi Francesc,

Post by Francesc Alted
That could be a good start, but even in case we reach an agreement then
again, resources would be needed for this, and I am not sure where

these

Post by Francesc Alted
could be obtained from.

I think this is an excellent point. Whatever solution people converge
on, at the end of the pipeline there need to be one or two people
writing PyTables code or at the very least responding on the bug
tracker and approving pull requests. But the first step is to
identify the interested parties... people with something to lose if
PyTables becomes unmaintained.
Btw, I will be at SciPy in July, and so will a couple of people from
the HDF Group.
Andrew

[Francesc Alted]

Hi Mike,

Thanks for your kind message. Sometimes people doing software maintenance
are only aware of complains of things not working and typically nobody says
how useful a tool is being for them. As others said, this happens even
more with mature products, so it is quite natural that maintainers can get
tired of trying to fix things with no other reward that a very scarce
community appreciation.

Glad that the HDF Group would be interested in sponsoring future PyTables
maintainers. I won't be in SciPy, but apparently Anthony Scopatz will, and
I can try to be available for some remote (if needed).

Best,
Francesc

Post by m***@hdfgroup.org
On behalf of The HDF Group, I would like to say that countless people have
benefited from PyTables, but I nobody more than the HDF Group. There is a
good chance our company would not still exist were not for Fransesc and his
colleagues’ creation.
Please count us among those who want very much to keep PyTables alive and
well. We want to be part of the conversation, and we have some resources
we can apply to the effort.
As Andrew mentioned we will have someone at SciPy 2015. They will come
ready to discuss ways we can be involved.
Mike Folk

Post by Francesc Alted
Hi Anthony,

Post by Anthony Scopatz
Hi All,
I want to second some of what Jeff has said. I do consider PyTables to
be stable and not in need of a lot of constant love and attention.
However, it does need *someone* occasionally, to Francesc's point.
Andrew, I think there are a number of interested parties that rely or
use on PyTables (pandas, banks, pyne [a project I do a lot of work on])
etc. However, I think it is also useful to consider projects that might
benefit from pytables-like features in h5py. yt is the one that jumps out
at me here, though there are likely others.

I agree with your assessment that yt may benefit from functions in
PyTables; in fact, our trajectory originally went PyTables -> hand-coded-C
+ some h5py -> pure h5py. We've succeeded in pushing a great deal of data
selection back into the IO layer in recent months, and PyTables has been on
at least my mind quite a lot recently.
What would the best way to explore this type of project be? Would it be
to engage with groups like the HDF Group, to try to find interested project
members that could participate more in PyTables, to try to explore a joint
project with h5py, PyTables, etc? What I'm getting at is, what's the best
way to help, from the perspective of concerned non-developers?
Thanks,
Matt

Post by Anthony Scopatz
And for the record, I always thought that a potential merger would look
similar to what Francesc described - whether or not the PyTables side lived
on as its own project or not. My argument for a full merger rests on the
easier-to-maintain and easier-to-find-funding statements.
I was sort of on the fence about coming to SciPy 2015, this has tipped
it over. If we set aside a healthy chunk of time to talk about this issue
one way or the other. Francesc, are you planning on going? If not, I am
sure we can remote you and Antonio and anyone else in....
Again, I think we need a plan or a set of plans that we can propose to
potential sponsors.
Be Well
Anthony

Post by Andrew Collette
Hi Francesc,

Post by Francesc Alted
That could be a good start, but even in case we reach an agreement

then

Post by Francesc Alted
again, resources would be needed for this, and I am not sure where

these

Post by Francesc Alted
could be obtained from.

I think this is an excellent point. Whatever solution people converge
on, at the end of the pipeline there need to be one or two people
writing PyTables code or at the very least responding on the bug
tracker and approving pull requests. But the first step is to
identify the interested parties... people with something to lose if
PyTables becomes unmaintained.
Btw, I will be at SciPy in July, and so will a couple of people from
the HDF Group.
Andrew

[Francesc Alted]

Oops, this was meant to be sent to the list...

---------- Forwarded message ----------
From: Francesc Alted <***@gmail.com>
Date: 2015-04-28 11:44 GMT+02:00
Subject: Re: [pytables-users] Re: [IMPORTANT] How to contribute to the
future of PyTables?
To: Andrew Collette <***@gmail.com>


Hi Andrew,

Thanks for your thoughts on this. While I admit that not everybody using
PyTables is subscribed to this list, it is also true that my tweet about
the situation has been retweeted a good number of times:

https://twitter.com/search?q=%20%20PyTables%3A%20worth%20keeping%20it%20alive&src=typd

and I know that others have tweeted independently as well, so I would say
that a significant share of the PyData community is aware of the problem,
and the fact that not a single user contributed opinions is quite telling
IMO. But as I said in my previous message, there is nothing wrong with that
and I really believe in natural selection (at least in software, not in
human society where the situations can be quite different :).

Regarding h5py not ready for coping the needs of pandas users, I think that
is a nice opportunity for improving it (but more on this on my other reply
to Anthony).

Cheers,
Francesc

Post by Andrew Collette
Hi Francesc,
I suspect there's still a great deal of interest in PyTables, but it may
have a different form than in years past. PyTables and h5py are now mature
projects that are substantially feature-complete, with less "green field"
available for people to contribute totally new things. So the average
NumPy/Python developer may be less engaged these days. That raises
problems for Kickstarter-style solutions, which depend on broad mass-market
appeal.
However, PyTables is a critical dependency for a number of high-profile
projects, including pandas. You (or Antonio) might try contacting a few
key people on those projects directly, since they have a vested interest in
seeing PyTables in good shape. I'm almost certain someone would be willing
to take over at least as a nominal maintenance contact. There are lots of
people who don't want a PyTables-shaped hole in their projects, and given
the compelling feature set PyTables adds to HDF5, porting to e.g. h5py is
not going to be an option for most of them.
Andrew

Post by faltet
Ok, so I suppose this 2 week-long silence should be interpreted as there
is not that much interest in keeping PyTables going on. Although I must
recognize that this is a bit sad for me as the original creator and
long-time maintainer of the project, one should accept when it is time to
let a project die as such is the fate of open source (and life in general),
and just acknowledge that the trip has been worth it.
Personally, what I am going to do is to take the time to release 3.2.0 as
soon as possible (will release a RC2 probably today and a final release
next week), and then basically stop the maintenance of it. Just for not
letting people down immediately, those out there that still are using
PyTables and want some basic maintenance I can still be contacted privately
to do some work on it as a freelance.
Thanks to everybody, it has been a good ride, and please continue
enjoying data! :)
Francesc

Post by Francesc Alted
Hi there,
We have been recently discussing about how to put more resources in the
https://groups.google.com/forum/#!topic/pytables-dev/6fCsVzO6Ayg
Although Antonio Valentino and Anthony Scopatz have been doing a
terrific work maintaining the library, it seems like nowadays they are too
busy to continue putting time in PyTables anymore. We know that there are
many projects out there using the library, and we are in a point now where
we should decide what to do with the beast.
My first thought is to ask for current users that have benefited from
using PyTables in the past or are planning to continue using it, to
contribute economically to the sustainability of the project. For this,
perhaps a nice initiative would be to setup a site where people can do
contributions to the project (in the style of
https://www.kickstarter.com or http://www.verkami.com/), mainly for
maintenance or even for new developments. That would be a good way to
measure the interest in keeping PyTables alive, IMO. If you think that
is a good idea, we would like to hear about you. If you think there are
better ways for doing the maintenance, we are interested too.
As you know, PyTables is software that allows you not only to store and
retrieve data in HDF5, but implements a series of features that are not in
HDF5, like indexing or out-of-core expression evaluation, that set it apart
from other solutions. I personally think this is worth to be kept alive,
but it is ultimately you who has to decide if the project must go on.
Thanks,
--
Francesc Alted

--

You received this message because you are subscribed to the Google Groups
"pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
Francesc Alted

[Francesc Alted]

I think most of you should already know, but for archival purposes, let me
say that PyTables has reached the status of a sponsored project by NumFOCUS.
http://numfocus.org/news/2015/08/31/numfocus-announces-new-fiscally-sponsored-project-pytables/

That does not mean that the future of the project is solved, but rather
that you can do donations via NumFOCUS and get fiscal advantages. Please
feel fee to suggest goals for PyTables and, specially, donate for keeping
it kicking and alive!

Thanks to all who has participated in this long discussion!

Francesc

Post by faltet
Ok, so I suppose this 2 week-long silence should be interpreted as there
is not that much interest in keeping PyTables going on. Although I must
recognize that this is a bit sad for me as the original creator and
long-time maintainer of the project, one should accept when it is time to
let a project die as such is the fate of open source (and life in general),
and just acknowledge that the trip has been worth it.
Personally, what I am going to do is to take the time to release 3.2.0 as
soon as possible (will release a RC2 probably today and a final release
next week), and then basically stop the maintenance of it. Just for not
letting people down immediately, those out there that still are using
PyTables and want some basic maintenance I can still be contacted privately
to do some work on it as a freelance.
Thanks to everybody, it has been a good ride, and please continue enjoying
data! :)
Francesc

Post by Francesc Alted
Hi there,
We have been recently discussing about how to put more resources in the
https://groups.google.com/forum/#!topic/pytables-dev/6fCsVzO6Ayg
Although Antonio Valentino and Anthony Scopatz have been doing a terrific
work maintaining the library, it seems like nowadays they are too busy to
continue putting time in PyTables anymore. We know that there are many
projects out there using the library, and we are in a point now where we
should decide what to do with the beast.
My first thought is to ask for current users that have benefited from
using PyTables in the past or are planning to continue using it, to
contribute economically to the sustainability of the project. For this,
perhaps a nice initiative would be to setup a site where people can do
contributions to the project (in the style of https://www.kickstarter.com
or http://www.verkami.com/), mainly for maintenance or even for new
developments. That would be a good way to measure the interest in keeping
PyTables alive, IMO. If you think that is a good idea, we would like to
hear about you. If you think there are better ways for doing the
maintenance, we are interested too.
As you know, PyTables is software that allows you not only to store and
retrieve data in HDF5, but implements a series of features that are not in
HDF5, like indexing or out-of-core expression evaluation, that set it apart
from other solutions. I personally think this is worth to be kept alive,
but it is ultimately you who has to decide if the project must go on.
Thanks,
--
Francesc Alted

[Francesc Alted]

Oh, I forgot to mention that we already have a donate button ready to use
at:

http://www.pytables.org/

Thanks!

Post by Francesc Alted
I think most of you should already know, but for archival purposes, let me
say that PyTables has reached the status of a sponsored project by NumFOCUS.
http://numfocus.org/news/2015/08/31/numfocus-announces-new-fiscally-sponsored-project-pytables/
That does not mean that the future of the project is solved, but rather
that you can do donations via NumFOCUS and get fiscal advantages. Please
feel fee to suggest goals for PyTables and, specially, donate for keeping
it kicking and alive!
Thanks to all who has participated in this long discussion!
Francesc

Post by faltet
Ok, so I suppose this 2 week-long silence should be interpreted as there
is not that much interest in keeping PyTables going on. Although I must
recognize that this is a bit sad for me as the original creator and
long-time maintainer of the project, one should accept when it is time to
let a project die as such is the fate of open source (and life in general),
and just acknowledge that the trip has been worth it.
Personally, what I am going to do is to take the time to release 3.2.0 as
soon as possible (will release a RC2 probably today and a final release
next week), and then basically stop the maintenance of it. Just for not
letting people down immediately, those out there that still are using
PyTables and want some basic maintenance I can still be contacted privately
to do some work on it as a freelance.
Thanks to everybody, it has been a good ride, and please continue
enjoying data! :)
Francesc

Post by Francesc Alted
Hi there,
We have been recently discussing about how to put more resources in the
https://groups.google.com/forum/#!topic/pytables-dev/6fCsVzO6Ayg
Although Antonio Valentino and Anthony Scopatz have been doing a
terrific work maintaining the library, it seems like nowadays they are too
busy to continue putting time in PyTables anymore. We know that there are
many projects out there using the library, and we are in a point now where
we should decide what to do with the beast.
My first thought is to ask for current users that have benefited from
using PyTables in the past or are planning to continue using it, to
contribute economically to the sustainability of the project. For this,
perhaps a nice initiative would be to setup a site where people can do
contributions to the project (in the style of
https://www.kickstarter.com or http://www.verkami.com/), mainly for
maintenance or even for new developments. That would be a good way to
measure the interest in keeping PyTables alive, IMO. If you think that
is a good idea, we would like to hear about you. If you think there are
better ways for doing the maintenance, we are interested too.
As you know, PyTables is software that allows you not only to store and
retrieve data in HDF5, but implements a series of features that are not in
HDF5, like indexing or out-of-core expression evaluation, that set it apart
from other solutions. I personally think this is worth to be kept alive,
but it is ultimately you who has to decide if the project must go on.
Thanks,
--
Francesc Alted

--
Francesc Alted

[mythsmith]

Hi Francesc,
I have an quite strong interest in the future of pytables.
I think HDF5 is one of the best formats for scientific applications, and
pytables makes viable to easily use it with unmatched performance.

I don't think a kickstarter-style campaign will meet any success. As
somebody said in the dev list, pytables is not a mass-market product
capable of exciting a large number of people. Moreover, it is not new, and
already has a sort of more recent "concurrent".

Francesc, I came to your announcement via the HDFgroup mailing list, to
which I subscribed long ago for a small bug and never cared enough even to
unsubscribe.
I think very few of the actual pytable users will ever reach your message -
imagine a ks campaign!

On the countrary, a direct campaign towards existing users might increase
awareness and participation in the decisions about the future of this great
package.

Searching on github code language:Python:
"tables.openFile", 1.705 files
"tables.open_file", 773 files

Unfortunately github does not tell us the number of independent
repositories, but it should not be too difficult to use some sort of github
api to extract them and then directly contact them, one by one.

But I suspect even this action would reach a minimal subset of pytables
users, which I think use pytables for in-house projects in their
company/university, not on github. This is another reason why a ks-campaign
will most probably fail: even when aware of, they will hardly convince
their employer to give significative economical support.

A (not elegant) way to reach users and make them aware of the situation
would be to introduce a deprecation warning at import time or open_file,
which can be manually switched off some way.
This will be slow (will show only on lib upgrade), but will ultimately
reach everybody.

Best regards,
Daniele

ps: My application needs live data-intensive acquisition and analysis. I
considered a number of other formats, and also the h5py wrapper, and I
still strongly believe pytables is the best solution.
The most important feature missing for my case is the single-writer
multi-reader mode, experimentally supported in h5py. I partially solved
this by serializing access in a separate process.
Below a feature comparison with weights and votes for my case.


*Feature* *Weight* *pytables* *h5py* *sqlite* *directory* *zipfile* Efficient
slicing *10* 2 2 1 0 0 Efficient appending *10* 2 0 2 2 0 Format stability
*10* 2 1 2 2 2 Single-Writer Multi-Reader *9* 0 1 1 2 0 Searching, in-k
*8* 2 0 1 0 0 Compression *6* 2 2 1 0 2 Dependencies/ecosystem *5* 0 0 2 2
2 File-like objects *2* 2 0 1 2 0







*Result:* 7,7 4,3 7,1 6,0 3,5

Post by Francesc Alted
Hi there,
We have been recently discussing about how to put more resources in the
https://groups.google.com/forum/#!topic/pytables-dev/6fCsVzO6Ayg
Although Antonio Valentino and Anthony Scopatz have been doing a terrific
work maintaining the library, it seems like nowadays they are too busy to
continue putting time in PyTables anymore. We know that there are many
projects out there using the library, and we are in a point now where we
should decide what to do with the beast.
My first thought is to ask for current users that have benefited from
using PyTables in the past or are planning to continue using it, to
contribute economically to the sustainability of the project. For this,
perhaps a nice initiative would be to setup a site where people can do
contributions to the project (in the style of https://www.kickstarter.com
or http://www.verkami.com/), mainly for maintenance or even for new
developments. That would be a good way to measure the interest in keeping
PyTables alive, IMO. If you think that is a good idea, we would like to
hear about you. If you think there are better ways for doing the
maintenance, we are interested too.
As you know, PyTables is software that allows you not only to store and
retrieve data in HDF5, but implements a series of features that are not in
HDF5, like indexing or out-of-core expression evaluation, that set it apart
from other solutions. I personally think this is worth to be kept alive,
but it is ultimately you who has to decide if the project must go on.
Thanks,
--
Francesc Alted

[Francesc Alted]

Hi Daniele,

Thanks for your thoughts. You and Jeff are the first PyTables users that
are actually responding to this call, and the information that you provided
is very useful.

Your proposal of introducing a 'DeprecationWarning' may sound a bit strong,
but is nonetheless worth of consideration. But now that at least The HDF
Group has shown interest in sponsoring the maintenance of the project, I
think it is better to wait and see how things are evolving in the near
future.

Thanks again for the time that you took for demonstrating that PyTables is
still valuable for the community. We need more feedback like this!

Francesc

Post by Andrew Collette
Hi Francesc,
I have an quite strong interest in the future of pytables.
I think HDF5 is one of the best formats for scientific applications, and
pytables makes viable to easily use it with unmatched performance.
I don't think a kickstarter-style campaign will meet any success. As
somebody said in the dev list, pytables is not a mass-market product
capable of exciting a large number of people. Moreover, it is not new, and
already has a sort of more recent "concurrent".
Francesc, I came to your announcement via the HDFgroup mailing list, to
which I subscribed long ago for a small bug and never cared enough even to
unsubscribe.
I think very few of the actual pytable users will ever reach your message
- imagine a ks campaign!
On the countrary, a direct campaign towards existing users might increase
awareness and participation in the decisions about the future of this great
package.
"tables.openFile", 1.705 files
"tables.open_file", 773 files
Unfortunately github does not tell us the number of independent
repositories, but it should not be too difficult to use some sort of github
api to extract them and then directly contact them, one by one.
But I suspect even this action would reach a minimal subset of pytables
users, which I think use pytables for in-house projects in their
company/university, not on github. This is another reason why a ks-campaign
will most probably fail: even when aware of, they will hardly convince
their employer to give significative economical support.
A (not elegant) way to reach users and make them aware of the situation
would be to introduce a deprecation warning at import time or open_file,
which can be manually switched off some way.
This will be slow (will show only on lib upgrade), but will ultimately
reach everybody.
Best regards,
Daniele
ps: My application needs live data-intensive acquisition and analysis. I
considered a number of other formats, and also the h5py wrapper, and I
still strongly believe pytables is the best solution.
The most important feature missing for my case is the single-writer
multi-reader mode, experimentally supported in h5py. I partially solved
this by serializing access in a separate process.
Below a feature comparison with weights and votes for my case.
*Feature* *Weight* *pytables* *h5py* *sqlite* *directory* *zipfile* Efficient
slicing *10* 2 2 1 0 0 Efficient appending *10* 2 0 2 2 0 Format
stability *10* 2 1 2 2 2 Single-Writer Multi-Reader *9* 0 1 1 2 0 Searching,
in-k *8* 2 0 1 0 0 Compression *6* 2 2 1 0 2 Dependencies/ecosystem *5*
0 0 2 2 2 File-like objects *2* 2 0 1 2 0
*Result:* 7,7 4,3 7,1 6,0 3,5

Post by Francesc Alted
Hi there,
We have been recently discussing about how to put more resources in the
https://groups.google.com/forum/#!topic/pytables-dev/6fCsVzO6Ayg
Although Antonio Valentino and Anthony Scopatz have been doing a terrific
work maintaining the library, it seems like nowadays they are too busy to
continue putting time in PyTables anymore. We know that there are many
projects out there using the library, and we are in a point now where we
should decide what to do with the beast.
My first thought is to ask for current users that have benefited from
using PyTables in the past or are planning to continue using it, to
contribute economically to the sustainability of the project. For this,
perhaps a nice initiative would be to setup a site where people can do
contributions to the project (in the style of https://www.kickstarter.com
or http://www.verkami.com/), mainly for maintenance or even for new
developments. That would be a good way to measure the interest in keeping
PyTables alive, IMO. If you think that is a good idea, we would like to
hear about you. If you think there are better ways for doing the
maintenance, we are interested too.
As you know, PyTables is software that allows you not only to store and
retrieve data in HDF5, but implements a series of features that are not in
HDF5, like indexing or out-of-core expression evaluation, that set it apart
from other solutions. I personally think this is worth to be kept alive,
but it is ultimately you who has to decide if the project must go on.
Thanks,
--
Francesc Alted

[David Verelst]

Hi All,

I don't follow the pytables development, but I came here after reading the
latest release notes. As a user (mainly via pandas to_hdf interface) I
wanted to indicate how useful pytables is for me. At the university I work
we are finally moving towards open standards to store large amounts of
simulation data (I do wind turbine aeroelastic simulations in case you're
interested ;-), although we're not quite there yet.

Unfortunately, it is very difficult to convince the senior staff that using
open source also brings a certain responsibility. It is not easy to
"distract" funding to invest in something that is "just" tools. That
doesn't stop me and others from trying, but I am sure you've heard this
kind of story before. However, it is good to know that pytables is in need
for help, so I can makes others aware of it as well. I do not have the
right skill set to step up as a maintainer for a project such as pytables
(I am really struggling with my first PR for pandas :-), but I could help
support others in doing so. Please feel free to contact me if anything
comes up.

Let me use this opportunity to thank the pytables developers for creating
and maintaining this amazing piece of software. Francesc, I had once the
pleasure of attending your courses at the Python summer school (2009, St
Andrews) and I was most impressed by your lectures on numexpr and pytables,
so also another thanks for that!

Best regards,
David

Post by Francesc Alted
Hi Daniele,
Thanks for your thoughts. You and Jeff are the first PyTables users that
are actually responding to this call, and the information that you provided
is very useful.
Your proposal of introducing a 'DeprecationWarning' may sound a bit
strong, but is nonetheless worth of consideration. But now that at least
The HDF Group has shown interest in sponsoring the maintenance of the
project, I think it is better to wait and see how things are evolving in
the near future.
Thanks again for the time that you took for demonstrating that PyTables is
still valuable for the community. We need more feedback like this!
Francesc

Post by Andrew Collette
Hi Francesc,
I have an quite strong interest in the future of pytables.
I think HDF5 is one of the best formats for scientific applications, and
pytables makes viable to easily use it with unmatched performance.
I don't think a kickstarter-style campaign will meet any success. As
somebody said in the dev list, pytables is not a mass-market product
capable of exciting a large number of people. Moreover, it is not new, and
already has a sort of more recent "concurrent".
Francesc, I came to your announcement via the HDFgroup mailing list, to
which I subscribed long ago for a small bug and never cared enough even to
unsubscribe.
I think very few of the actual pytable users will ever reach your message
- imagine a ks campaign!
On the countrary, a direct campaign towards existing users might increase
awareness and participation in the decisions about the future of this great
package.
"tables.openFile", 1.705 files
"tables.open_file", 773 files
Unfortunately github does not tell us the number of independent
repositories, but it should not be too difficult to use some sort of github
api to extract them and then directly contact them, one by one.
But I suspect even this action would reach a minimal subset of pytables
users, which I think use pytables for in-house projects in their
company/university, not on github. This is another reason why a ks-campaign
will most probably fail: even when aware of, they will hardly convince
their employer to give significative economical support.
A (not elegant) way to reach users and make them aware of the situation
would be to introduce a deprecation warning at import time or open_file,
which can be manually switched off some way.
This will be slow (will show only on lib upgrade), but will ultimately
reach everybody.
Best regards,
Daniele
ps: My application needs live data-intensive acquisition and analysis. I
considered a number of other formats, and also the h5py wrapper, and I
still strongly believe pytables is the best solution.
The most important feature missing for my case is the single-writer
multi-reader mode, experimentally supported in h5py. I partially solved
this by serializing access in a separate process.
Below a feature comparison with weights and votes for my case.
*Feature* *Weight* *pytables* *h5py* *sqlite* *directory* *zipfile* Efficient
slicing *10* 2 2 1 0 0 Efficient appending *10* 2 0 2 2 0 Format
stability *10* 2 1 2 2 2 Single-Writer Multi-Reader *9* 0 1 1 2 0 Searching,
in-k *8* 2 0 1 0 0 Compression *6* 2 2 1 0 2 Dependencies/ecosystem *5*
0 0 2 2 2 File-like objects *2* 2 0 1 2 0
*Result:* 7,7 4,3 7,1 6,0 3,5

Post by Francesc Alted
Hi there,
We have been recently discussing about how to put more resources in the
https://groups.google.com/forum/#!topic/pytables-dev/6fCsVzO6Ayg
Although Antonio Valentino and Anthony Scopatz have been doing a
terrific work maintaining the library, it seems like nowadays they are too
busy to continue putting time in PyTables anymore. We know that there are
many projects out there using the library, and we are in a point now where
we should decide what to do with the beast.
My first thought is to ask for current users that have benefited from
using PyTables in the past or are planning to continue using it, to
contribute economically to the sustainability of the project. For this,
perhaps a nice initiative would be to setup a site where people can do
contributions to the project (in the style of
https://www.kickstarter.com or http://www.verkami.com/), mainly for
maintenance or even for new developments. That would be a good way to
measure the interest in keeping PyTables alive, IMO. If you think that
is a good idea, we would like to hear about you. If you think there are
better ways for doing the maintenance, we are interested too.
As you know, PyTables is software that allows you not only to store and
retrieve data in HDF5, but implements a series of features that are not in
HDF5, like indexing or out-of-core expression evaluation, that set it apart
from other solutions. I personally think this is worth to be kept alive,
but it is ultimately you who has to decide if the project must go on.
Thanks,
--
Francesc Alted

--
Francesc Alted

[Francesc Alted]

Hi David,

Thank you very much for showing up and telling us that PyTables is still
valuable for you.

And you are right, open source is becoming so 'embedded' in computer
sciences nowadays that people tend to think about it just as a 'facility'
that can be used for free with no more hassles to think about. Problem is,
of course, what happens when developers/maintainers do not have more free
time to put in the project.

The only solution, as I see it, is to transmit the problem to the community
and see if they are willing to do something about this. This has been
done, and for the moment only the HDF Group has shown interest in providing
some sponsorship (we still don't know how much and for how long though, and
that apparently will have to wait until next SciPy conference due in July
at least).

And glad to see that you attended to our course in St Andrews (but it was
2011: https://python.g-node.org/python-summerschool-2011/start). We are
always very happy to teach in the Summer School, but the best reward is to
see that it served to make more people using Python for scientific
applications.

Francesc

Post by Anthony Scopatz
Hi All,
I don't follow the pytables development, but I came here after reading the
latest release notes. As a user (mainly via pandas to_hdf interface) I
wanted to indicate how useful pytables is for me. At the university I work
we are finally moving towards open standards to store large amounts of
simulation data (I do wind turbine aeroelastic simulations in case you're
interested ;-), although we're not quite there yet.
Unfortunately, it is very difficult to convince the senior staff that
using open source also brings a certain responsibility. It is not easy to
"distract" funding to invest in something that is "just" tools. That
doesn't stop me and others from trying, but I am sure you've heard this
kind of story before. However, it is good to know that pytables is in need
for help, so I can makes others aware of it as well. I do not have the
right skill set to step up as a maintainer for a project such as pytables
(I am really struggling with my first PR for pandas :-), but I could help
support others in doing so. Please feel free to contact me if anything
comes up.
Let me use this opportunity to thank the pytables developers for creating
and maintaining this amazing piece of software. Francesc, I had once the
pleasure of attending your courses at the Python summer school (2009, St
Andrews) and I was most impressed by your lectures on numexpr and pytables,
so also another thanks for that!
Best regards,
David

Post by Francesc Alted
Hi Daniele,
Thanks for your thoughts. You and Jeff are the first PyTables users that
are actually responding to this call, and the information that you provided
is very useful.
Your proposal of introducing a 'DeprecationWarning' may sound a bit
strong, but is nonetheless worth of consideration. But now that at least
The HDF Group has shown interest in sponsoring the maintenance of the
project, I think it is better to wait and see how things are evolving in
the near future.
Thanks again for the time that you took for demonstrating that PyTables
is still valuable for the community. We need more feedback like this!
Francesc

Post by Andrew Collette
Hi Francesc,
I have an quite strong interest in the future of pytables.
I think HDF5 is one of the best formats for scientific applications, and
pytables makes viable to easily use it with unmatched performance.
I don't think a kickstarter-style campaign will meet any success. As
somebody said in the dev list, pytables is not a mass-market product
capable of exciting a large number of people. Moreover, it is not new, and
already has a sort of more recent "concurrent".
Francesc, I came to your announcement via the HDFgroup mailing list, to
which I subscribed long ago for a small bug and never cared enough even to
unsubscribe.
I think very few of the actual pytable users will ever reach your
message - imagine a ks campaign!
On the countrary, a direct campaign towards existing users might
increase awareness and participation in the decisions about the future of
this great package.
"tables.openFile", 1.705 files
"tables.open_file", 773 files
Unfortunately github does not tell us the number of independent
repositories, but it should not be too difficult to use some sort of github
api to extract them and then directly contact them, one by one.
But I suspect even this action would reach a minimal subset of pytables
users, which I think use pytables for in-house projects in their
company/university, not on github. This is another reason why a ks-campaign
will most probably fail: even when aware of, they will hardly convince
their employer to give significative economical support.
A (not elegant) way to reach users and make them aware of the situation
would be to introduce a deprecation warning at import time or open_file,
which can be manually switched off some way.
This will be slow (will show only on lib upgrade), but will ultimately
reach everybody.
Best regards,
Daniele
ps: My application needs live data-intensive acquisition and analysis. I
considered a number of other formats, and also the h5py wrapper, and I
still strongly believe pytables is the best solution.
The most important feature missing for my case is the single-writer
multi-reader mode, experimentally supported in h5py. I partially solved
this by serializing access in a separate process.
Below a feature comparison with weights and votes for my case.
*Feature* *Weight* *pytables* *h5py* *sqlite* *directory* *zipfile* Efficient
slicing *10* 2 2 1 0 0 Efficient appending *10* 2 0 2 2 0 Format
stability *10* 2 1 2 2 2 Single-Writer Multi-Reader *9* 0 1 1 2 0 Searching,
in-k *8* 2 0 1 0 0 Compression *6* 2 2 1 0 2 Dependencies/ecosystem
*5* 0 0 2 2 2 File-like objects *2* 2 0 1 2 0
*Result:* 7,7 4,3 7,1 6,0 3,5

Post by Francesc Alted
Hi there,
We have been recently discussing about how to put more resources in the
https://groups.google.com/forum/#!topic/pytables-dev/6fCsVzO6Ayg
Although Antonio Valentino and Anthony Scopatz have been doing a
terrific work maintaining the library, it seems like nowadays they are too
busy to continue putting time in PyTables anymore. We know that there are
many projects out there using the library, and we are in a point now where
we should decide what to do with the beast.
My first thought is to ask for current users that have benefited from
using PyTables in the past or are planning to continue using it, to
contribute economically to the sustainability of the project. For this,
perhaps a nice initiative would be to setup a site where people can do
contributions to the project (in the style of
https://www.kickstarter.com or http://www.verkami.com/), mainly for
maintenance or even for new developments. That would be a good way to
measure the interest in keeping PyTables alive, IMO. If you think
that is a good idea, we would like to hear about you. If you think there
are better ways for doing the maintenance, we are interested too.
As you know, PyTables is software that allows you not only to store and
retrieve data in HDF5, but implements a series of features that are not in
HDF5, like indexing or out-of-core expression evaluation, that set it apart
from other solutions. I personally think this is worth to be kept alive,
but it is ultimately you who has to decide if the project must go on.
Thanks,
--
Francesc Alted

--
Francesc Alted

--
You received this message because you are subscribed to the Google Groups
"pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[Gonçalo Lopes]

Hi everyone,

I would just like to echo David's sentiment. I'm also an indirect PyTables
user via the Pandas to_hdf interface, but I have to say this module is now
invaluable to our data analysis pipeline and data sharing strategies. I
work in a Neuroscience research institute where we have to collect and
analyze massive amounts of neural and behavior data. If it weren't for HDF5
storage via PyTables we would have to struggle with how to structure in
disk our complex table layouts. By using PyTables and Pandas we were able
to skip a huge technical problem just like that, which is kind of awesome
and allowed us to get down to doing our analysis right away!

I'm also a maintainer of an open-source project myself which is used
heavily at the institute for the control and acquisition of many parallel
data streams. I can totally understand the feeling you are having, believe
me! As David said, when something becomes ingrained in the daily pipeline
people tend to forget about it, but that certainly doesn't mean it's not
needed anymore. In fact, from my experience with the data stream framework,
I realized that paradoxically the opposite can be true: the less you hear
people talk about it, the smoother the software is running.

So don't give up! I'm sure PyTables has a huge established base of avid
users; most of them probably don't even know what PyTables is but would go
crazy if the rug was suddenly to be pulled underneath them.

Unfortunately I cannot contribute much concrete help at this time other
than voicing my support, but I will make sure to emphasize the need for
PyTables and the other open-source software tools we use in our work, to
try and raise awareness for their growing importance.

Keep up the amazing and excellent work! It already changed completely the
way we do science :-)

Gonçalo

Post by Francesc Alted
Hi David,
Thank you very much for showing up and telling us that PyTables is still
valuable for you.
And you are right, open source is becoming so 'embedded' in computer
sciences nowadays that people tend to think about it just as a 'facility'
that can be used for free with no more hassles to think about. Problem is,
of course, what happens when developers/maintainers do not have more free
time to put in the project.
The only solution, as I see it, is to transmit the problem to the
community and see if they are willing to do something about this. This has
been done, and for the moment only the HDF Group has shown interest in
providing some sponsorship (we still don't know how much and for how long
though, and that apparently will have to wait until next SciPy conference
due in July at least).
And glad to see that you attended to our course in St Andrews (but it was
2011: https://python.g-node.org/python-summerschool-2011/start). We are
always very happy to teach in the Summer School, but the best reward is to
see that it served to make more people using Python for scientific
applications.
Francesc

Post by Anthony Scopatz
Hi All,
I don't follow the pytables development, but I came here after reading
the latest release notes. As a user (mainly via pandas to_hdf interface) I
wanted to indicate how useful pytables is for me. At the university I work
we are finally moving towards open standards to store large amounts of
simulation data (I do wind turbine aeroelastic simulations in case you're
interested ;-), although we're not quite there yet.
Unfortunately, it is very difficult to convince the senior staff that
using open source also brings a certain responsibility. It is not easy to
"distract" funding to invest in something that is "just" tools. That
doesn't stop me and others from trying, but I am sure you've heard this
kind of story before. However, it is good to know that pytables is in need
for help, so I can makes others aware of it as well. I do not have the
right skill set to step up as a maintainer for a project such as pytables
(I am really struggling with my first PR for pandas :-), but I could help
support others in doing so. Please feel free to contact me if anything
comes up.
Let me use this opportunity to thank the pytables developers for creating
and maintaining this amazing piece of software. Francesc, I had once the
pleasure of attending your courses at the Python summer school (2009, St
Andrews) and I was most impressed by your lectures on numexpr and pytables,
so also another thanks for that!
Best regards,
David

Post by Francesc Alted
Hi Daniele,
Thanks for your thoughts. You and Jeff are the first PyTables users
that are actually responding to this call, and the information that you
provided is very useful.
Your proposal of introducing a 'DeprecationWarning' may sound a bit
strong, but is nonetheless worth of consideration. But now that at least
The HDF Group has shown interest in sponsoring the maintenance of the
project, I think it is better to wait and see how things are evolving in
the near future.
Thanks again for the time that you took for demonstrating that PyTables
is still valuable for the community. We need more feedback like this!
Francesc

Post by Andrew Collette
Hi Francesc,
I have an quite strong interest in the future of pytables.
I think HDF5 is one of the best formats for scientific applications,
and pytables makes viable to easily use it with unmatched performance.
I don't think a kickstarter-style campaign will meet any success. As
somebody said in the dev list, pytables is not a mass-market product
capable of exciting a large number of people. Moreover, it is not new, and
already has a sort of more recent "concurrent".
Francesc, I came to your announcement via the HDFgroup mailing list, to
which I subscribed long ago for a small bug and never cared enough even to
unsubscribe.
I think very few of the actual pytable users will ever reach your
message - imagine a ks campaign!
On the countrary, a direct campaign towards existing users might
increase awareness and participation in the decisions about the future of
this great package.
"tables.openFile", 1.705 files
"tables.open_file", 773 files
Unfortunately github does not tell us the number of independent
repositories, but it should not be too difficult to use some sort of github
api to extract them and then directly contact them, one by one.
But I suspect even this action would reach a minimal subset of pytables
users, which I think use pytables for in-house projects in their
company/university, not on github. This is another reason why a ks-campaign
will most probably fail: even when aware of, they will hardly convince
their employer to give significative economical support.
A (not elegant) way to reach users and make them aware of the situation
would be to introduce a deprecation warning at import time or open_file,
which can be manually switched off some way.
This will be slow (will show only on lib upgrade), but will ultimately
reach everybody.
Best regards,
Daniele
ps: My application needs live data-intensive acquisition and analysis.
I considered a number of other formats, and also the h5py wrapper, and I
still strongly believe pytables is the best solution.
The most important feature missing for my case is the single-writer
multi-reader mode, experimentally supported in h5py. I partially solved
this by serializing access in a separate process.
Below a feature comparison with weights and votes for my case.
*Feature* *Weight* *pytables* *h5py* *sqlite* *directory* *zipfile* Efficient
slicing *10* 2 2 1 0 0 Efficient appending *10* 2 0 2 2 0 Format
stability *10* 2 1 2 2 2 Single-Writer Multi-Reader *9* 0 1 1 2 0 Searching,
in-k *8* 2 0 1 0 0 Compression *6* 2 2 1 0 2 Dependencies/ecosystem
*5* 0 0 2 2 2 File-like objects *2* 2 0 1 2 0
*Result:* 7,7 4,3 7,1 6,0 3,5

Post by Francesc Alted
Hi there,
We have been recently discussing about how to put more resources in
https://groups.google.com/forum/#!topic/pytables-dev/6fCsVzO6Ayg
Although Antonio Valentino and Anthony Scopatz have been doing a
terrific work maintaining the library, it seems like nowadays they are too
busy to continue putting time in PyTables anymore. We know that there are
many projects out there using the library, and we are in a point now where
we should decide what to do with the beast.
My first thought is to ask for current users that have benefited from
using PyTables in the past or are planning to continue using it, to
contribute economically to the sustainability of the project. For this,
perhaps a nice initiative would be to setup a site where people can
do contributions to the project (in the style of
https://www.kickstarter.com or http://www.verkami.com/), mainly for
maintenance or even for new developments. That would be a good way to
measure the interest in keeping PyTables alive, IMO. If you think
that is a good idea, we would like to hear about you. If you think there
are better ways for doing the maintenance, we are interested too.
As you know, PyTables is software that allows you not only to store
and retrieve data in HDF5, but implements a series of features that are not
in HDF5, like indexing or out-of-core expression evaluation, that set it
apart from other solutions. I personally think this is worth to be kept
alive, but it is ultimately you who has to decide if the project must go on.
Thanks,
--
Francesc Alted

--
Francesc Alted

--
You received this message because you are subscribed to the Google Groups
"pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
Francesc Alted
--
You received this message because you are subscribed to the Google Groups
"pytables-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[Gaëtan de Menten]

Hi everyone,

Well, our story is probably similar to many other projects out there.
We use pytables in a few projects and it has been immensely useful
there. We are obviously very grateful to you (Francesc) and others who
have contributed to it. But the "problem" is that we are mostly happy
with it as it is. Sure there are many ways it could be improved to be
even more awesome but the thing is none of it is crucial to what we
do, so I would have a hard time convincing my employer (*) to invest
any significant portion of my time into the project and an even harder
time to invest money into it. Note that I agree that it needs at least
some maintenance work, but I doubt I could convince anybody I know to
pay for it. It is kind of a sad situation, and I sincerely hope
someone will step up to fund the project.

Gaëtan

(*) government funded, on a tight budget given declining budget of our
institution in recent years.

PS: I did not react initially because I thought saying "I use it but I
cannot help" does not really help...

Post by Francesc Alted
Hi there,
We have been recently discussing about how to put more resources in the
https://groups.google.com/forum/#!topic/pytables-dev/6fCsVzO6Ayg
Although Antonio Valentino and Anthony Scopatz have been doing a terrific
work maintaining the library, it seems like nowadays they are too busy to
continue putting time in PyTables anymore. We know that there are many
projects out there using the library, and we are in a point now where we
should decide what to do with the beast.
My first thought is to ask for current users that have benefited from using
PyTables in the past or are planning to continue using it, to contribute
economically to the sustainability of the project. For this, perhaps a nice
initiative would be to setup a site where people can do contributions to the
project (in the style of https://www.kickstarter.com or
http://www.verkami.com/), mainly for maintenance or even for new
developments. That would be a good way to measure the interest in keeping
PyTables alive, IMO. If you think that is a good idea, we would like to
hear about you. If you think there are better ways for doing the
maintenance, we are interested too.
As you know, PyTables is software that allows you not only to store and
retrieve data in HDF5, but implements a series of features that are not in
HDF5, like indexing or out-of-core expression evaluation, that set it apart
from other solutions. I personally think this is worth to be kept alive,
but it is ultimately you who has to decide if the project must go on.
Thanks,
--
Francesc Alted