Python evolution: Unease

[Robert Kern]

I've never needed numeric stuff either. I just need to do things like:

.>>> table.sort(column_name) # that obviously would sort rows of table
by the values of column column_name

[snip for brevity]

Now suppose a programmer could write a custom complement function
that detects all the irregularly distributed "anomalous" data points
(be it whatever, missing surnames from personnel records or values
from a physical experiments that are below some threshold) in this
table and returns, say, a list of tuples that are coordinates of those
data points. Getting it from a specific table would be a matter of one
instruction!

Yes, I know, it can be written by hand. But by this line of logic why
bother learning VHLL and not just stay with C?

I'm not sure what you mean by "written by hand." Someone is going to
have to write the functions in the first place. Sure, they can be
written once, well, and placed in the standard library so they don't
have to be *re*written by anyone else again.

I still think numarray is a good start for this. It handles more than
just numbers. And RecArray (an array that has different types in each
column, as you seem to require) can be subclassed to add these methods
to it.

True, and I may scratch enough time together to learn all the
necessary stuff (I'm not even half done in learning Python)
to write it myself.

That is not the point, however: the biggest boost and one of the
main points of getting into Python, at least for me, but I'm sure
this is also motivation for quite a lot of other people, is precisely
the ease of exploiting capabilities of data structures like
dictionaries and lists, which when coupled with this data structure's
object-style .method are simply very convenient and fast. This is
where IMHO Python excels among the VHLL languages.

I'm about to post reworked version of my program that doesn't
use a _single_ traditional loop to do all the data transformations
I need (I just still need to solve some problems there / polish
it).

This is not just about that damn CSV file that I already have
the way I wanted it and sent it to customer, this is about _terse
and clear_ manipulations of rich data structures in Python. Why not
extend them with flexible tables / matrices / arrays that would work
in as "Pythonic" ways as dictionaries and lists already do?

Sure. We're working on it! Come check out numarray; I think you'll like
it. And if, along the way, you write a CSV-to-RecArray converter, we'd
*love* to include it in the distribution. I think that a more complete
integration with the other core Python facilities like the csv module
will help numarray become more suited for inclusion into the standard
library.

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[Steve Holden]

The first three PSF grants were all in some way not directly related to
changing the core language. One was for a library, one for improving
Jython, and one for improving docs. Giving the PSF more money increases
the chances for additional work.

I would, however, like to make the point that the Foundation is much
more likely to fund work that actually gets proposed for a grant, since
the officers don't generally have the time to inspire teams and promote
specific topics. So if a bunch of like-minded people wanted to get
together to propose a specific set of changes, it would then at least be
possible to consider such changes for funding.

There's also a lot of good work that goes on unfunded, of course, and I
wouldn't like to see the grants process subvert such public-spirited
efforts in any way. So there's nothing to stop a bunch of like-minded
individuals proposing changes anyway, through the existing process.

Lastly, the Foundation has only just started to make grants, and so we
need a little time to see how the existing projects go before we can
evaluate the success or failure of the process (at least on the existing
projects, whose outcomes may or may not be related to the grant process).

regards
Steve

 

[Bulba!]

I'm not sure what you mean by "written by hand."

I mean the same way as you do mylist.sort() in Python
instead of writing the list sorting function yourself in
the application / script.

Someone is going to
have to write the functions in the first place. Sure, they can be
written once, well, and placed in the standard library so they don't
have to be *re*written by anyone else again.

Well of course - the only point I mean here is getting
high-level stuff is one of the main motives of getting
into VHLL (aside from issues like automatic memory
management, of course).

I still think numarray is a good start for this. It handles more than
just numbers. And RecArray (an array that has different types in each
column, as you seem to require) can be subclassed to add these methods
to it.

I have downloaded it, playing with it and like it. I esp. like things
like:
[2 4 6]

or
(4,3)

Very Pythonic.

However, not all things are generic enough like I meant. That
is not to criticize, but merely to point out that the library is for
obvious reasons slanted towards numerical work, so e.g. while the
following works:

..>>> from numarray import transpose
..>>> transpose([[1,2],[3,4]])
array([[1, 3],
[2, 4]])

....this doesn't:

transpose([('phone1', 12345), ('phone2', 67890)])

Traceback (most recent call last):
File "<interactive input>", line 1, in ?
[....]
TypeError: Expecting a python numeric type, got something else.

Why would someone want to do such a thing: suppose he
wants 'phone1' and 'phone2' and number records sequenced
horizontally instead vertically, while when he read that from
a file, such was the file structure. It's a boneheaded example,
but you get the idea.

It's obvious why that exception happens: in recarray I have to have
the same type of the data type in every column, so transpose(recarray)
cannot be done.

This is not the situation with other builtin Python data types, where
whatever with can be mixed, i.e. placed in any "cell" of data type
and all the functions will still work as long as the operand type is
proper.

This is obviously the consequence of numarray being based on
the matrices designed for working on numerical data, so it's
probably very fast and I'm sure people who use it find it useful.

But personally I would happily sacrifice much or even most of that
speed for sake of flexibility. Because, sure, I can keep the
matrix header with strings separately in a list, do the transformation
just on numbers, adjust the list as well - it's not like I'm just lazy
to make an additional programming effort, but that eliminates
one of the greatest advantages of Python: terseness without
foregoing clarity.

Yes, now it's probably much harder to do once the simpler things
have been done. All I'm saying is that it would be so frigging nice
to have it with even richer data structures.

Sure. We're working on it!

Excellent! Great, I'm really grateful to you all for doing that.
Honestly.

Come check out numarray; I think you'll like
it. And if, along the way, you write a CSV-to-RecArray converter, we'd
*love* to include it in the distribution.

First I have to learn Python (and programming in general) well enough
to produce smth that isn't as bug-infested as a software of certain
well-known vendor and that I would not be ashamed of to show to the
world. I'm afraid it will take me some time.

I think that a more complete
integration with the other core Python facilities like the csv module
will help numarray become more suited for inclusion into the standard
library.

I'm sure people would find it useful. I play with numarray to get the
idea how I could apply it in my work.

 

[Jarek Zgoda]

But I see little to no efforts from the core python team to address my
needs as listed above. They seem mainly to focus on the core attributes
and syntax of the language. Very little or no efforts are taken to
improve the infrastructure around the language.

And then I read the following sentence by Van Rossum:

"In order to make type inferencing a little more useful, I'd like to
restrict certain forms of extreme dynamic behavior in Python"

In the end, it's mindset which counts. And I think that mindset is going
to be determine the way foreward for Python: more features, increased
complexity, less dynamism. Lots of syntax crud, without addressing the
need to improve the infrastructure around the language.

I saw this once somewhere:

http://groups.google.com/[email protected]

 

[Robert Kern]

I still think numarray is a good start for this. It handles more than
just numbers. And RecArray (an array that has different types in each
column, as you seem to require) can be subclassed to add these methods
to it.

I have downloaded it, playing with it and like it. I esp. like things
like:


[2 4 6]

or


(4,3)

Very Pythonic.

However, not all things are generic enough like I meant. That
is not to criticize, but merely to point out that the library is for
obvious reasons slanted towards numerical work, so e.g. while the
following works:

.>>> from numarray import transpose
.>>> transpose([[1,2],[3,4]])
array([[1, 3],
[2, 4]])

...this doesn't:

transpose([('phone1', 12345), ('phone2', 67890)])

Traceback (most recent call last):
File "<interactive input>", line 1, in ?
[....]
TypeError: Expecting a python numeric type, got something else.

Why would someone want to do such a thing: suppose he
wants 'phone1' and 'phone2' and number records sequenced
horizontally instead vertically, while when he read that from
a file, such was the file structure. It's a boneheaded example,
but you get the idea.

http://stsdas.stsci.edu/numarray/numarray-1.1.html/module-numarray.objects.html

In [1]: from numarray import objects, transpose

In [2]: a = objects.array([['phone1', 12345], ['phone2', 67890]])

In [3]: a
Out[3]:
ObjectArray([['phone1', 12345],
['phone2', 67890]])

In [4]: transpose(a)
Out[4]:
ObjectArray([['phone1', 'phone2'],
[12345, 67890]])

Note the use of lists. Using tuples makes the constructor think you want
a vector of tuple objects.

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter