Awkwardness of C API for making tuples

[Dave Opstad]

One of the functions in a C extension I'm writing needs to return a
tuple of integers, where the length of the tuple is only known at
runtime. I'm currently doing a loop calling PyInt_FromLong to make the
integers, then PyTuple_New, and finally a loop calling PyTuple_SET_ITEM
to set the tuple's items. Whew.

Does anyone know of a simpler way? I can't use Py_BuildValue because I
don't know at compile-time how many values there are going to be. And
there doesn't seem to be a PyTuple_FromArray() function.

If I'm overlooking something obvious, please clue me in!

Thanks,
Dave Opstad

 

[Diez B. Roggisch]

One of the functions in a C extension I'm writing needs to return a

tuple of integers, where the length of the tuple is only known at
runtime. I'm currently doing a loop calling PyInt_FromLong to make the
integers, then PyTuple_New, and finally a loop calling PyTuple_SET_ITEM
to set the tuple's items. Whew.

Does anyone know of a simpler way? I can't use Py_BuildValue because I
don't know at compile-time how many values there are going to be. And
there doesn't seem to be a PyTuple_FromArray() function.

If I'm overlooking something obvious, please clue me in!

If you already know _how_ to create a function like PyTuple_FromArray() -
why don't you define it yourself and use it?

 

[John Machin]

One of the functions in a C extension I'm writing needs to return a
tuple of integers, where the length of the tuple is only known at
runtime. I'm currently doing a loop calling PyInt_FromLong to make the
integers,

What is the purpose of this first loop?

In what variable-length storage are you storing these (Python) integers
during this first loop? Something you created with (a) PyMem_Malloc (b)
malloc (c) alloca (d) your_own_malloc?

then PyTuple_New, and finally a loop calling PyTuple_SET_ITEM
to set the tuple's items. Whew.

Whew indeed.

Does anyone know of a simpler way? I can't use Py_BuildValue because I
don't know at compile-time how many values there are going to be. And

there doesn't seem to be a PyTuple_FromArray() function.

If I'm overlooking something obvious, please clue me in!

1. Determine the length of the required tuple; this may need a loop,
but only to _count_ the number of C longs that you have.
2. Use PyTuple_New.
3. Loop to fill the tuple, using PyInt_FromLong and PyTuple_SetItem.

Much later, after thoroughly testing your code, gingerly change
PyTuple_SetItem to PyTuple_SET_ITEM. Benchmark the difference. Is it
anywhere near what you saved by cutting out the store_in_temp_array
thing in the first loop?

 

[Dave Opstad]

What is the purpose of this first loop?

Error handling. If I can't successfully create all the PyInts then I can
dispose the ones I've made and not bother making the tuple at all.

In what variable-length storage are you storing these (Python) integers
during this first loop? Something you created with (a) PyMem_Malloc (b)
malloc (c) alloca (d) your_own_malloc?

(b) malloc. The sequence here is: 1) malloc; 2) check for malloc
success; 3) loop to create PyInts (if failure, Py_DECREF those made so
far and free the malloc'ed buffer); 4) create new tuple (error checks
again); and 5) PyTuple_SET_ITEM (no error checks needed)

1. Determine the length of the required tuple; this may need a loop,
but only to _count_ the number of C longs that you have.
2. Use PyTuple_New.
3. Loop to fill the tuple, using PyInt_FromLong and PyTuple_SetItem.

This would certainly be simpler, although I'm not sure I'm as clear as
to what happens if, say, in the middle of this loop a PyInt_FromLong
fails. I know that PyTuple_SetItem steals the reference; does that mean
I could just Py_DECREF the tuple and all the pieces will be
automagically freed? If so, I'll take your recommendation and rework the
logic this way.

Thanks!
Dave

 

[John Machin]

Error handling. If I can't successfully create all the PyInts then I can
dispose the ones I've made and not bother making the tuple at all.

(b) malloc. The sequence here is: 1) malloc; 2) check for malloc
success; 3) loop to create PyInts (if failure, Py_DECREF those made so
far and free the malloc'ed buffer); 4) create new tuple (error checks

again); and 5) PyTuple_SET_ITEM (no error checks needed)

Don't. If you _must_ allocate your own storage, use PyMem_Malloc.

PyTuple_SetItem.

This would certainly be simpler, although I'm not sure I'm as clear as
to what happens if, say, in the middle of this loop a PyInt_FromLong
fails. I know that PyTuple_SetItem steals the reference; does that mean
I could just Py_DECREF the tuple and all the pieces will be
automagically freed? If so, I'll take your recommendation and rework the
logic this way.

This is what I believe happens. However even if you did need to do more
cleaning up, you shouldn't penalise the normal case i.e. when
PyInt_FromLong works. The only failure cause AFAIK is running out of
memory.
This should be rare unless it's triggered by your calling malloc

 

[Fredrik Lundh]

This would certainly be simpler, although I'm not sure I'm as clear as
to what happens if, say, in the middle of this loop a PyInt_FromLong
fails. I know that PyTuple_SetItem steals the reference; does that mean
I could just Py_DECREF the tuple and all the pieces will be
automagically freed?

yes. tuples release their members when you release the tuple (this is pretty
obvious, if you think about it). they also handle uninitialized items property
(this is also pretty obvious).

here's the code, btw (from tupledealloc in Objects/tupleobject.c):

while (--i >= 0)
Py_XDECREF(op->ob_item);

</F>

 

[Dave Cole]

What is the purpose of this first loop?

In what variable-length storage are you storing these (Python) integers
during this first loop? Something you created with (a) PyMem_Malloc (b)
malloc (c) alloca (d) your_own_malloc?




Whew indeed.




1. Determine the length of the required tuple; this may need a loop,
but only to _count_ the number of C longs that you have.
2. Use PyTuple_New.
3. Loop to fill the tuple, using PyInt_FromLong and PyTuple_SetItem.

Much later, after thoroughly testing your code, gingerly change
PyTuple_SetItem to PyTuple_SET_ITEM. Benchmark the difference. Is it
anywhere near what you saved by cutting out the store_in_temp_array
thing in the first loop?

Shouldn't something like this (uncompiled) do what you want?

- Dave

PyObject *PyTuple_FromArray(int *values, int num_values)
{
PyObject *tuple;
int i;

tuple = PyTuple_New(num_values);
if (tuple == NULL)
return NULL;

for (i = 0; i < num_values; i++) {
PyObject *obj;

obj = PyInt_FromLong(value);
if (obj == NULL
|| PyTuple_SetItem(tuple, i, obj) != 0) {
Py_DECREF(tuple);
return NULL;
}
}

return tuple;
}

 

[Fredrik Lundh]

for (i = 0; i < num_values; i++) {
PyObject *obj;

obj = PyInt_FromLong(value);
if (obj == NULL
|| PyTuple_SetItem(tuple, i, obj) != 0) {
Py_DECREF(tuple);
return NULL;
}
}

in theory, if PyInt_FromLong succeeds, and PyTuple_SetItem fails, you'll leak
an object.

</F>

 

[Steve Holden]

Dave Cole wrote:

for (i = 0; i < num_values; i++) {
PyObject *obj;

obj = PyInt_FromLong(value);
if (obj == NULL
|| PyTuple_SetItem(tuple, i, obj) != 0) {
Py_DECREF(tuple);
return NULL;
}
}

in theory, if PyInt_FromLong succeeds, and PyTuple_SetItem fails, you'll leak
an object.

</F>

And in practice this will only happen during a period when you are
relying critically on it *not* to ...

regards
Steve

 

[Jeremy Bowers]

And in practice this will only happen during a period when you are
relying critically on it *not* to ...

One of the steps to attaining the Tao of Programming is to entirely stop
caring what happens when your program runs, thus completely eliminating
this class of bugs. A programmer who has the Tao never critically relies
on a program, and thus the program, knowing that it can not spite the
programmer in this way, does not try.



*.5-wink* - I meant it as a joke but as I thought about it it started to
make sense, especially the first clause of the last sentence, which I
suppose means the Tao may actually be speaking through me without my
knowledge or intent...

 

[Fredrik Lundh]

And in practice this will only happen during a period when you are relying critically on it *not*
to ...

yeah, but if PyTuple_SetItem fails in this case, you better move that
computer out of the radiation chamber.

</F>