custom allocation and custom type

[castironpi]

Hi all,

I am trying to create a custom tuple type. I want it to live in a
custom memory region, which will be a memory-mapped file. Its
contents cannot be PyObject*. They have to be offsets into mapped
memory. GetItem( i ) would return:

(PyObject *)( t->ob_item[ i ]+ mmap_base_addr );

In the constructor, each item x would be stored as:

BYTE* x_offset= x- mmap_base_addr;
t->ob_item[ 0 ]= x_offset;

For starters, only ints, strings, and other tuples will be members.
I'll work out the list type and user-defined classes at a later time.
Each will be required to be already existing in the mmap; that is, be
of custom types too.

I had partial success with strings and ints just memcpy-ing the entire
PyObject structure. They are easier because they are immutable, and
don't contain references. That's why tuple is my next step: it's
immutable, but can contain references.

I want to back up a step and ask some questions about ints first.

What worries are there in memcpy-ing a PyIntObject? Would it be
better to compile my own MmapIntObject? I do Py_INCREF on the object
I return, to ensure it lives on, but I am getting an occasional crash
on exit. What did I miss? Do I need to readjust the ob_type pointer
every time I load the module (and lock for exclusive access)? Does
Python try to free the memory when it exits?

For the tuples, is there any way I can reuse the tuple code? The
constructor uses a custom pool for speed which I'd probably eliminate,
for the time. Slices would have to be MmapTupleObject instances too.
Should I also create MmapTupleIterObject?

PyObject_GC_New and PyObject_GC_NewVar are confusing. I can't follow
them to the actual call to 'malloc', since they themselves lead around
Python's custom memory pool. What steps do I need to be sure to take
in my own?

I'm having trouble building PerstTupleObject.c (persistent tuple
object) to try it out. I'm using MinGW on WinXP. I added #include
"persttupleobject.h" to the top of "persttupleobject.c" already, but
it's a duplicate otherwise, with the name replaced.

---
C>c:/programs/mingw/bin/g++ persttupleobject.c -c -Ic:/programs/
python25/include

persttupleobject.c:650: warning: 'PyPerstTuple_Type' defined locally
after being
referenced with dllimport linkage

C>c:/programs/mingw/bin/g++ -shared persttupleobject.o -o
persttupleobject.pyd -L
c:/programs/python25/libs -lpython25

persttupleobject.o(.text+0xce)ersttupleobject.c: undefined reference
to `_imp_
_PyPerstTuple_Type'
persttupleobject.o(.text+0x15c)ersttupleobject.c: undefined
reference to `_PyG
C_generation0'
[snip]

---
I get some of the same errors when compiling tupleobject.c from the
Python-2.5.2 distribution:

C>c:/programs/mingw/bin/g++ tupleobject.c -c -Ic:/programs/python25/
include

tupleobject.c:649: warning: 'PyTuple_Type' defined locally after being
reference
d with dllimport linkage

C>c:/programs/mingw/bin/g++ -shared tupleobject.o -o tupleobject.pyd -
Lc:/progra
ms/python25/libs -lpython25

c:/programs/python25/libs/libpython25.a(dcbbs00622.o)(.text+0x0):
multiple defin
ition of `PyTuple_Type'
tupleobject.o(.data+0x140):tupleobject.c: first defined here
tupleobject.o(.text+0x15c):tupleobject.c: undefined reference to
`_PyGC_generati
on0'
[snip]