ob_type in shared memory

[Aaron Brady]

Hello,

I am writing an extension using shared memory. I need a data type
that is able to reassign its 'ob_type' field depending on what process
is calling it.

Object 'A' is of type 'Ta'. When process 'P' is looking at it, it
needs to have an 'ob_type' that is 'Ta' as process 'P' sees it. When
process 'Q' is looking at it, it needs to have an 'ob_type' that is
'Ta' as process 'Q' sees it. If it referred to 'Ta' in process 'P'
when 'Q' was calling it, 'Q' would have to access memory that is in
another process.

Therefore, I need a field and an array. The field indicates which
type should be loaded, and the array contains the types. Quick
example:

PyTypeObject* array_of_types[]= { &SharedList, &SharedTuple };

Then, when a list is being accessed, it can set its own 'ob_type'
field to 'array_of_types[ 0 ]', and similarly for a tuple.

However, I'm having trouble getting 'array_of_types' in the right
module during compilation. My question is: Where do 'array_of_types'
and the forward declarations for the types go?

My fallback is what 'pickle' does: store types as strings, then load
them dynamically for 'ob_type'. That is, obtain a pointer to the type
from the string.

Thank you for reading! Any ideas? Thanks in advance.

 

[Mark Wooding]

I am writing an extension using shared memory. I need a data type
that is able to reassign its 'ob_type' field depending on what process
is calling it.

That sounds scary!

Object 'A' is of type 'Ta'. When process 'P' is looking at it, it
needs to have an 'ob_type' that is 'Ta' as process 'P' sees it. When
process 'Q' is looking at it, it needs to have an 'ob_type' that is
'Ta' as process 'Q' sees it. If it referred to 'Ta' in process 'P'
when 'Q' was calling it, 'Q' would have to access memory that is in
another process.

I see. My immediate reaction was to suggest that you just put the
PyTypeObject in shared memory -- but then I realised that the shared
memory region will probably be in different addresses in the two
processes. I'll assume that you've got enough synchronization between
the processes involved to stop everything from turning to mush.

Therefore, I need a field and an array. The field indicates which
type should be loaded, and the array contains the types. Quick
example:

PyTypeObject* array_of_types[]= { &SharedList, &SharedTuple };

Then, when a list is being accessed, it can set its own 'ob_type'
field to 'array_of_types[ 0 ]', and similarly for a tuple.

However, I'm having trouble getting 'array_of_types' in the right
module during compilation. My question is: Where do 'array_of_types'
and the forward declarations for the types go?

I'm not sure I understand the difficulty. They'll want to go in your C
module somewhere, but as long as SharedList and SharedTuple are either
in the same source file or not declared `static' you just write the
definition you've got somewhere after declaring or defining the actual
types in question.

There's a comment in Extending and Embedding (2.1) about initializing
PyTypeObjects:

: PyObject_HEAD_INIT(NULL)
:
: This line is a bit of a wart; what we'd like to write is:
:
: PyObject_HEAD_INIT(&PyType_Type)
:
: as the type of a type object is "type", but this isn't strictly
: conforming C and some compilers complain.

The comment here is wrong: &PyType_Type is a perfectly good constant
expression as far as C is concerned, but alas Microsoft's dynamic
linking system isn't clever enough to cope with it even so (because it's
in a separate link unit). But this isn't a problem in our case:
presumably SharedList and SharedTuple are in the same module, so this
should all just work.

-- [mdw]

 

[Aaron Brady]

Hi Mark, nice to have your comment.

I see.  My immediate reaction was to suggest that you just put the
PyTypeObject in shared memory -- but then I realised that the shared
memory region will probably be in different addresses in the two
processes.

Yes, exactly.

 I'll assume that you've got enough synchronization between
the processes involved to stop everything from turning to mush.

Have at thee, synchronization! Yes, it is possible theoretically.
(By the way, did you know you can detect deadlock before it strikes
with a simple breadth-first search?) The idea is still not out of the
proof-of-concept stage-- the concept not having been proved.

Therefore, I need a field and an array.  The field indicates which
type should be loaded, and the array contains the types.  Quick
example:

PyTypeObject* array_of_types[]= { &SharedList, &SharedTuple };

Then, when a list is being accessed, it can set its own 'ob_type'
field to 'array_of_types[ 0 ]', and similarly for a tuple.

However, I'm having trouble getting 'array_of_types' in the right
module during compilation.  My question is: Where do 'array_of_types'
and the forward declarations for the types go?

I'm not sure I understand the difficulty.  They'll want to go in your C
module somewhere, but as long as SharedList and SharedTuple are either
in the same source file or not declared `static' you just write the
definition you've got somewhere after declaring or defining the actual
types in question.

The problem I ran into was that different modules (C files) were
seeing different versions of the structure. '&SharedList' showed up
as different addresses in the debugger! I might be missing something,
but I think the forward declaration got interpreted as an actual
declaration in the different files that included the header.
Mysteriously, it disappeared when I tried it again now. It's still a
possibility.

The solution I looked at today was declare the array like this:

PyTypeObject* keep_types[];

It's even ok in a shared header (multiply included). Then the actual
definition just gives it NULLs and a size, which are filled in as the
needed modules are loaded. That solves the (additional) problem of
not having called the owner module's initialization code.

The catch is that if you have an object in a SharedList, you need to
have imported its module before you load it. However 'pickle' has
that problem too, so I don't feel bad.

There's a comment in Extending and Embedding (2.1) about initializing
PyTypeObjects:

:          PyObject_HEAD_INIT(NULL)
:
: This line is a bit of a wart; what we'd like to write is:
:
:          PyObject_HEAD_INIT(&PyType_Type)
:
: as the type of a type object is "type", but this isn't strictly
: conforming C and some compilers complain.

The comment here is wrong: &PyType_Type is a perfectly good constant
expression as far as C is concerned, but alas Microsoft's dynamic
linking system isn't clever enough to cope with it even so (because it's
in a separate link unit).  But this isn't a problem in our case:
presumably SharedList and SharedTuple are in the same module, so this
should all just work.

Here is another quote from the docs:

"Portability therefore requires not to make any assumptions about
symbol visibility. This means that all symbols in extension modules
should be declared static, except for the module’s initialization
function, in order to avoid name clashes with other extension modules
(as discussed in section The Module’s Method Table and Initialization
Function)."

All declared static? Dear. So I don't know what to make of it,
especially given the need to call 'PyType_Ready'.

 

[Aaron Brady]

Hi, Mark,

Do you mind if I approach you off the group about this?

Aaron

That sounds scary!

snip

 

[Aaron Brady]

Hi, Mark, snip

Ha ha, whoops.