Python module import loop issue

[Kottiyath]

This might not be pure python question. Sorry about that. I couldnt
think of any other place to post the same.
I am creating a _medium_complex_ application, and I am facing issues
with creating the proper module structure.
This is my first application and since this is a run-of-the-mill
application, I hope someone would be able to help me.

Base Module:
Contains definitions for Class A1, Class A2

Module 1.1:
Class B1 (refines A1)
Module 1.2:
Class C1 (refines A1)
Module 1.3:
Class D1 (refines A1)

Module 2.1:
Class B2 (refines A2):
Uses objects of B1, C1, D1
Module 2.2:
Class C2 (refines A2)
Module 2.3:
Class D2 (refines A2)

-->Python Entry Module : Module EN<--
Calls objects of B1, C1 and D1

Module EN and also Module 2 creates and calls the objects during run
time - and so calls cannot be hardcoded.
So, I want to use Factory methods to create everything.

Module Factory:
import 1.1,1.2,1.3, 2.1,2.2,2.3
A1Factory: {'B1Tag':1.1.B1, 'C1Tag':1.2.C1, 'D1Tag':1.3.D1'}
A2Factory: {'B2Tag':2.1.B2, 'C2Tag':2.2.C2, 'D2Tag':2.3.D2'}

But, since Module requires objects of B1, C1 etc, it has to import
Factory.
Module 2.1:
import Factory.

Now, there is a import loop. How can we avoid this loop?

The following ways I could think of
1. Automatic updation of factory inside superclass whenever a subclass
is created. But, since there is no object created, I cannot think of
a way of doing this.
2. Update A1Factory in each module which implements refinements.
_Very_important_, how do I make sure each module is hit - so that the
factory is updated? The module EN will be looking only at base module,
so the other modules is not hit. I will have to import every module in
EN - just to make sure that the A1Factory updation code is hit. This
looks in-elegent.

If somebody could help me out, I would be very thankful.

 

[Carl Banks]

This might not be  pure python question. Sorry about that. I couldnt
think of any other place to post the same.
I am creating a _medium_complex_ application, and I am facing issues
with creating the proper module structure.
This is my first application and since this is a run-of-the-mill
application, I hope someone would be able to help me.

Base Module:
Contains definitions for Class A1, Class A2

Module 1.1:
Class B1 (refines A1)
Module 1.2:
Class C1 (refines A1)
Module 1.3:
Class D1 (refines A1)

Module 2.1:
Class B2 (refines A2):
        Uses objects of B1, C1, D1
Module 2.2:
Class C2 (refines A2)
Module 2.3:
Class D2 (refines A2)

-->Python Entry Module : Module EN<--
Calls objects of B1, C1 and D1

Module EN and also Module 2 creates and calls the objects during run
time - and so calls cannot be hardcoded.
So, I want to use Factory methods to create everything.

Module Factory:
import 1.1,1.2,1.3,  2.1,2.2,2.3
A1Factory: {'B1Tag':1.1.B1, 'C1Tag':1.2.C1, 'D1Tag':1.3.D1'}
A2Factory: {'B2Tag':2.1.B2, 'C2Tag':2.2.C2, 'D2Tag':2.3.D2'}

But, since Module requires objects of B1, C1 etc, it has to import
Factory.
Module 2.1:
import Factory.

Now, there is a import loop. How can we avoid this loop?

The following ways I could think of
1. Automatic updation of factory inside superclass whenever a subclass
is created. But, since there is no object created,  I cannot think of
a way of doing this.

I'm going to suggest three ways: a straightforward, good-enough way; a
powerful, intelligent, badass way; and a sneaky way.


1. The straightforward, good-enough way

Define functions in Factory.py called register_A1_subclass and
register_A2_subclass, then call them whenever you create a new
subclass.

Factory.py
-----------------------------
A1Factory = {}
A2Factory = {}

def register_A1_subclass(tag,cls):
A1Factory[tag] = cls

def register_A2_subclass(tag,cls):
A2Factory[tag] = cls
-----------------------------

package1/module1.py:
-----------------------------
import Factory

class B1(A1):
# define class B1 here

Factory.register_A1_subclass("B1Tag",B1)
-----------------------------

So after you define B1, call Factory.register_A1_subclass to add it to
the A1Factory. Factory.py no longer has to import package1.module2,
so the circular import is broken, at the paltry price of having to add
a boilerplate function call after every class definition.


2. The powerful, intelligent, badass way

Metaclasses. I would guess you do not want to do this, and I wouldn't
recommend it if you haven't studied up on how metaclasses work, but
it's a textbook example of their usefulness. If you expect to use
factory functions like this a lot, it might be worth your while to
learn them.

Anyway, here's a simple example to illustrate. It doesn't meet your
requirements since all classes use the same factory; updating it to
your needs is left as an exercise.

Factory.py:
-----------------------------
Factory = {}

class FactoryMetaclass(type):
def __new__(metaclass,name,bases,dct):
cls = type.__new__(metaclass,name,bases,dct)
tag = dct.get("tag")
if tag is not None:
Factory[tag] = cls
return cls
------------------------------

Base.py:
------------------------------
import Factory

class A2(object):
__metaclass__ = FactoryMetaclass
# define rest of A2
------------------------------

package1/module2.py:
------------------------------
class B2(A2):
tag = "B2Tag"
#define rest of B2
------------------------------

When the class B2 statement is executed, Python notes that the
metaclass for A2 was set to FactoryMetaclass (subclasses inherit the
metaclass), so it calls FactoryMetaclass's __new__ method to create
the class object. The __new__ method checks to see if the class
defines a "tag" attribute, and if so, adds the class to the Factory
with that tag. Voila.

(As a footnote, I will mention that I've created a library, Dice3DS,
that uses metaclass programming in exactly this way.)


3. The sneaky way

New-style classes maintain a list of all their subclasses, which you
can retrieve by calling the __subclassess__ class method. You could
use this to define a factory function that searches through this list
for the appropriate subclass.

Factory.py:
-----------------------------
def _create_subclass(basecls,name):
for cls in basecls.__subclasses__():
if cls.__name__ == name:
return cls()
cls2 = _create_subclass(cls,name)
if cls2 is not None:
return cls2()
return None

def create_A1_subclass(name):
cls = _create_subclass(A1,name)
if cls is None:
raise ValueError("no subclass of A1 by that name")
return cls
-----------------------------

So here you search through A1's subclasses for a class matching the
class's name. Note that we do it recursively, in case B1 (for
instance) has its own subclasses, and I presume we do want those.

You can change it to do a search by a tag class attribute if you wish;
left as an exercise.

2. Update A1Factory in each module which implements refinements.
_Very_important_, how do I make sure each module is hit - so that the
factory is updated? The module EN will be looking only at base module,
so the other modules is not hit. I will have to import every module in
EN - just to make sure that the A1Factory updation code is hit. This
looks in-elegent.

Not worth it. The straightforward, good-enough way above is good
enough.


Carl Banks

 

[Gabriel Genellina]

I'm going to suggest three ways: a straightforward, good-enough way; a
powerful, intelligent, badass way; and a sneaky way.

In Python 2.6 (and 3.0) there is a fourth way: class decorators.

1. The straightforward, good-enough way

Define functions in Factory.py called register_A1_subclass and
register_A2_subclass, then call them whenever you create a new
subclass.

Class decorators are a clean variant of this approach (in my opinion).

package1/module1.py:
-----------------------------
import Factory

class B1(A1):
# define class B1 here

Factory.register_A1_subclass("B1Tag",B1)
-----------------------------

That would become:

@Factory.register_A1_subclass("B1Tag")
class B1(A1):
...

(for an adequate variant of register_A1_subclass). The advantage is that
the "register" stuff appears prominently near the name of the class, and
there is no need to repeat the name.
Also, "B1Tag" can be left out, if it is stored as a class attribute of B1
(in some cases using __name__ is enough)

2. The powerful, intelligent, badass way

Metaclasses. I would guess you do not want to do this, and I wouldn't
recommend it if you haven't studied up on how metaclasses work, but
it's a textbook example of their usefulness. If you expect to use
factory functions like this a lot, it might be worth your while to
learn them.

A problem with metaclasses is when you have intermediate subclasses that
are not meant to be registered, but the metaclass applies equally to all
of them.

 

[Carl Banks]

In Python 2.6 (and 3.0) there is a fourth way: class decorators.


Class decorators are a clean variant of this approach (in my opinion).


That would become:

@Factory.register_A1_subclass("B1Tag")
class B1(A1):
...

(for an adequate variant of register_A1_subclass). The advantage is that
the "register" stuff appears prominently near the name of the class, and
there is no need to repeat the name.
Also, "B1Tag" can be left out, if it is stored as a class attribute of B1
(in some cases using __name__ is enough)

Thanks for the additional suggestion.

A problem with metaclasses is when you have intermediate subclasses that
are not meant to be registered, but the metaclass applies equally to all
of them.

Not the way I wrote it. If you'll note, the metaclass only added the
class to the factory map if a tag attribute was defined in the class
dict.


Carl Banks

 

[Kottiyath]

In Python 2.6 (and 3.0) there is a fourth way: class decorators.



Class decorators are a clean variant of this approach (in my opinion).




That would become:

@Factory.register_A1_subclass("B1Tag")
class B1(A1):
   ...

(for an adequate variant of register_A1_subclass). The advantage is that  
the "register" stuff appears prominently near the name of the class, and  
there is no need to repeat the name.
Also, "B1Tag" can be left out, if it is stored as a class attribute of B1  
(in some cases using __name__ is enough)



A problem with metaclasses is when you have intermediate subclasses that  
are not meant to be registered, but the metaclass applies equally to all  
of them.

Hi Gabriel, Carl,
Thank you very much for your help.
I never knew about metaclassess and class decorators. Thank you
again.
I am actually inclined towards the straightforward way (1). But
still one of the issues that I have mentioned in the first mail
remains. How do I actually hit the code because my entry point is the
EN module.
Importing every module in EN module so that it hits the code atleast
once is fraught with danger because later, someone might delete it to
clean it up and will start facing issues.
Could you give me some pointers in such a case?
Regards
K

 

[Gabriel Genellina]

Gabriel Genellina wrote:

Not the way I wrote it. If you'll note, the metaclass only added the
class to the factory map if a tag attribute was defined in the class
dict.

Ah, I didn't notice that, sorry!