breaking up is hard to do

[bbands]

I've a 2,000 line and growing Python script that I'd like to break up
into a modules--one class alone is currently over 500 lines. There is a
large config.ini file involved (via ConfigParser), a fair number of
passed and global variables as well as interaction with external
programs such as MySQL (via MySQLdb), R (via Rpy) and gnuplot (via
Gnuplot). Every time I have tried to break it up I end up with thorny
name-space problems and have had to stitch it back together gain. Any
pointers to materials on how best to modularize a script such as this
would be appreciated. I'd like to end up with most of the code in
modules and with just the main logic and housekeeping stuff in the main
module.

John

PS The script works fine as is and I am a very happy and productive
camper with Python. I was a BASIC user for many years who switched to
Python a year and a half ago rather than moving to .NET.

 

[Bill Mill]

I've a 2,000 line and growing Python script that I'd like to break up
into a modules--one class alone is currently over 500 lines. There is a
large config.ini file involved (via ConfigParser), a fair number of
passed and global variables as well as interaction with external
programs such as MySQL (via MySQLdb), R (via Rpy) and gnuplot (via
Gnuplot). Every time I have tried to break it up I end up with thorny
name-space problems and have had to stitch it back together gain.

What sort of namespace problems? I think you need to tell us what your
specific problems were, so that we can help you more.

Peace
Bill Mill
bill.mill at gmail.com

 

[bbands]

For example I have a class named Indicators. If I cut it out and put it
in a file call Ind.py then "from Ind import Indicators" the class can
no longer see my globals. This is true even when the import occurs
after the config file has been read and parsed.

John

 

[Larry Bates]

Kind of vague, but I'll give it a shot:

1) Don't read config.ini values in main and pass to class if they
aren't needed in the main program. Instead create instance of
ConfigParser and pass it as an argument to the class and extract
the individual section/option information in the class.

INI=ConfigParser.ConfigParser()
INI.read(inifilename)

b=newclass(INI)

then in newclass you just get what you want from the .INI file.
You aren't actually reading from the file, the read put it in
a dictionary.

2) Pump variables into dictionary and pass it as **kwargs argument
to functions/classes.

Example:

class foo:
def __init__(self, **kwargs):
self.trace=kwargs.get('trace',0)
self.debug=kwargs.get('debug',0)
self.outputfile=kwargs.get('outputfile', 'out.txt')
.
.
.


global_dict={'trace': 0, 'debuglevel': 2, 'outputfile': 'outfile.txt'}

F=foo(**global_dict)

Then if your nesting goes deeper, just pass **kwargs to functions or
classes that are lower down.

Hope this helps.

Larry Bates

 

[Kent Johnson]

For example I have a class named Indicators. If I cut it out and put it
in a file call Ind.py then "from Ind import Indicators" the class can
no longer see my globals. This is true even when the import occurs
after the config file has been read and parsed.

globals aren't all that global, they have module scope. One solution is to have a module just for
your config data. Other modules that need to access it import the module and use its data. The
config module can read it's values from a file or whatever when it is loaded.

Very simple example:

# Config.py

maxSize = 1000
# read values from config file and put them in module scope
# etc...


# Indicators.py

import Config

if xx > Config.maxSize:
...

If you really don't like to prefix all the references to Config with the module name you can use
from Config import *
but that is not recommended...

Another solution is to put your config data into some kind of object or structure that you pass
around to anyone who needs it.

Kent

 

[Kent Johnson]

For example I have a class named Indicators. If I cut it out and put it
in a file call Ind.py then "from Ind import Indicators" the class can
no longer see my globals. This is true even when the import occurs
after the config file has been read and parsed.

globals aren't all that global, they have module scope. One solution is to have a module just for
your config data. Other modules that need to access it import the module and use its data. The
config module can read it's values from a file or whatever when it is loaded.

Very simple example:

# Config.py

maxSize = 1000
# read values from config file and put them in module scope
# etc...


# Indicators.py

import Config

if xx > Config.maxSize:
...

If you really don't like to prefix all the references to Config with the module name you can use
from Config import *
but that is not recommended...

Another solution is to put your config data into some kind of object or structure that you pass
around to anyone who needs it.

Kent

 

[David M. Cooke]

For example I have a class named Indicators. If I cut it out and put it
in a file call Ind.py then "from Ind import Indicators" the class can
no longer see my globals. This is true even when the import occurs
after the config file has been read and parsed.

Don't use globals? Or put all the globals into a separate module,
which you import into Ind and into whatever uses Ind.

Putting the globals into a separate namespace (module, class, class
instance, whatever) also makes it easier to know what is a global

 

[Cameron Laird]

.
[puzzlement about globals,
on which several others
have already aptly counseled]
.
.

camper with Python. I was a BASIC user for many years who switched to
Python a year and a half ago rather than moving to .NET.

Please be aware that, on a technical plane, you can move to .NET *and*
stay with Python--or at least Jim Hugunin, a very accomplished guy, is
on track to make that combination possible. For details, see <URL:
http://www.technorati.com/cosmos/search.html?rank=&url=ironpython >.

 

[bbands]

Thanks for your reply.

That seems like an interesting and practical approach. However, I have
one worry. In addition to the config file I am parsing command-line
overrides to the config values via optparse. Many modules, classes and
functions depend on these values, which means a lot of code duplication
and changes that must be made in many places when a change is needed.
It would seem simpler and more elegant to have all these values held by
one global resource that is referenced whenever needed. Am I missing
something?

jab
--

 

[bbands]

Cameron,

Thanks for the heads up on that. (I have been
following it, but from a long distance, as I very
happy with my garden-variety Python.)

Separately, let me offer you my thanks for your
contributions to the community; they are great and
they are much appreciated. The community around Python
was a significant reason in my switch decision, a
decision I have had no reason to regret.

Thanks again,

jab