Project source code layout?

[Allen Fowler]

Hello,

I'm new to Python, and am looking for some suggestions as to the source code layout for a new project.

The project will be a tg/pylons daemon, a static website, and a collection of other scripts that will interact with the app and DB.

Here is what I am thinking so far:

root_folder/
- app/ -- Code for pylons/TG web app
- web/ -- Public static web files (and wsgi / fastCGI connector files)
- db/ -- SQlite DB
- scripts/ -- Various custom programs that will also interact with the DB / app. (Some cron, some interactive.)
- config/ -- 3-rd party API tokens, DB parameters, etc.

However, I am still wondering about a few items:

1) Where to create the virtualpython installation that will be used by both the app and the scripts.

2) Where to put our in-house created python modules so that they can be imported by both the TG app and our own scripts.

3) How to cleanly provide the various config settings to both the web app and scripts.

Any suggestions? ideas?

fwiw, I am planing on keeping the whole thing in a Mercurial repository.

Thank you,
Allen

 

[Lawrence D'Oliveiro]

I'm new to Python, and am looking for some suggestions as to the source
code layout for a new project.

Is this the development layout or the deployment layout? The two need not
bear any resemblance.

 

[Allen Fowler]

Is this the development layout or the deployment layout? The two need not
bear any resemblance.

Looking for suggestions on both.

I was hoping to keep the dev layout as close to deployment possible.

Thank you,

 

[Lawrence D'Oliveiro]

I was hoping to keep the dev layout as close to deployment possible.

Completely different purposes. For example, the actual production database
and config files form no part of your development project, do they? And
conversely, utility scripts that might be used, for example, to set up a
database, should not be part of the production installation.

Here's one layout I used in a production system for an online shop:

/home/shop/bin -- binaries (CGI and daemon)
/home/shop/lib -- common libraries for binaries
/home/shop/config -- configuration files, incl format templates
/home/shop/state -- socket for interprocess communication, log files

 

[Allen Fowler]

Completely different purposes. For example, the actual production database
and config files form no part of your development project, do they? And
conversely, utility scripts that might be used, for example, to set up a
database, should not be part of the production installation.

The data will certainly be different. Good point. (Though for now even the production DB will be sqlite.)

Here's one layout I used in a production system for an online shop:

/home/shop/bin -- binaries (CGI and daemon)
/home/shop/lib -- common libraries for binaries
/home/shop/config -- configuration files, incl format templates
/home/shop/state -- socket for interprocess communication, log files

Thank you.

Couple of questions:

1) Do you use virtualpython?

2) How do you load the modules in your lib directory?

3) How do you reference your configuration directives from within your modules and CGI/daemon scripts?

Thank you,

Allen

 

[Lawrence D'Oliveiro]

1) Do you use virtualpython?

No idea what that is.

2) How do you load the modules in your lib directory?

At the beginning of my scripts, I have a sequence like

test_mode = False # True for testing, False for production

if test_mode :
home_dir = "/home/shop-test"
else :
home_dir = "/home/shop"
#end if

sys.path.append(home_dir + "/lib")

import common
[etc]

I have an installation script that looks for that "test_mode = True/False"
assignment and edits it accordingly. That flag is used to select the top-
level directory (as above) as well as the database name, etc. This allows me
to run two complete parallel sets of code and data, so I can mess around
with the testing version without impacting the production system.

3) How do you reference your configuration directives from within your
modules and CGI/daemon scripts?

For my last project using the above system, I used XML as the config file
format.

 

[Dave Angel]

In message <[email protected]>, Allen
Fowler wrote:

1) Do you use virtualpython?

No idea what that is.

2) How do you load the modules in your lib directory?

At the beginning of my scripts, I have a sequence like

test_mode = False # True for testing, False for production

if test_mode :
home_dir = "/home/shop-test"
else :
home_dir = "/home/shop"
#end if

sys.path.append(home_dir + "/lib")

import common
[etc]

I have an installation script that looks for that "test_mode = True/False"
assignment and edits it accordingly. That flag is used to select the top-
level directory (as above) as well as the database name, etc. This allows me
to run two complete parallel sets of code and data, so I can mess around
with the testing version without impacting the production system.

3) How do you reference your configuration directives from within your
modules and CGI/daemon scripts?

For my last project using the above system, I used XML as the config file
format.

Rather than editing the source files at install time, consider just
using an environment variable in your testing environment, which would
be missing in production environment. Each command shell has its own
set of environment variables, so this would make testing pretty easy,
without the risk of things getting out of synch. Alternatively, if the
script is located in a fixed place, relative to the home directory, just
do some manipulation of the __file__ string.

 

[Lawrence D'Oliveiro]

Rather than editing the source files at install time, consider just
using an environment variable in your testing environment, which would
be missing in production environment.

I'd still need to define that environment variable in a wrapper script,
which means editing that script at install time ... back to square one ...

 

[Dave Angel]

In message <[email protected]>, Dave Angel
wrote:



I'd still need to define that environment variable in a wrapper script,
which means editing that script at install time ... back to square one ...

No, the whole point is it's an environment variable which is *missing"
in production environment. Make sure you make it an obscure name, like
set MyProductName_TestingMode=1

So the way you know you're in a production environment is that you do
not have such an environment variable.

 

[Lawrence D'Oliveiro]

No, the whole point is it's an environment variable which is *missing"
in production environment. Make sure you make it an obscure name, like
set MyProductName_TestingMode=1

So the way you know you're in a production environment is that you do
not have such an environment variable.

Sounds like a very roundabout solution to the wrong problem.