Python Makefiles... are they possible?

[Malcolm White]

I have written a piece of code that will be part of a larger repository of related programs. Within this repository, it is standard to issue a 'make' command to compile any desired program. Is it possible to create a Makefileto compile a simple Python program? Based on what I have come across so far, this is not (at least not typically) the way things are done with Python..

Thanks for any pointers in the right direction!

 

[Joel Goldstick]

I have written a piece of code that will be part of a larger repository of
related programs. Within this repository, it is standard to issue a 'make'
command to compile any desired program. Is it possible to create a Makefile
to compile a simple Python program? Based on what I have come across so
far, this is not (at least not typically) the way things are done with
Python.

Thanks for any pointers in the right direction!

Python is an interpreted language. The source code is 'compiled' into
bytecode when invoked. Make is a process that converts source code to
compiled code, so it doesn't apply to python

 

[Oscar Benjamin]

I have written a piece of code that will be part of a larger repository of related programs. Within this repository, it is standard to issue a 'make' command to compile any desired program. Is it possible to create a Makefile to compile a simple Python program? Based on what I have come across so far, this is not (at least not typically) the way things are done with Python.

You can use a Makefile for anything you want in a Python project.
However Python code is not (typically) compiled so it is not common
practise to compile it with or without a Makefile. When part of a
Python project is compiled because, for example it bundles some C code
to be used within Python, the compilation needs to performed in way
that will be compatible with Python so the process is normally
controlled by Python, through a setup.py file. In this case
compilation is done with something like 'python setup.py build' (Of
course there's nothing to stop you from adding that command to a
Makefile and invoking it with 'make').

I often use Makefiles in Python projects for other purposes, though,
such as running tests with 'make test' or building documentation with
'make doc'.


Oscar

 

[Roy Smith]

You can use a Makefile for anything you want in a Python project.
However Python code is not (typically) compiled so it is not common
practise to compile it with or without a Makefile. When part of a
Python project is compiled because, for example it bundles some C code
to be used within Python, the compilation needs to performed in way
that will be compatible with Python so the process is normally
controlled by Python, through a setup.py file. In this case
compilation is done with something like 'python setup.py build' (Of
course there's nothing to stop you from adding that command to a
Makefile and invoking it with 'make').

I often use Makefiles in Python projects for other purposes, though,
such as running tests with 'make test' or building documentation with
'make doc'.

One thing we do in our Makefiles is "find . -name '*.pyc' | xargs rm".
It avoids all sorts of nasty and hard to track down bugs (consider what
happens if you move a .py file from one place in your source tree to
another and leave the old .pyc behind).

 

[Steven D'Aprano]

One thing we do in our Makefiles is "find . -name '*.pyc' | xargs rm".
It avoids all sorts of nasty and hard to track down bugs (consider what
happens if you move a .py file from one place in your source tree to
another and leave the old .pyc behind).

How often do you move files around in the source tree? Meanwhile, *every*
time you run make, you take a performance hit on every Python module in
your project, whether it has moved or not.

Seems to me like a fairly heavy-handed response for something quite rare,
but I suppose that depends on how often you run make.

 

[Benjamin Schollnick]

One thing we do in our Makefiles is "find . -name '*.pyc' | xargs rm".

How often do you move files around in the source tree? Meanwhile, *every*
time you run make, you take a performance hit on every Python module in
your project, whether it has moved or not.

Seems to me like a fairly heavy-handed response for something quite rare,
but I suppose that depends on how often you run make.

If the performance hit doesn't really matter.

Then simply walk the build tree, compare time date stamps, anything that doesn't match up in the make directory, gets deleted. Anything that has different Date Created / Date Modified time from the build tree match, get's deleted.

This way, we are preserving any files that should be identical. But there should be some mechanism documented to forcibly clear the build cache.

- Benjamin

 

[Dave Angel]

How often do you move files around in the source tree? Meanwhile, *every*
time you run make, you take a performance hit on every Python module in
your project, whether it has moved or not.

Seems to me like a fairly heavy-handed response for something quite rare,
but I suppose that depends on how often you run make.

That's why I assumed that his command was triggered by the "make clean"
command.

 

[Roy Smith]

How often do you move files around in the source tree?

It has happened enough times to make us look for a solution. Which
means "more than once".

Meanwhile, *every* time you run make, you take a performance hit on
every Python module in your project, whether it has moved or not.

The performance hit is minimal. The hours of tearing out your hair
trying to figure out why bizarre things are happening is not.

Another solution would be if there was a flag you could give to Python
to tell it, "Only import a .pyc if the corresponding .py file exists".
It's already checking to see if the .py is newer, so this wouldn't even
cost anything.

 

[Steven D'Aprano]

It has happened enough times to make us look for a solution. Which
means "more than once".

Maybe the solution is education, not a technical fix.

I've suspicious of technical fixes for developer problems, because in my
experience that strategy ends in a race to the bottom where you end up with
coding standards and procedures that assume everyone is a code monkey who
can barely spell PC. It doesn't protect the monkeys, because there is no
end to the ways they can screw up, while the competent developers suffer
under repressive, B&D procedures that hinder more than help.

YMMV.

I prefer to keep the .pyc files, and only remove them when necessary, rather
than to remove them whether it's necessary or not. It's not just because
I'm an arrogant SOB who expects my team of developers to know at least more
than me, therefore if I know enough to look for orphaned .pyc files so
should they. It's because I am a big believer that your development system
should be as close as possible to the eventual deployment system as is
practical. Your app will (probably) use .pyc files when it is deployed, so
you should do the same when developing. Otherwise you can get bugs in
deployment that you cannot reproduce in development because the
environments are different.

The performance hit is minimal.

I guess that depends on the size of your project and how much you care about
the start up time. But as general advice, no, it may not be minimal.

[...]

Another solution would be if there was a flag you could give to Python
to tell it, "Only import a .pyc if the corresponding .py file exists".
It's already checking to see if the .py is newer, so this wouldn't even
cost anything.

That's called "Python 3.3"

 

[Roy Smith]

I prefer to keep the .pyc files, and only remove them when necessary, rather
than to remove them whether it's necessary or not. It's not just because
I'm an arrogant SOB who expects my team of developers to know at least more
than me, therefore if I know enough to look for orphaned .pyc files so
should they. It's because I am a big believer that your development system
should be as close as possible to the eventual deployment system as is
practical. Your app will (probably) use .pyc files when it is deployed, so
you should do the same when developing.

Heh. Our deployment system rolls out all the source code from scratch
on every deploy.

Meanwhile, *every* time you run make, you take a performance hit on
every Python module in your project, whether it has moved or not.

Yup.