Using Python to automate builds

[Kosta]

I am a Python newbie, tasked with automating (researching) building
Windows drivers using the WDK build environment. I've been looking
into Python for this (instead of writing a bunch of batch files).

What I would like to do, is to open a cmd window, and start a Python
script. This script would then (based upon input arguments), build
different flavors of the driver (fre, chk, x86, x64) and do some post
processing (create cat files, sign, etc.).

I was hoping to take advantage as much as possible of exisiting
infrastructure from the WDK. I am able to call setenv.bat and provide
all the input parameters. One of the things setenv.bat does is change
the path environment variable. However, this is not captured by
Python. I could duplicate the functionality, but I'd rather not. Is
there a way to capture all enviroment variable changes being made by a
batch file from Python?

Thanks!

 

[Dave Angel]

I am a Python newbie, tasked with automating (researching) building
Windows drivers using the WDK build environment. I've been looking
into Python for this (instead of writing a bunch of batch files).

What I would like to do, is to open a cmd window, and start a Python
script. This script would then (based upon input arguments), build
different flavors of the driver (fre, chk, x86, x64) and do some post
processing (create cat files, sign, etc.).

I was hoping to take advantage as much as possible of exisiting
infrastructure from the WDK. I am able to call setenv.bat and provide
all the input parameters. One of the things setenv.bat does is change
the path environment variable. However, this is not captured by
Python. I could duplicate the functionality, but I'd rather not. Is
there a way to capture all enviroment variable changes being made by a
batch file from Python?

Thanks!

An excellent question. It's been a few years, but last time I remember
looking, setenv.bat only changed environment variables, and didn't
change current directory, or run any external programs.. If that's
still true, I figure you have three options.

I don't know of any way to capture them explicitly. So here are the
three choices I can come up with:

1) have the first python program invoke a new shell which runs setenv,
then runs another python program. This second python program would be
where all the work is done, invoking compilers, linkers etc. So the
first python program invokes a batch file which looks something like:
call setenv.bat arg1
python second.py arg2 arg3 arg4 arg5

This batch file could even be generated on the fly, just to simplify
complex argument passing.

2) Have the python program invoke a batch file something like the
following, and then analyze the resulting text file
call setenv.bat arg1
set > tempfile.txt

The contents of that tempfile could then be loaded into a dict similar
to the usual environment. Now the Python script can continue, and just
use these environment variable instead of the original set

3) Have the python program parse the setenv.bat instead of running it.

There was a post on this newsgroup announcing pyKook. It's still in
early form, but you might get ideas from it. I haven't looked, I just
pasted the announcement to look at later.

+ I have released pyKook 0.0.2.
+ http://pypi.python.org/pypi/Kook/0.0.2
+ http://www.kuwata-lab.com/kook/
+ http://www.kuwata-lab.com/kook/pykook-users-guide.html

Other possibilities:
+ http://pypi.python.org/pypi/vellum/ flexible small 'make'
alternative

+ http://code.google.com/p/waf/

+ http://code.google.com/p/fabricate/

DaveA

 

[Kosta]

An excellent question.  It's been a few years, but last time I remember
looking, setenv.bat only changed environment variables, and didn't
change current directory, or run any external programs..  If that's
still true, I figure you have three options.

I don't know of any way to capture them explicitly.  So here are the
three choices I can come up with:

1) have the first python program invoke a new shell which runs setenv,
then runs another python program.  This second python program would be
where all the work is done, invoking compilers, linkers etc.  So the
first python program invokes a batch file which looks something like:
          call setenv.bat  arg1
          python second.py    arg2 arg3  arg4  arg5

 This batch file could even be generated on the fly, just to simplify
complex argument passing.

2) Have the python program invoke a batch file something like the
following, and then analyze the resulting text file
           call setenv.bat   arg1
           set > tempfile.txt

The contents of that tempfile could then be loaded into a dict similar
to the usual environment.  Now the Python script can continue, and just
use these environment variable instead of the original set

3) Have the python program parse the setenv.bat instead of running it.

There was a post on this newsgroup announcing pyKook.  It's still in
early form, but you might get ideas from it.  I haven't looked, I just
pasted the announcement to look at later.

+ I have released pyKook 0.0.2.
+http://pypi.python.org/pypi/Kook/0.0.2
+http://www.kuwata-lab.com/kook/
+http://www.kuwata-lab.com/kook/pykook-users-guide.html

Other possibilities:
+  http://pypi.python.org/pypi/vellum/      flexible small 'make'
alternative

+  http://code.google.com/p/waf/

+  http://code.google.com/p/fabricate/

DaveA- Hide quoted text -

- Show quoted text -

Thanks Dave. I had thought about those three options, and was
honestly hoping for a foruth (I know, some people are never
satisfied . I'll look into pyKook. Thank you for your help.

 

[Philip Semanchuk]

Thanks Dave. I had thought about those three options, and was
honestly hoping for a foruth (I know, some people are never
satisfied . I'll look into pyKook. Thank you for your help.

Poof! Your wish is granted! =)

http://www.scons.org/

Dunno if you'll find it better, worse or different than the
alternatives, but there it is.

have fun
P

 

[Hendrik van Rooyen]

I am a Python newbie, tasked with automating (researching) building
Windows drivers using the WDK build environment. I've been looking
into Python for this (instead of writing a bunch of batch files).

Why do you not use make and a makefile - it was designed to
do exactly this.

- Hendrik

 

[Marcus Wanner]

Poof! Your wish is granted! =)

http://www.scons.org/

Dunno if you'll find it better, worse or different than the
alternatives, but there it is.

have fun
P

I can highly recommend scons. At dolphin-emu, we use it to daily compile
a project containing over 500,000 lines of c/c++ code, which is
modified very frequently. It works like a charm, and is seamlessly
cross- platform (in my experience). However, it might not be exactly the
thing you're looking for.

Marcus

 

[David Cournapeau]

Why do you not use make and a makefile - it was designed to
do exactly this.

Because make is a pain on windows (lack of a proper shell) ?

cheers,

David

 

[Hendrik van Rooyen]

Because make is a pain on windows (lack of a proper shell) ?

Well, you could always use Dosbox - I am doing exactly that
on Linux to be able to run some old DOS based assembler
tools. It will run on windows too, they claim.

And it is very simple to use too - no fancy fidgeting, all you have
to do is to point it at a directory that becomes your "C" drive.

Bit slow - but hey, nobody's perfect.

- Hendrik

 

[Kosta]

Well, you could always use Dosbox - I am doing exactly that
on Linux to be able to run some old DOS based assembler
tools.  It will run on windows too, they claim.

And it is very simple to use too - no fancy fidgeting, all you have
to do is to point it at a directory that becomes your "C" drive.

Bit slow - but hey, nobody's perfect.

- Hendrik

Sorry for not being more clear. The WDK provides build.exe, a utility
that calls the compiler & linker with the approprite switches. In
essence, it manages all the nmake details away from the developer.
For some, this is a bad thing in terms of loss of control. But in
reality, this is a great thing, as this is the way Microsoft builds
Windows. As far as I can tell, the WDK build environment is dervied
from the Microsoft Windows build environment, which is great for
developing kernel components in the similar enviornment in which
Microsoft builds its OS. It is a requirement to use the WDK.

Setenv.bat sets up the path and other environment variables build.exe
needs to compile and link (and even binplace) its utilities. So
building itself is not the issue. The problem is that if I call
setenv.bat from Python and then build.exe, but the modifications to
the path (and other environment settings) are not seen by Python, so
the attempt to build without a specified path fails.

Now I can open up a build window that targets a build type (check or
release), an architecture target (x86 or x64) and an OS target (which
will be Windows Xp except if in the future, the driver(s) uses some
infrastructure introduced in later versions of WDM.

However, what I was hoping to do was to have a python script that
would allow me to build 4 versions of the driver: x86 check, x86
release, x64 check, and x64 release. I may have to resort to using
batch files, but I was hoping as much as possible to use Python.

Thanks,
Kosta

 

[Tim Arnold]

I am a Python newbie, tasked with automating (researching) building
Windows drivers using the WDK build environment. I've been looking
into Python for this (instead of writing a bunch of batch files).

What I would like to do, is to open a cmd window, and start a Python
script. This script would then (based upon input arguments), build
different flavors of the driver (fre, chk, x86, x64) and do some post
processing (create cat files, sign, etc.).

I was hoping to take advantage as much as possible of exisiting
infrastructure from the WDK. I am able to call setenv.bat and provide
all the input parameters. One of the things setenv.bat does is change
the path environment variable. However, this is not captured by
Python. I could duplicate the functionality, but I'd rather not. Is
there a way to capture all enviroment variable changes being made by a
batch file from Python?

Thanks!

It sounds like you know what you're doing. I just wanted to point out that
you can specify environment dictionaries in the subprocess.Popen module. You
can set a different dictionary for each call you make to the system. Not
sure if that helps,
--Tim

 

[David Cournapeau]

Setenv.bat sets up the path and other environment variables build.exe
needs to compile and link (and even binplace) its utilities.  So
building itself is not the issue.  The problem is that if I call
setenv.bat from Python and then build.exe, but the modifications to
the path (and other environment settings) are not seen by Python, so
the attempt to build without a specified path fails.

It sounds like you do not propagate the environment when calling
setenv.bat from python. There is an option to do so in
subprocess.Popen init method, or you can define your own environment
if you do not want to propagate the whole environment (but this is
often difficult to avoid for build environment in my experience,
expecially if you don't have access to the sources of the whole system
to check which variables are necessary).

David

 

[Kosta]

It sounds like you do not propagate the environment when calling
setenv.bat from python. There is an option to do so in
subprocess.Popen init method, or you can define your own environment
if you do not want to propagate the whole environment (but this is
often difficult to avoid for build environment in my experience,
expecially if you don't have access to the sources of the whole system
to check which variables are necessary).

David

David,

Thanks you. I looked up the docs on Popen (http://docs.python.org/
library/subprocess.html) where I read:

On Windows: the Popen class uses CreateProcess() to execute the child
program, which operates on strings. If args is a sequence, it will be
converted to a string using the list2cmdline() method. Please note
that not all MS Windows applications interpret the command line the
same way: list2cmdline() is designed for applications using the same
rules as the MS C runtime.

My interpretation of the above (and your email) is that using Popen
allows one to pass the Python environment to a child processs (in my
case, setenv.bat). I need the reverse, to propagate from the child
to the parent.

Thanks,
Kosta

 

[Piet van Oostrum]

K> My interpretation of the above (and your email) is that using Popen
K> allows one to pass the Python environment to a child processs (in my
K> case, setenv.bat). I need the reverse, to propagate from the child
K> to the parent.

I don't think there is any modern OS that allows that. Unless you use
your own protocol of course, like letting the child write the
environment to its stdout and reading and interpreting it in the parent.

 

[Aahz]

Bit slow - but hey, nobody's perfect.

YM "pobody's nerfect" HTH HAND
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"...string iteration isn't about treating strings as sequences of strings,
it's about treating strings as sequences of characters. The fact that
characters are also strings is the reason we have problems, but characters
are strings for other good reasons." --Aahz

 

[Kosta]

I don't think there is any modern OS that allows that. Unless you use
your own protocol of course, like letting the child write the
environment to its stdout and reading and interpreting it in the parent.

Piet,

Yes you are correct. Thinking more about parent/child processes and
what I am doing: opening a cmd window (its own process), starting up
Python (a child process), and then attempting to run setenv.bat (a
child process to Python), and yes I'm out of luck.

 

[Gabriel Genellina]

My interpretation of the above (and your email) is that using Popen
allows one to pass the Python environment to a child processs (in my
case, setenv.bat). I need the reverse, to propagate from the child
to the parent.

In addition to just calling setenv, dump the modified environment. Parse
the output into a dictionary that you can use as the env argument to later
subprocess calls:

py> import subprocess
py> cmdline = '(call setenv >nul 2>&1 & set)'
py> p = subprocess.Popen(cmdline, stdout=subprocess.PIPE, shell=True)
py> env = dict(line.rstrip().split("=",1) for line in p.stdout)
py> p.wait()
0
py> env
{'TMP': 'D:\\USERDATA\\Gabriel\\CONFIG~1\\Temp', 'COMPUTERNAME': 'LEPTON',
'HOMEDRIVE': 'D:', ...
py> env['FOO']='Salta Violeta!'
py> subprocess.call("echo %FOO%", shell=True, env=env)
Salta Violeta!
0