[ANN] DoIt 0.1.0 Released (build tool)

[Eduardo Schettino]

DoIt - A task execution tool (build-tool)
=========================================

This is the first public release of DoIt

Website: http://python-doit.sourceforge.net/
Release: DoIt 0.1.0
License: MIT

About
-----

DoIt is a build tool that focus not only on making/building things but on
executing any kind of tasks in an efficient way. Designed to be easy to use
and "get out of your way".

DoIt like most build tools is used to execute tasks defined in a
configuration file. Configuration files are python modules. The tasks can be
python functions (or any callable) or an external shell script. DoIt
automatically keeps track of declared dependencies executing only tasks that
needs to be update (based on which dependencies have changed).

In DoIt, unlike most(all?) build-tools, a task doesn't need to define a target
file to use the execute only if not up-to-date feature. This make DoIt
specially suitable for running test suites.

DoIt can be used to perform any task or build anything, though it doesn't
support automatic dependency discovery for any language.

Cheers,
Eduardo

 

[John Machin]

DoIt - A task execution tool (build-tool)
=========================================

This is the first public release of DoIt

Website: http://python-doit.sourceforge.net/
Release: DoIt 0.1.0
License: MIT

About
-----

DoIt is a build tool that focus not only on making/building things but on
executing any kind of tasks in an efficient way. Designed to be easy to use
and "get out of your way".

You may like to consider the possibility of confusion caused by the
similarity of some characters in some fonts (DoIt, Do1t, Dolt) ...
google("dictionary dolt")

 

[Eduardo Schettino]

You may like to consider the possibility of confusion caused by the
similarity of some characters in some fonts (DoIt, Do1t, Dolt) ...
google("dictionary dolt")

hehehehe. i feel like a DOLT

I never realized that it is confusing. I also didnt know this word
"dolt" before.
i wont use capital letters anymore.
thanks

 

[Ville M. Vainio]

DoIt is a build tool that focus not only on making/building things but on
executing any kind of tasks in an efficient way. Designed to be easy to use
and "get out of your way".

I took a look at dolt syntax, and saw this:

QQQ

def create_folder(path):
"""Create folder given by "path" if it doesnt exist"""
if not os.path.exists(path):
os.mkdir(path)
return True

def task_create_build_folder():
buildFolder = jsPath + "build"
return {'action':create_folder,
'args': (buildFolder,)
}

QQQ

Wouldn't it be more convenient to provide syntax like this:

@task("create_build_folder")
@depend("dep1 some_other_dep")
def buildf():
buildFolder = jsPath + "build"
create_folder(buildFolder)

I find the doit syntax a bit cumbersome, especially as you can avoid
'args' by just returning a lamda in 'action'.

I've looked around a bit for python "make" replacement, but there does
not seem to be a simple & straightforward solution around (read -
straight-python syntax, one .py file installation, friendly license).

 

[Paul Boddie]

Wouldn't it be more convenient to provide syntax like this:

@task("create_build_folder")
@depend("dep1 some_other_dep")
def buildf():
buildFolder = jsPath + "build"
create_folder(buildFolder)

I'd want to make the "grunt work" a bit easier before breaking out the
decorators.

I find the doit syntax a bit cumbersome, especially as you can avoid
'args' by just returning a lamda in 'action'.

I've looked around a bit for python "make" replacement, but there does
not seem to be a simple & straightforward solution around (read -
straight-python syntax, one .py file installation, friendly license).

Have you surveyed the landscape...?

http://wiki.python.org/moin/ConfigurationAndBuildTools

I'm inclined to think that Waf would probably meet your requirements:

http://code.google.com/p/waf/

Paul

 

[Eduardo Schettino]

I took a look at dolt syntax, and saw this:

QQQ

def create_folder(path):
"""Create folder given by "path" if it doesnt exist"""
if not os.path.exists(path):
os.mkdir(path)
return True

def task_create_build_folder():
buildFolder = jsPath + "build"
return {'action':create_folder,
'args': (buildFolder,)
}

QQQ

Wouldn't it be more convenient to provide syntax like this:

@task("create_build_folder")
@depend("dep1 some_other_dep")
def buildf():
buildFolder = jsPath + "build"
create_folder(buildFolder)

I find the doit syntax a bit cumbersome, especially as you can avoid
'args' by just returning a lamda in 'action'.

My idea was to: do *not* add any new syntax (to avoid being
cumbersome). It is just python, you dont have to import or subclass
anything. You just need to create a function that returns a dictionary
with some predefined keys.

I though about using decorators in the beginning... but returning a
dictionary looked easier to implement and more flexible. one important
feature is how easy to define a group of task with the same action.
Take a look at the example below on running pychecker in all python
files from a folder. I couldnt figure out an easy way of doing it with
decorators.

import glob;
pyFiles = glob.glob('*.py')

def task_checker():
for f in pyFiles:
yield {'action': "pychecker %s"% f,
'name':f,
'dependencies'f,)}

Another advantage of using just a dictionary to define a task is that
it will be easy to read tasks from a text file (if it is very simple
and you dont need to write any python script). but not implemented
yet.

I though about using decorator for simple python-tasks but in a different way:

@task
def create_folder(path):
"""Create folder given by "path" if it doesnt exist"""
if not os.path.exists(path):
os.mkdir(path)
return True

so if your python function is a task and you will use it only once you
dont need to define a function for the 'action' and another function
to create the task. but not implement yet also.

I've looked around a bit for python "make" replacement, but there does
not seem to be a simple & straightforward solution around (read -
straight-python syntax, one .py file installation, friendly license).

apart from one .py file installation (easy_install is not enough?)
thats what i am trying to do.

thanks for the feedback.
cheers,
Eduardo

 

[Eduardo Schettino]

I guess I should post a link to the project in this thread...

http://python-doit.sourceforge.net/

 

[Ville M. Vainio]

My idea was to: do *not* add any new syntax (to avoid being
cumbersome). It is just python, you dont have to import or subclass

Yeah, decorators get around this.

I though about using decorators in the beginning... but returning a
dictionary looked easier to implement and more flexible. one important
feature is how easy to define a group of task with the same action.
Take a look at the example below on running pychecker in all python
files from a folder. I couldnt figure out an easy way of doing it with
decorators.

import glob;
pyFiles = glob.glob('*.py')

def task_checker():
for f in pyFiles:
yield {'action': "pychecker %s"% f,
'name':f,
'dependencies'f,)}

Perhaps you could do:

for f in pyFiles:
@task("checker")
@depend(f)
def check():
c("pychecker %s" % f)

Never underestimate the magic that is nested scopes and name-agnostic
function object creation...

Another advantage of using just a dictionary to define a task is that
it will be easy to read tasks from a text file (if it is very simple
and you dont need to write any python script). but not implemented
yet.

It is easy with the above syntax as well.

I though about using decorator for simple python-tasks but in a different way:

@task
def create_folder(path):
"""Create folder given by "path" if it doesnt exist"""
if not os.path.exists(path):
os.mkdir(path)
return True

so if your python function is a task and you will use it only once you
dont need to define a function for the 'action' and another function
to create the task. but not implement yet also.

Yeah, this is what I consider much friendlier syntax (the aim is to not
be much more verbose than make).

apart from one .py file installation (easy_install is not enough?)
thats what i am trying to do.

easy_install is not really enough - it introduces a dependency that you
can't get around by just shipping a short .py file with your project.

This problem domain seems simple enough that it should be covered by a
very short and simple module (Paul, the Waf you suggested was ~ 300k, no
doubt caused by all the c compiler stuff that I don't need).

'Paver' seems to have the right idea:

http://www.blueskyonmars.com/projects/paver/index.html

But it's still *slightly* too big:

http://bazaar.launchpad.net/~dangoor/paver/main/files

 

[Eduardo Schettino]

Yeah, decorators get around this.


Perhaps you could do:

for f in pyFiles:
@task("checker")
@depend(f)
def check():
c("pychecker %s" % f)

Never underestimate the magic that is nested scopes and name-agnostic
function object creation...

ok. it is possible to do everything with decorators... i agree.
but i dont want "name-agnostic function object creation". i want to be
able to execute every single task without executing all other tasks.
thats why when defining sub-tasks it is required an extra parameter
"name". I know you could add another decorator for this also
I mean decorators could do the job i dont say they cant. *I* prefer to
work with dictionaries though. Even if they are a bit more verbose.

Yeah, this is what I consider much friendlier syntax (the aim is to not be
much more verbose than make).

Maybe it is just a bad example. I tried to put all features on single
example. but in *real* life I could do like this.

import os
jsPath = "./"
jsFiles = ["file1.js", "file2.js"]
sourceFiles = [jsPath + f for f in jsFiles]
compressedFiles = [jsPath + "build/" + f + ".compressed" for f in jsFiles]

def task_shrink_js():
# create output folder
if not os.path.exists(path):
os.mkdir(path)
for jsFile,compFile in zip(sourceFiles,compressedFiles):
action = 'java -jar custom_rhino.jar -c %s > %s'% (jsFile, compFile)
yield {'action':action,
'name':jsFile,
'dependencies'jsFile,),
'targets'compFile,)
}

Can I challenge you to do this with any other build tool?

i guess anything more complicated than this you want to create a
python function separate from the task definition anyway.

My examples are from scratch. I dont use a "standard library".

I could also document the short-cuts . I was afraid that including
the short-cuts would make it more complicated for who is trying to
learn it.

if you dont define any dependencies or args to the function you dont
need to return a dictionary. just the action:

def task_simple_shell():
return "echo spam"

def task_simple_python():
def say_spam():
print "spam"
return True
return say_spam

# or from our discussion

def task_create_build_folder():
def create_folder():
path = jsPath + "build"
if not os.path.exists(path):
os.mkdir(path)
return True
return create_folder

easy_install is not really enough - it introduces a dependency that you
can't get around by just shipping a short .py file with your project.

what about shipping a single egg file with the whole package? ( I am
not very much familiar with this subject)

'Paver' seems to have the right idea:

http://www.blueskyonmars.com/projects/paver/index.html

Thanks for the link. I took a quick look on its documentation.
It seems that "paver" doesnt keep track of file-dependencies at all.
So it is more like a setuptools extension commands than a
"full-feature" build system with "smart" re-build and dependency
support.

"doit" target is not on creating releases for python projects only. it
can do it also, but it can do much more.

You can read about my motivation to start another build tool project
on http://schettino72.wordpress.com/2008/04/14/doit-a-build-tool-tale/

But it's still *slightly* too big:

man, it is hard to make you happy

the doit egg file containg the whole packge is 27782 bytes on my
system. but you also need the command line script 154 bytes.

cheers,
Eduardo