Using Python to generate code?

[Tran Tuan Anh]

Dear all:

I need your advice on this matter. I am working on a program which
takes some pieces of System-C code in and generate some other System-C
code. (System-C code is just C++ with some special libraries, hence
you can consider it as C++).

Right now, the generator program is written in C++. And I feel not so
comfortable with C++. I feel C++ is an overkill. Because, I need to
generate some code, hence in the program there are a lot of something
like this:

printf(out, "for (%s = 1, %s < %s, %s < %s )", varName, varName,
varName1, varname, varName2);

It is just too messy if I have more than 20 lines like this.

So my question is:
1. Can Python help to solve this issue?
2. Does Python has a parser? It seems to me that there is no-standard
Parser for Python right now. If it is the case, can I interface Python
with existing cup and lex?

Many thanks!
Tuan Anh
P/S: Also because it is NO big code generator, it only needs to
generate some simple codes. Hence, using some abstract-syntax-tree
library is also too an overkill.

 

[Jeremy Bowers]

printf(out, "for (%s = 1, %s < %s, %s < %s )", varName, varName,
varName1, varname, varName2);

It is just too messy if I have more than 20 lines like this.

It looks like you want a templater. I don't know what such programs exist
for C++, if any, or how good they are.

You may get a lot of milage out of just adding some utility functions, to
turn that to something like

forFromOne(out, var, max, min);

Changing languages is pretty drastic if you have working, trusted code.
(If those words don't describe your code, then that assessment may change.)

So my question is:
1. Can Python help to solve this issue?

Maybe a little, but if the above code snippet is representative, the
Python equivalent of that isn't much shorter. Python would start to shine
if you want to start doing smarter things with the data, but without
knowing about the data it is hard to be specific. Basically, the moment
you aren't just assembling strings, Python will *quite likely* have an
advantage.

(I find it very likely that a Pythonic outlook on the problem would allow
significant simplification at acceptable effort levels, but I can't
*promise* that; if all you really, honestly need is printf, which I think
would be rare but certainly conceivable, then you may get no advantage.)

2. Does Python has a parser? It seems to me that there is no-standard
Parser for Python right now. If it is the case, can I interface Python
with existing cup and lex?

Why are you asking about a parser when you are talking about *outputting*
code? Is there something left out of the problem description?

There are a lot of parsers for Python, covering a wide range of needs.

 

[Andrei]

1. Can Python help to solve this issue?

I have used Python for quite simple code generation/templating - not in C++, but
that makes no difference. There are even special and not that complex tools for
code generation, e.g. Cog (http://www.nedbatchelder.com/code/cog/index.html).

2. Does Python has a parser? It seems to me that there is no-standard

There are all kinds of parsers, but I've never used any of them. Google gives
tons of links when searching for "python parser", so a more specific search
would be wise.

Yours,

Andrei

 

[Larry Bates]

I use Python to generate quite a bit of HTML and
JavaScript code and it works very well. If you
can use Python's classes to abstract individual
elements and give each of them a __str__ method
you can build very complex code generator classes
that know how to render their own output.

Larry Bates
Syscon, Inc.

 

[Albert Hofkamp]

Right now, the generator program is written in C++. And I feel not so
comfortable with C++. I feel C++ is an overkill. Because, I need to
generate some code, hence in the program there are a lot of something
like this:

printf(out, "for (%s = 1, %s < %s, %s < %s )", varName, varName,
varName1, varname, varName2);

It is just too messy if I have more than 20 lines like this.

Agreed, reached that same conclusion.
Therefore, I switched to something like

output="""for ($VAR = 1, $VAR < $START, $VAR < $END)"""

output=string.replace(output,"$VAR",varName)
output=string.replace(output,"$START",varName1)
output=string.replace(output,"$END",varName2)

Obviously, this can be enhanced.
(use other conventions than $identifier, and performing the substitution
in a loop).
BTW: I am not sure string.replace works OK as shown here.

In a sense, I am using the string replacement to substitute variables
names thus making my own text templating system.

1. Can Python help to solve this issue?
yep.

2. Does Python has a parser? It seems to me that there is no-standard

yes, several.
I have good experiences with spark. you have to be careful what you
enter, and it is not fast, but it can handle just about any
grammar you can think of.

P/S: Also because it is NO big code generator, it only needs to
generate some simple codes. Hence, using some abstract-syntax-tree
library is also too an overkill.

Anything small, simple, and beautiful grows ......




Albert

 

[Alex Martelli]

Why are you asking about a parser when you are talking about *outputting*
code? Is there something left out of the problem description?

the original poster started his original post with:

I need your advice on this matter. I am working on a program which
takes some pieces of System-C code in and generate some other System-C

That "takes some pieces of ... code" sounded to me like an indication
that a parser is needed -- nothing left out of the problem description,
IMHO.


Alex

 

[Alex Martelli]

Agreed, reached that same conclusion.
Therefore, I switched to something like

output="""for ($VAR = 1, $VAR < $START, $VAR < $END)"""

Uh, OT, but, doesn't this work better with semicolons than commas...?

output=string.replace(output,"$VAR",varName)
output=string.replace(output,"$START",varName1)
output=string.replace(output,"$END",varName2)

Obviously, this can be enhanced.
(use other conventions than $identifier, and performing the substitution
in a loop).
BTW: I am not sure string.replace works OK as shown here.

Yep, though output.replace(...) would be neater. But the best
replacement is already in the Python 2.4 standard library: it uses
exactly the $identifier convention, and takes a dictionary of mapping of
identifier to string...:

In [4]: tpl=string.Template('for ($VAR = 1, $VAR < $START, $VAR <
$END)')

In [5]: tpl % dict(VAR='foo', START='bar', END='baz')
Out[5]: u'for (foo = 1, foo < bar, foo < baz)'

In [6]: tpl % dict(VAR='fee', START='fie', END='fofum')
Out[6]: u'for (fee = 1, fee < fie, fee < fofum)'



Alex

 

[Alexis Roda]

output=string.replace(output,"$VAR",varName)
output=string.replace(output,"$START",varName1)
output=string.replace(output,"$END",varName2)

Obviously, this can be enhanced.
(use other conventions than $identifier, and performing the substitution
in a loop).
BTW: I am not sure string.replace works OK as shown here.

Yep, though output.replace(...) would be neater. But the best
replacement is already in the Python 2.4 standard library: it uses
exactly the $identifier convention, and takes a dictionary of mapping of
identifier to string...:

In [4]: tpl=string.Template('for ($VAR = 1, $VAR < $START, $VAR <
$END)')

In [5]: tpl % dict(VAR='foo', START='bar', END='baz')
Out[5]: u'for (foo = 1, foo < bar, foo < baz)'

In [6]: tpl % dict(VAR='fee', START='fie', END='fofum')
Out[6]: u'for (fee = 1, fee < fie, fee < fofum)'

If you are not contrained to using $ identifiers:

d={'var':'foo', 'start':1, 'end': 100}
'for (%(var)s = %(start)i; %(var)s < %(end)i; %(var)s++)' % d

-> 'for (foo = 1; foo < 100; foo++)'



Regards
--
////
(@ @)
----------------------------oOO----(_)----OOo--------------------------
<> Ojo por ojo y el mundo acabara ciego
/\ Alexis Roda - Universitat Rovira i Virgili - Reus, Tarragona (Spain)
-----------------------------------------------------------------------

 

[Alex Martelli]

Yep, though output.replace(...) would be neater. But the best
replacement is already in the Python 2.4 standard library: it uses
exactly the $identifier convention, and takes a dictionary of mapping of
identifier to string...: ...
In [6]: tpl % dict(VAR='fee', START='fie', END='fofum')
Out[6]: u'for (fee = 1, fee < fie, fee < fofum)'

If you are not contrained to using $ identifiers:

d={'var':'foo', 'start':1, 'end': 100}
'for (%(var)s = %(start)i; %(var)s < %(end)i; %(var)s++)' % d

yeah, but the template string for this long-standing Python feature is
hard to read and hard to write, which is why 2.4 finally grew a simpler
approach -- the string.Template class.


Alex

 

[Alexis Roda]

Yep, though output.replace(...) would be neater. But the best
replacement is already in the Python 2.4 standard library: it uses
exactly the $identifier convention, and takes a dictionary of mapping of
identifier to string...:

...

In [6]: tpl % dict(VAR='fee', START='fie', END='fofum')
Out[6]: u'for (fee = 1, fee < fie, fee < fofum)'

If you are not contrained to using $ identifiers:

d={'var':'foo', 'start':1, 'end': 100}
'for (%(var)s = %(start)i; %(var)s < %(end)i; %(var)s++)' % d

yeah, but the template string for this long-standing Python feature is
hard to read and hard to write, which is why 2.4 finally grew a simpler
approach -- the string.Template class.


Alex

If you come from Perl its not so hard . Anyway, I agree, templates
are nicer, but not everyone is running python 2.4 ...


Alex
--
////
(@ @)
----------------------------oOO----(_)----OOo--------------------------
<> Ojo por ojo y el mundo acabara ciego
/\ Alexis Roda - Universitat Rovira i Virgili - Reus, Tarragona (Spain)
-----------------------------------------------------------------------

 

[Steven Bethard]

yeah, but the template string for this long-standing Python feature is
hard to read and hard to write, which is why 2.4 finally grew a simpler
approach -- the string.Template class.

Just to make the comparison as fair as possible:

2.4 Template solution
tpl = string.Template('for ($VAR = 1, $VAR < $START, $VAR < $END)')
tpl % dict(VAR='fee', START='fie', END='fofum')

<2.4 format string solution
tpl = 'for (%(VAR)s = 1, %(VAR)s < %(START)s, %(VAR)s < %(END)s)'
tpl % dict(VAR='fee', START='fie', END='fofum')

So, yes, the Template solution is cleaner, but not drastically so. It saves
you 3 characters per variable -- '(', ')' and 's'. Of course the more
variables you have to write, the bigger the deal it is, so I suspect the OP
would much prefer the Template solution.

Steve

 

[Tim Williams]

How can I get the name (or ip) of the machine where my python script is
running? I'd like to add it to my log entry.

import socket

print socket.gethostname()
print socket.getfqdn()

this_host = socket_gethostname()
print socket.gethostbyname(this_host)

# the following will give several IP addresses if you are
#multi-homed

print socket.gethostbyname_ex(this_host)

 

[Jeffrey Froman]

Steven Bethard wrote:

So, yes, the Template solution is cleaner, but not drastically so.  It
saves you 3 characters per variable -- '(', ')' and 's'.  Of course the
more variables you have to write, the bigger the deal it is

True, and calling string.Template() is an extra 17 characters, so there
would have to be 6 variables in your string before you got a typing
payoff. Of course, some people might prefer the look of the Template.

Jeffrey

 

[Tran Tuan Anh]

Many thanks for your comments and advice!

Another concern is that how I can interface Python with Cup/Lex.
Writing parser is a no-joke and tedious to me, hence I would like to
re-use
the existing Cup/Lex scanner and parser.
Could anyone confirm that it is possible to do so? (I mean naturally


In general, the program I am dealing with needs to manipulate strings
quite a lot. Eventhough we can certainly do with C++, it does not
appeal naturally to me.

 

[Tran Tuan Anh]

- I'm familar with Java, C, C++, and Pascal, some experiences with
SML. Could you elaborate about some Python's string-manipulation
features?

- So now imagine if I have to convince my supervisor, an
theoretic-non-progammer, about switching to Python for this code
generator program, what kind of advantages I can get from doing so?

Thanks!
Tuan Anh

 

[Albert Hofkamp]

- I'm familar with Java, C, C++, and Pascal, some experiences with
SML. Could you elaborate about some Python's string-manipulation
features?

I can never beat the online documentation:

http://www.python.org/doc, click 'Module index', click 'strings'.

I don't know where the string template thingies are described, but it
has to be there somewhere.

- So now imagine if I have to convince my supervisor, an
theoretic-non-progammer, about switching to Python for this code
generator program, what kind of advantages I can get from doing so?

Same:

http://www.python.org/ look in the 'Documentation links', in particular
the 'introductions to python'. There is a section of links to convince
those that control what we do (they think ).


[ python documentation is awesome, very complete and very good. Browse
the website for answers to just about any question.
]


Albert

 

[Albert Hofkamp]

the original poster started his original post with:


That "takes some pieces of ... code" sounded to me like an indication
that a parser is needed -- nothing left out of the problem description,
IMHO.

For the original poster:

I just discovered that there is a Topic guide about scanners/parsers at
python.org !!
It has quite a list of module links.


Albert