Too Many if Statements?

[slogging_away]

Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

Due to the number of checks perfromed by the script on the text files,
(over 500), there are quite a few 'if' statements in the script, (over
1150). It seems that it is at the point that when I add any additional
'if' statements the script will not run. No error is produced - it
just returns to the python prompt much the same as when a successful
'Check Module' command is selected. If I delete some other 'if'
statements the new ones work so it appears that it has hit a limit on
the number of 'if' statements. This has stunted any further checks for
the script to make on the text files.

Hs anyone ever run into this sort of thing?

 

[bruno at modulix]

Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

Due to the number of checks perfromed by the script on the text files,
(over 500), there are quite a few 'if' statements in the script, (over
1150). It seems that it is at the point that when I add any additional
'if' statements the script will not run. No error is produced - it
just returns to the python prompt much the same as when a successful
'Check Module' command is selected. If I delete some other 'if'
statements the new ones work so it appears that it has hit a limit on
the number of 'if' statements. This has stunted any further checks for
the script to make on the text files.

Hs anyone ever run into this sort of thing?

Nope - but I never saw any code block with 1150+ tests in it neither :-/

Smells like a design problem anyway. Have you considered refactoring
your code ?

 

[snoe]

Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

Due to the number of checks perfromed by the script on the text files,
(over 500), there are quite a few 'if' statements in the script, (over
1150). It seems that it is at the point that when I add any additional
'if' statements the script will not run. No error is produced - it
just returns to the python prompt much the same as when a successful
'Check Module' command is selected. If I delete some other 'if'
statements the new ones work so it appears that it has hit a limit on
the number of 'if' statements. This has stunted any further checks for
the script to make on the text files.

Hs anyone ever run into this sort of thing?

I can't say I've run into it before but your description makes me think
there are other approaches you could take. Does splitting the large
number of if statements into separate functions help at all?
If you're checking some text to see if a number of static strings are
contained therein, you could put the test strings into a list and loop
through...something like this (untested):

tests = [
'looking for this string',
'and this one',
'am i in the text'
] # etc...

for test in tests:
if test not in text:
raise LookupError

 

[slogging_away]

I don't consider myself to be a seasoned programmer so if you mean
redesigning the script to make the checks and therefore reduce the
number of 'if' statements, I'm not sure if that can be done. The
script needs to make numerous checks for the existence of particular
strings within the configuration file. It also uses 'if' statements to
determine what type of file is being examined, etc.. If an error is
encounterd it writes warning messages to a master file. I guess what I
am trying to say is that in order to make the many checks on the
configuration files I do not know of any other way than to check for
the existance of particular statements, (strings), and then report on
those if they are incorrect or missing - hence at least one 'if'
statement for every check.

I appreciate the feedback though!

 

[slogging_away]

Ah! I see what you are saying snoe, (and most likely what bruno at
modulix was recommending). That technique should provide a workaround
to the direct 'if' approach currently used and also offer some
modularity to the logic as well.

Thank you for pointing me in the right direction - I'll give it a go.

 

[bruno at modulix]

I don't consider myself to be a seasoned programmer

nor do I.

so if you mean
redesigning the script to make the checks and therefore reduce the
number of 'if' statements, I'm not sure if that can be done.

I strongly doubt it could *not* be done !-)

The
script needs to make numerous checks for the existence of particular
strings within the configuration file. It also uses 'if' statements to
determine what type of file is being examined, etc.. If an error is
encounterd it writes warning messages to a master file.

Yeps, that's pretty common with this kind of scripts. I recently had a
script doing thousands of regexp substitutions, image resizing, file
moves, database inserts etc, and of course a fair amount of logging. And
I can tell you that there many few "if" in this code.

I guess what I
am trying to say is that in order to make the many checks on the
configuration files I do not know of any other way than to check for
the existance of particular statements, (strings), and then report on
those if they are incorrect or missing - hence at least one 'if'
statement for every check.

Suppose you have to match a line against a list of regexp and log if it
doesn't match. You could of course repeat the whole code for each
regexp, ie:

if not re.match(r'a/regexp/here', line):
log('a first message')

if not re.match(r'another/regexp/here', line):
log('another message')

(... 150 regexps later ...)

if not re.match(r'150/regexps/later', line):
log('pfww, getting tired of copy/pasting')

etc...

But you could also factor much of it:

def checkMatch(line, regexp, msg):
if not re.match(regexp, line):
log(msg)

then have a list of regexps/messages pairs and:
for exp, msg in regexps:
checkMatch(line, exp, msg)

And now, you can add as many thousands regexps you want, you still have
one (and only one) if in the code (well, in this snippet at least...).

I appreciate the feedback though!

You're welcome !-)

 

[Alan Morgan]

Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

Due to the number of checks perfromed by the script on the text files,
(over 500), there are quite a few 'if' statements in the script, (over
1150). It seems that it is at the point that when I add any additional
'if' statements the script will not run. No error is produced - it
just returns to the python prompt much the same as when a successful
'Check Module' command is selected. If I delete some other 'if'
statements the new ones work so it appears that it has hit a limit on
the number of 'if' statements. This has stunted any further checks for
the script to make on the text files.

Hs anyone ever run into this sort of thing?

I generated files with 10000, 25000, and 50000 simple if statements and ran
them. 10000 was okay, 25000 gave a bizarre internal error, and 50000 segfaulted
and died. My system has plenty of memory and it isn't obvious to me why python
should be so bothered about this. I'm not sure why I can have 10x the number of
if statements that cause you trouble. There might be some overall limitation
on the number of statements in a file.

Alan

 

[Casey Hawthorne]

Try the state(s) pattern!

Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

Due to the number of checks perfromed by the script on the text files,
(over 500), there are quite a few 'if' statements in the script, (over
1150). It seems that it is at the point that when I add any additional
'if' statements the script will not run. No error is produced - it
just returns to the python prompt much the same as when a successful
'Check Module' command is selected. If I delete some other 'if'
statements the new ones work so it appears that it has hit a limit on
the number of 'if' statements. This has stunted any further checks for
the script to make on the text files.

Hs anyone ever run into this sort of thing?

 

[Bryan Olson]

Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

Due to the number of checks perfromed by the script on the text files,
(over 500), there are quite a few 'if' statements in the script, (over
1150). It seems that it is at the point that when I add any additional
'if' statements the script will not run. No error is produced - it
just returns to the python prompt much the same as when a successful
'Check Module' command is selected. If I delete some other 'if'
statements the new ones work so it appears that it has hit a limit on
the number of 'if' statements. This has stunted any further checks for
the script to make on the text files.

Hs anyone ever run into this sort of thing?

I generated files with 10000, 25000, and 50000 simple if statements and ran
them. 10000 was okay, 25000 gave a bizarre internal error, and 50000 segfaulted
and died. My system has plenty of memory and it isn't obvious to me why python
should be so bothered about this. I'm not sure why I can have 10x the number of
if statements that cause you trouble. There might be some overall limitation
on the number of statements in a file.

I made a script with 100,000 if's, (code below) and it appears
to work on a couple systems, including Python 2.4.2 on Win32-XP.
So at first cut, it doesn't seem to be just the if-count that
triggers the bug.

Code that does *not* demo the error on my systems:

#! /usr/bin/env python

lines = ["""#! /usr/bin/env python

import random
c = 0
n = random.randrange(10L**10)
"""]

for i in range(100000):
lines.append('if n % random.randrange(2, 1000) == 0: c += 1')
lines.append('print c')
lines.append('##############')
progtext = '\n'.join(lines)

f = file('manyifs.py', 'w')
f.write(progtext)
f.close()

exec progtext

 

[slogging_away]

Hmmm - good responses all around. Thank you all for your valued
feedback.

Perhaps it's too may 'if' statements under the for XXX in range(x,x,x)
statement as most of the 'if' statements appear there. It could be
something entirely else. I'm afraid its a bug with Python, (if I try
and run it several times it keeps going to the IDLE console prompt and
it eventually crashes out of Python entirely).

Some useful suggestions were provided in terms of better design so that
may be my route at this point. Thanks again for all of your help!

 

[Terry Reedy]

I made a script with 100,000 if's, (code below) and it appears
to work on a couple systems, including Python 2.4.2 on Win32-XP.
So at first cut, it doesn't seem to be just the if-count that
triggers the bug.

The OP did not specify whether all of his if-tests were sequential as in
your test or if some were nested. I vaguely remember there being an indent
limit (40??).

tjr

 

[Alan Morgan]

Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

I generated files with 10000, 25000, and 50000 simple if statements and ran
them. 10000 was okay, 25000 gave a bizarre internal error, and 50000 segfaulted
and died. My system has plenty of memory and it isn't obvious to me why python
should be so bothered about this. I'm not sure why I can have 10x the number of
if statements that cause you trouble. There might be some overall limitation
on the number of statements in a file.

I made a script with 100,000 if's, (code below) and it appears
to work on a couple systems, including Python 2.4.2 on Win32-XP.
So at first cut, it doesn't seem to be just the if-count that
triggers the bug.

Mine was a simple

#!/usr/local/bin/python

zot=24999
if zot == 0:
print "It's 0"

if zot == 1:
print "It's 1"

.....

if zot == 24999:
print "It's 24999"

generated (I'm ashamed to admit) by a perl script. Is there any good reason why
it is failing? I'd prefer a "Too many silly walks in your program. Reduce!" to
a crash. I could experiment with putting the matching 'if' at the beginning
rather than at the end, but I'm not sure what that would tell me.

Alan

 

[Bryan Olson]

The OP did not specify whether all of his if-tests were sequential as in
your test or if some were nested. I vaguely remember there being an indent
limit (40??).

A 40-level indent limit is reasonable (I may get around to looking
it up and/or testing it), but if we believe 'slogging' and Alan, what
we have here is an outright Python bug. They did not report a message
about static code being too deeply nested, nor about dynamic calls
exceeding a recursion limit. They reported incorrect behavior.

My favorite joke goes:

Patient: Doctor...doctor -- it hurts when I do that! (With 'that'
being some movement that is unusual but normally innocuous.)

Doctor: Don't do that.

Ah...cracks me up. Funny on so many levels.

 

[mensanator]

Hmmm - good responses all around. Thank you all for your valued
feedback.

Perhaps it's too may 'if' statements under the for XXX in range(x,x,x)

Have you tried xrange() instead of range()?

 

[nnorwitz]

generated (I'm ashamed to admit) by a perl script. Is there any good reason why
it is failing? I'd prefer a "Too many silly walks in your program. Reduce!" to
a crash.

Everyone,

Please file a bug report anytime you make Python crash!

http://sourceforge.net/tracker/?group_id=5470&atid=105470

If you aren't willing to deal with SF, at least send me source code
(not a description) that makes python crash.

n

 

[Pierre Quentel]

This is because Python has a hidden mechanism to detect programs
generated by Perl scripts, and make them crash with no explanation

Pierre

 

[Juho Schultz]

Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

Due to the number of checks perfromed by the script on the text files,
(over 500), there are quite a few 'if' statements in the script, (over
1150). It seems that it is at the point that when I add any additional
'if' statements the script will not run. No error is produced - it
just returns to the python prompt much the same as when a successful
'Check Module' command is selected. If I delete some other 'if'
statements the new ones work so it appears that it has hit a limit on
the number of 'if' statements. This has stunted any further checks for
the script to make on the text files.

Hs anyone ever run into this sort of thing?

I generated files with 10000, 25000, and 50000 simple if statements
and ran
them. 10000 was okay, 25000 gave a bizarre internal error, and 50000
segfaulted
and died. My system has plenty of memory and it isn't obvious to me
why python
should be so bothered about this. I'm not sure why I can have 10x the
number of
if statements that cause you trouble. There might be some overall
limitation
on the number of statements in a file.

I made a script with 100,000 if's, (code below) and it appears
to work on a couple systems, including Python 2.4.2 on Win32-XP.
So at first cut, it doesn't seem to be just the if-count that
triggers the bug.

I tried that code. It runs fine.

However, the following gives a SystemError with only 2500 elif's.

#!/usr/bin/env python
lines = [ 'n = -1','m = 0','if n < 0:',' m = 2*n',]
for i in range(2500):
lines.append('elif n == %i:' % i)
lines.append(' m = 2*n')
prog = '\n'.join(lines)
progfile = file('if.py','w')
progfile.writelines(prog)
progfile.close()
exec prog

Traceback (most recent call last):
File "iftest.py", line 10, in ?
exec prog
SystemError: com_backpatch: offset too large

I tried this with Python 2.3.3 and 2.3.4 (Linux) and both fail.

 

[bruno at modulix]

This is because Python has a hidden mechanism to detect programs
generated by Perl scripts, and make them crash with no explanation

KEYBOARD !

 

[Nicola Musatti]

bruno at modulix wrote:
[...]

Suppose you have to match a line against a list of regexp and log if it
doesn't match. You could of course repeat the whole code for each
regexp, ie:

if not re.match(r'a/regexp/here', line):
log('a first message')

if not re.match(r'another/regexp/here', line):
log('another message')

(... 150 regexps later ...)

if not re.match(r'150/regexps/later', line):
log('pfww, getting tired of copy/pasting')

etc...

But you could also factor much of it:

def checkMatch(line, regexp, msg):
if not re.match(regexp, line):
log(msg)

then have a list of regexps/messages pairs and:
for exp, msg in regexps:
checkMatch(line, exp, msg)

And now, you can add as many thousands regexps you want, you still have
one (and only one) if in the code (well, in this snippet at least...).

If your checks are this complicated, I think you should consider
writing a parser for your configuration file. If you use a parser
generator it's not that difficult. Moreover a lexical analyzer could be
enough if your syntax is simple. I found Dave Beazley's PLY reasonably
easy to use: http://www.dabeaz.com/ply/

Cheers,
Nicola Musatti

 

[Alan Morgan]

This is because Python has a hidden mechanism to detect programs
generated by Perl scripts, and make them crash with no explanation

In my case it turned out to be python having a hidden method to detect when
you are using an ancient version of python. Retesting with a newer version
didn't find any problems.

Alan