lint for Python?

[Pat]

I've been searching for a good multi-module lint checker for Python and
I haven't found one yet.

Pylint does a decent job at checking for errors only within a single module.

Here's one of my problems. I have two modules.

In module one, I have a function:

def foo( host, userid, password ):
pass

In module two, I call that function:

foo( userid, password)

lint doesn't find that error and it won't be caught until it's called
while the program is running. I don't want that error found at 3AM.

I've never used a language that didn't catch that type of error. I'm
quite surprised that Python is being used by a number of major
companies. How you catch these types of errors?

I've spoken to tech support at Wing and they weren't aware of a
multi-module lint but they're considering putting their own lint into
their IDE.

Thank you

 

[Chris Rebert]

I've been searching for a good multi-module lint checker for Python and I
haven't found one yet.

Pylint does a decent job at checking for errors only within a single module.

Here's one of my problems. I have two modules.

In module one, I have a function:

def foo( host, userid, password ):
pass

In module two, I call that function:

foo( userid, password)

lint doesn't find that error and it won't be caught until it's called while
the program is running. I don't want that error found at 3AM.

I've never used a language that didn't catch that type of error. I'm quite
surprised that Python is being used by a number of major companies. How you
catch these types of errors?

In a word (or phrase rather): unit testing. Lots of unit testing,
which catches both silly errors like this and more complicated logic
errors.
Check out the 'unittest' module
[http://docs.python.org/library/unittest.html] and any of the many
writings on the subject for more information. And I'm sure some other
posters will respond with more thorough explanations.

Cheers,
Chris

 

[Lawrence D'Oliveiro]

In module one, I have a function:

def foo( host, userid, password ):
pass

In module two, I call that function:

foo( userid, password)

lint doesn't find that error and it won't be caught until it's called
while the program is running. I don't want that error found at 3AM.

I've never used a language that didn't catch that type of error.

Because in Python, it's not necessarily an error. Every name you define in
Python is just a variable, and can have its value changed at any time: the
compiler can't tell that foo won't have a different value put into it
between the definition of the function and the actual call.

 

[Marc 'BlackJack' Rintsch]

Pylint does a decent job at checking for errors only within a single
module.

Here's one of my problems. I have two modules.

In module one, I have a function:

def foo( host, userid, password ):
pass

In module two, I call that function:

foo( userid, password)

lint doesn't find that error and it won't be caught until it's called
while the program is running. I don't want that error found at 3AM.

Then don't run the unit tests that late at night (or early in the
morning). ;-)

Besides the `unittest` module from the standard library you might look
into `py.test` and `nose`.

Ciao,
Marc 'BlackJack' Rintsch

 

[bearophileHUGS]

Pat:

I know about 3 different lints for Python, there's PyFlake too. But I
don't know if it does what you want.

I've never used a language that didn't catch that type of error.

What dynamic languages have you used in the past?

I'm quite surprised that Python is being used by a number of major
companies. How you catch these types of errors?

Writing tests. You must adapt your coding style to the language you
use.
In Java/C++ your static type system catches those bugs for you, in
dynamic languages you have to catch them with testing (or lints, if
they are present). Languages like Haskell with a type system much
stronger than Java/C++ ones help you catch even more bugs/problems at
compile time, so you need more time to have a clean compilation, but
you have less bugs later (but tests are useful in Haskell too, see
QuickCheck, that has inspired lot of similar tools for other
languages, Python included).

Bye,
bearophile

 

[George Sakkis]

I've been searching for a good multi-module lint checker for Python and
I haven't found one yet.

Pylint does a decent job at checking for errors only within a single module.

Here's one of my problems.  I have two modules.

In module one, I have a function:

def foo( host, userid, password ):
     pass

In module two, I call that function:

foo( userid, password)

lint doesn't find that error and it won't be caught until it's called
while the program is running.  I don't want that error found at 3AM.

I've never used a language that didn't catch that type of error.  I'm
quite surprised that Python is being used by a number of major
companies.  How you catch these types of errors?

With a decent testing suite of course. Even if this specific issue
might be handled more or less by a lint-like tool (which is not
trivial in a dynamic language since the f() used in module2 might not
even be the function you think it is), there are dozens more potential
errors that can't be checked, e.g. changing the order and/or the
expected type of the arguments. There is no substitute of testing, and
while this is true in statically typed languages too, it's even more
critical in dynamic languages.

George

 

[Benjamin]

I've been searching for a good multi-module lint checker for Python and
I haven't found one yet.

Pylint does a decent job at checking for errors only within a single module.

Here's one of my problems.  I have two modules.

In module one, I have a function:

def foo( host, userid, password ):
     pass

In module two, I call that function:

foo( userid, password)

lint doesn't find that error and it won't be caught until it's called
while the program is running.  I don't want that error found at 3AM.

I've never used a language that didn't catch that type of error.  I'm
quite surprised that Python is being used by a number of major
companies.  How you catch these types of errors?

You have to write extensive tests for your code. Not only does this
catch trivial errors like the above, but it finds logic errors in your
implementation.

 

[Miki]

Hello,

In module one, I have a function:

def foo( host, userid, password ):
     pass

In module two, I call that function:

foo( userid, password)

lint doesn't find that error and it won't be caught until it's called
while the program is running.

pychecker does find these kind of errors.

I've never used a language that didn't catch that type of error.  I'm
quite surprised that Python is being used by a number of major
companies.  How you catch these types of errors?

By running a large test suite, I highly recommend "nose".

HTH,

 

[Pat]

Hello,

pychecker does find these kind of errors.

Before posting my original question, I tried pychecker and it didn't
catch that error.

By running a large test suite, I highly recommend "nose".

That's something I'll look into after I'm more proficient in Python.
Currently, I'm learning my way around Python.

Thank you.

 

[Aaron \Castironpi\ Brady]

I've been searching for a good multi-module lint checker for Python and
I haven't found one yet.

Pylint does a decent job at checking for errors only within a single
module.

Here's one of my problems. I have two modules.

In module one, I have a function:

def foo( host, userid, password ):
pass

In module two, I call that function:

foo( userid, password)

lint doesn't find that error and it won't be caught until it's called
while the program is running. I don't want that error found at 3AM.

I've never used a language that didn't catch that type of error. I'm
quite surprised that Python is being used by a number of major
companies. How you catch these types of errors?

I've spoken to tech support at Wing and they weren't aware of a
multi-module lint but they're considering putting their own lint into
their IDE.

Thank you

The example you gave is specific. You could catch that one by hand.
But, would it hurt, perhaps by causing neglect of other errors; and
would it be worth the effort? And, if you're using any dynamics at all,
catching that one's even in question.

from mod import foo
bar= foo
bar( arg1, arg2 )

It would add difficulty to catching by hand. Same with variable
argument lists.

from mod import foo
args= ( arg1, arg2 )
foo( *args )

If you could knock off an error and ignore the false positives, that
might be worth an afternoon.

 

[Aaron \Castironpi\ Brady]

I've been searching for a good multi-module lint checker for Python and
I haven't found one yet.

Pylint does a decent job at checking for errors only within a single
module.

Here's one of my problems. I have two modules.

In module one, I have a function:

def foo( host, userid, password ):
pass

In module two, I call that function:

foo( userid, password)

lint doesn't find that error and it won't be caught until it's called
while the program is running. I don't want that error found at 3AM.

I've never used a language that didn't catch that type of error. I'm
quite surprised that Python is being used by a number of major
companies. How you catch these types of errors?

I've spoken to tech support at Wing and they weren't aware of a
multi-module lint but they're considering putting their own lint into
their IDE.

Thank you

The example you gave is specific. You could catch that one by hand.
But, would it hurt, perhaps by causing neglect of other errors; and
would it be worth the effort? And, if you're using any dynamics at all,
catching that one's even in question.

from mod import foo
bar= foo
bar( arg1, arg2 )

It would add difficulty to catching by hand. Same with variable
argument lists.

from mod import foo
args= ( arg1, arg2 )
foo( *args )

If you could knock off an error and ignore the false positives, that
might be worth an afternoon.

 

[Aaron \Castironpi\ Brady]

I've been searching for a good multi-module lint checker for Python and
I haven't found one yet.

Pylint does a decent job at checking for errors only within a single
module.

Here's one of my problems. I have two modules.

In module one, I have a function:

def foo( host, userid, password ):
pass

In module two, I call that function:

foo( userid, password)

lint doesn't find that error and it won't be caught until it's called
while the program is running. I don't want that error found at 3AM.

I've never used a language that didn't catch that type of error. I'm
quite surprised that Python is being used by a number of major
companies. How you catch these types of errors?

I've spoken to tech support at Wing and they weren't aware of a
multi-module lint but they're considering putting their own lint into
their IDE.

Thank you

The example you gave is specific. You could catch that one by hand.
But, would it hurt, perhaps by causing neglect of other errors; and
would it be worth the effort? And, if you're using any dynamics at all,
catching that one's even in question.

from mod import foo
bar= foo
bar( arg1, arg2 )

It would add difficulty to catching by hand. Same with variable
argument lists.

from mod import foo
args= ( arg1, arg2 )
foo( *args )

If you could knock off an error and ignore the false positives, that
might be worth an afternoon.

 

[Aaron \Castironpi\ Brady]

Before posting my original question, I tried pychecker and it didn't
catch that error.

Sorry for the multiple posting earlier (you heard me right, not 2 but
3 identical). The basic checker of walking a parse tree, doing
imports by hand, and checking call signatures isn't awful. 60 lines.

/File:

from ng23mod1 import foo as foo
userid, password= 'abc', '123'
import random
foo( 'localhost', userid, password)
if random.uniform( 0, 1 )< .01: # 1 out of 100 times
foo( userid, password)

/Checker output:

foo ['Str', 'Name', 'Name'] found 3 expected
random ['Num', 'Num'] found 2 expected
foo ['Name', 'Name'] found 3 expected

But it's extremely delicate and at 60 lines only checks fixed-length
call signatures and functions, not even methods. If you have a
statement:

x= y.z()

Then 'y' has to be defined somewhere, so you could make some educated
guesses that way.

 

[Wolfgang Grafen]

No need to develop another lint tool. Just give the creator of pylint an
improvement proposal. This can be at least reported as a warning.

Even in a highly dynamic language like Python it is good to follow some
style guides. I try to avoid the same names if possible for different
functionality. This makes the code much more manageable and maintainable
and understandable for larger projects.

 

[Bruno Desthuilliers]

Pat a écrit :

I've been searching for a good multi-module lint checker for Python and
I haven't found one yet.

Pylint does a decent job at checking for errors only within a single
module.

Here's one of my problems. I have two modules.

In module one, I have a function:

def foo( host, userid, password ):
pass

In module two, I call that function:

foo( userid, password)

lint doesn't find that error

Nope, but even the most simple manual test should find it pretty quick.

and it won't be caught until it's called
while the program is running. I don't want that error found at 3AM.

Don't you ever test your code ???

I've never used a language that didn't catch that type of error.

It does. Just try to run your code, and you'll have a nice traceback.
Unless of course 'foo' is rebound in module two to another callable
expecting only two parameters...

I'm
quite surprised that Python is being used by a number of major
companies.

Perhaps do they rely more on testing and less on the compiler ? FWIW,
I've seen my share of bugs in declarativly statically typed languages,
and most of them were way nastier (and way less obvious) than the above one.

How you catch these types of errors?

Just like any other type of errors : testing, testing, and then add some
more tests.

 

[Pat]

Pat a écrit :

Nope, but even the most simple manual test should find it pretty quick.


Don't you ever test your code ???


It does. Just try to run your code, and you'll have a nice traceback.
Unless of course 'foo' is rebound in module two to another callable
expecting only two parameters...


Perhaps do they rely more on testing and less on the compiler ? FWIW,
I've seen my share of bugs in declarativly statically typed languages,
and most of them were way nastier (and way less obvious) than the above
one.


Just like any other type of errors : testing, testing, and then add some
more tests.

I haven't gotten into unittesting. I've just started learning Python.

It also dawned on me why my original question is a bit lame. Python
supports default arguments; something that is new to me. How could lint
possibly know the correct number of arguments passed to it? Unless, of
course, lint knew which functions had default arguments or not.

I'll come back with more intelligent questions after I've actually
learned some Python.

 

[George Sakkis]

I'll come back with more intelligent questions after I've actually
learned some Python.

That's a wise self-advice ;-)

 

[Bruno Desthuilliers]

Pat a écrit :

(snip)


I haven't gotten into unittesting. I've just started learning Python.

It also dawned on me why my original question is a bit lame.

I wouldn't qualify it as 'lame' - even if it could have been expressed
on a slightly less controversial tone !-)

Python
supports default arguments; something that is new to me.

Then prepare for quite a lot of new things...

How could lint
possibly know the correct number of arguments passed to it? Unless, of
course, lint knew which functions had default arguments or not.

import inspect
inspect.getargspecs(func)


Python is incredibly strong at introspection - even if that doesn't
solve the kind of "problems" one can possibly have with dynamic typing,
and yes, dynamism (and not only applied to typing) has its own share of
drawbacks, like the one you evoked (almost no compile time checking
except for syntax) and quite a few others. Like any other design
choices, it's a trade-off. You lose a bit here, and possibly (depending
on the kind of application and how your neurons are connected) win quite
a lot there.

From a very pragmatic POV (and Python is mostly a pragmatic language),
Python is pretty good at getting out of the way while you get the job
done - which is perhaps why more and more 'major companies' are using
it. The "downside" is that you lose in (perceived ?) "security" what you
gain in freedom and flexibility. Some like it, some just can't get used
to it, and there's no judgement call involved - it's just a matter of
what better fits your own brain.

wrt/ your question, it's indeed a fact that Python (like any other
dynamic language FWIW) can't catch this class of errors at compile time.
So the question 'how do I avoid this kind of error' is mostly sensible.

OTHO, not even Ada would catch the following :
foo(userid, password, host)
at compile time - unless you define specific distinct types for each
argument, which might be a bit overkill (well, IMHO at least - YMMV !-).

IOW : you *do* have to test, whatever the language and type system.

Practically and most of the time, given Python's very quick edit/run
cycle, you don't even have to write unit tests to catch this kind of
silly errors - my experience at least. But having unit tests is still
the best way to ensure you didn't break anything with your last edit.
While unit testing is not a panacea neither (hint: there's no silver
bullet), it's really an invaluable tool.

I'll come back with more intelligent questions after I've actually
learned some Python.

You can even ask "not that intelligent" questions while learning - in
the worst case, you'll have no anwer at all, but most probably you'll at
least have a short answer with a link to the relevant documentation.

Welcome onboard !-)