Multiprocessing bug, is information ever omitted from a traceback?

[John Ladasky]

Hi folks,

A tangent off of this thread:

http://groups.google.com/group/comp.lang.python/browse_frm/thread/751b7050c756c995#

I'm programming in Python 2.6 on Ubuntu Linux 10.10, if it matters.

I'm trying to track down a multiprocessing bug. Here's my traceback.
All lines of code referenced in the traceback are in the standard
library code:

Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 532, in
__bootstrap_inner
self.run()
File "/usr/lib/python2.6/threading.py", line 484, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.6/multiprocessing/pool.py", line 284, in
_handle_tasks
put(task)
TypeError: expected string or Unicode object, NoneType found

Fortunately, I have a working version of my code. I was trying to add
new features, and only my new code is causing trouble. This has
allowed me to examine the contexts of task when everything works.

Task is not a string when the program works. Task is not None when
the program doesn't work. In fact, task is a deeply-nested tuple. NO
PART of this tuple ever contains any strings, as far as I can tell.
More details in my original thread.

Now, of course I've seen that the standard traceback shows you the
lines where various steps in a chain of function calls were taken.
The traceback skips over any lines in the code between successive
function calls, and assumes that you can follow along. No problem, I
can do that.

But when multiprocessing is involved, can this traceback be truncated
in some way, for example when code execution switches over to a
subprocess? I'm wondering if more code is getting executed after
"put(task)" that I'm not seeing.

Thanks for any information!

 

[Terry Reedy]

http://groups.google.com/group/comp.lang.python/browse_frm/thread/751b7050c756c995#

I'm programming in Python 2.6 on Ubuntu Linux 10.10, if it matters.

It might, as many bugs have been fixed since.
Can you try the same code with the most recent 2.x release, 2.7.2?
Do you have working and non-working code that you can publicly release?
Can you reduce the size and dependencies so the examples are closer to
'small' than 'large'? And in any case, self-contained?

In my first response, I said you might have found a bug. A bogus
exception message qualifies. But to do much, we need minimal good/bad
examples that run or not on a current release (2.7.2 or 3.2.2).

 

[John Ladasky]

It might, as many bugs have been fixed since.
Can you try the same code with the most recent 2.x release, 2.7.2?
Do you have working and non-working code that you can publicly release?
Can you reduce the size and dependencies so the examples are closer to
'small' than 'large'? And in any case, self-contained?

In my first response, I said you might have found a bug. A bogus
exception message qualifies. But to do much, we need minimal good/bad
examples that run or not on a current release (2.7.2 or 3.2.2).

All right, Terry, you've convinced me to look. This will take some
time. I'll hack away at the two versions of my 500-line programs,
until I have a minimal example.

Why did you specify Python 2.7.2, instead of the 2.7.6 version that is
being offered to me by Ubuntu Software Center? Does it matter?

 

[Andrew Berg]

Why did you specify Python 2.7.2, instead of the 2.7.6 version that is
being offered to me by Ubuntu Software Center? Does it matter?

There is no Python 2.7.6. I think you have it confused with the version
2.7.2-6. If I'm not mistaken, that appended 6 has to do with packaging
and nothing at all to do with the software itself.

 

[John Ladasky]

On 12/10/2011 11:53 AM, John Ladasky wrote:> Why did you specify Python 2..7.2, instead of the 2.7.6 version that is

There is no Python 2.7.6. I think you have it confused with the version
2.7.2-6. If I'm not mistaken, that appended 6 has to do with packaging
and nothing at all to do with the software itself.

How annoying, here's how Ubuntu Software Center described it:

"Version: 2.7-6 (python2.7)"

And here it is in the Synaptic Package Manager:

"package: python2.7-minimal, installed version: 2.7-6"

At first I considered that this truncated name might be the
consequence of a Linux bug. But looking up and down my list of
installed software in both Ubuntu Software Center and Synaptic, I can
find version names which extend arbitrarily, such as my wxPython
installation, for which the version number reads
"2.8.11.0-0ubuntu4.1".

Now my hypothesis is that someone manually enters the revision numbers
into the Linux database, and they made a typo.

 

[Chris Angelico]

There is no Python 2.7.6. I think you have it confused with the version
2.7.2-6. If I'm not mistaken, that appended 6 has to do with packaging
and nothing at all to do with the software itself.

I'm programming in Python 2.6 on Ubuntu Linux 10.10, if it matters.

As with my Ubuntu 10.10 - it's version 2.6.6. (I also altinstalled 3.3
built straight from Mercurial, which is approximately as new a Python
as one can have, but that doesn't count.)

There's a few differences between 2.6 and 2.7; not usually enough to
be concerned about in daily use, but when dealing with weird issues,
it helps to have the latest release.

ChrisA

 

[Terry Reedy]

There's a few differences between 2.6 and 2.7; not usually enough to
be concerned about in daily use, but when dealing with weird issues,
it helps to have the latest release.

There are 2 issues. First, 2.7.2 has perhaps a couple hundred bug fixes
since 2.7.0/2.6.6 were released. So it is possible that this particular
problem was fixed, or at least changed. Second (leaving security issues
aside), a Py2 bug fix will only be applied to the post-2.7.2 tip and
only after the bug has been demonstrated to exist in the post-2.7.2 tip.

About the request for minimal code exhibiting the problem: each bug
exposes a hole in the test suite. We try to add a new test case with
each bug fix so fixed bugs stay fixed. So code examples not only show
that the bug is in the latest version, but also serve as the basis for
new tests.

 

[John Ladasky]

I hope that the interested parties who offered me help will remember
this thread title. It has been a few weeks.

After fussing around with an alternate installation of Python 2.7.2 on
top of Ubuntu Linux 10.04, I decided it would be easier to upgrade to
Ubuntu 11.10, for which Python 2.7.2 is the default Python version. I
have reproduced my Python 2.6.6 bug in Python 2.7.2, as you will see.

All lines of code referenced in the traceback are in the standard
library code:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in
__bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 284, in
_handle_tasks
    put(task)
TypeError: expected string or Unicode object, NoneType found

Fortunately, I have a working version of my code.  I was trying to add
new features, and only my new code is causing trouble.  This has
allowed me to examine the contexts of task when everything works.

Task is not a string when the program works.  Task is not None when
the program doesn't work.  In fact, task is a deeply-nested tuple.  NO
PART of this tuple ever contains any strings, as far as I can tell.
More details in my original thread.

Here's the traceback that I obtained tonight. It's essentially the
same, except for the line numbers in the library code and the thread
number which yielded the exception:

| Exception in thread Thread-2:
| Traceback (most recent call last):
| File "/usr/lib/python2.7/threading.py", line 552, in
__bootstrap_inner
| self.run()
| File "/usr/lib/python2.7/threading.py", line 505, in run
| self.__target(*self.__args, **self.__kwargs)
| File "/usr/lib/python2.7/multiprocessing/pool.py", line 315, in
_handle_tasks
| put(task)
| TypeError: expected string or Unicode object, NoneType found


I still have more work to do, I know. I will try to pare down my
fairly-complex code to a minimal example. Check back again in a few
days? Weeks? Thanks!