Urllib2/threading errors under Cygwin

[Jacek Trzmiel]

Hi,

I have a problem with using urllib2 with threading module under Cygwin.

$ cygcheck -cd cygwin python
Cygwin Package Information
Package Version
cygwin 1.5.5-1
python 2.3.2-1

Here is minimal app where I can reproduce errors:

--- MtUrllib2Test.py -------------------------------------------------
#!/usr/bin/env python

import urllib2
import threading
import sys
import time


def FetchPage():
# time.sleep(3)
# return
opener = urllib2.build_opener()
urlFile = opener.open( 'http://google.com/' )
pageData = urlFile.read()


def IncCounterAndPrint( count=[0] ):
count[0] += 1
print count[0]
# sys.stdout.flush()


def Main():
noOfThreads = 1
for unused in range(noOfThreads):
thread = threading.Thread( target=FetchPage, args=() )
thread.start()
# time.sleep(0.2)
IncCounterAndPrint()

while(threading.activeCount()>1):
IncCounterAndPrint()
time.sleep(0.5)
IncCounterAndPrint()


if __name__ == "__main__":
Main()
--- MtUrllib2Test.py -------------------------------------------------

0. Simple case. Here everything looks ok:

$ python MtUrllib2Test.py
1
2
3
4
5


1. First error.

$ python MtUrllib2Test.py | tee out.txt
3
4
5

Leading prints has been eaten somewhere. Uncommenting disabled code in
ANY of the functions does make output correct, but none of the solutions
looks good for me:

a) IncCounterAndPrint() - sys.stdout.flush()
As I understand if stdout is not console then output gets buffered (i.e.
it's not flushed automatically). Adding a flush call does make output
good, but this looks like a kludge for me, not a real fix. I am writing
to stdout from only one thread, so everything should be fine without
calling flush, shouldn't it?

b) Main() - time.sleep(0.2)
Adding a little sleep after starting thread does make output correct
too. For me it looks like race condition either in urllib2 or in
cygwin. Or am I completely off here?

c) FetchPage() - time.sleep(3), return
Disabling calls to urllib2 does make problem go away, too.


2. Second error.

If I increase number of threads:
noOfThreads = 20
and run this prog (you may need to run it several times, or rise number
of threads more to reproduce), then sometimes it does fail this way :

$ python MtUrllib2Test.py | tee out.txt
4 [win] python 1744 Winmain: Cannot register window class
C:\cygwin\bin\python2.3.exe: *** WFSO failed, Win32 error 6

or hangs this way:

$ python MtUrllib2Test.py | tee out.txt
243 [win] python 1696 Winmain: Cannot register window class
520 [win] python 1696 Winmain: Cannot register window class



Can anyone help me on those two?

Best regards,
Jacek.