Threaded alternatives to smtplib?

[Alex Jurkiewicz]

Hi all,
I'm writing a Python script to do a "mail merge" style email
distribution. I create a few python threads and in each one I call
`smtpserver = smtplib.SMTP(our.smtpserver.com)`. However, during the
sending process, there seems to be only one connection open to our mail
server at any one time. In other words, all these threads gain me no
speed benefit at all!

I seem to be running into the issue where smtplib is not fully
thread-safe, as mentioned in this thread:
http://mail.python.org/pipermail/python-list/2007-March/429067.html
http://mail.python.org/pipermail/python-list/2007-March/429172.html

Does anyone have suggestions as to a suitable workaround for this issue?
I was looking at the Twisted library, which seems possible but
significantly more complex.


Thanks,
Alex Jurkiewicz

 

[Diez B. Roggisch]

Hi all,
I'm writing a Python script to do a "mail merge" style email
distribution. I create a few python threads and in each one I call
`smtpserver = smtplib.SMTP(our.smtpserver.com)`. However, during the
sending process, there seems to be only one connection open to our mail
server at any one time. In other words, all these threads gain me no
speed benefit at all!

I seem to be running into the issue where smtplib is not fully
thread-safe, as mentioned in this thread:
http://mail.python.org/pipermail/python-list/2007-March/429067.html
http://mail.python.org/pipermail/python-list/2007-March/429172.html

Does anyone have suggestions as to a suitable workaround for this issue?
I was looking at the Twisted library, which seems possible but
significantly more complex.

I doubt that there is such issue. The above discussion is not concluding
that there actually *are* threading-issues within smtplib. And looking
at the source of smtplib, it certainly doesn't use any
locking-primitives or (directly) any c-extension that might explain such
behavior.

Without more code, it's impossible to tell if there is anything peculiar
in your usage of the lib. Maybe you close your connections to fast to
see several open?

Diez

 

[Alex Jurkiewicz]

Without more code, it's impossible to tell if there is anything
peculiar in your usage of the lib. Maybe you close your connections to
fast to see several open?

Here's the relevant stuff from my (python2.6) code:


CONCURRENCY = 3

def threadProcessRecipient():
# Each thread has its own SMTP connection
smtpserver = smtplib.SMTP(SMTPSERVER)

# Each thread pulls a recipient entry from the queue to process and
loops until the queue is empty.
try:
recipientData = recipientQueue.get_nowait()
except Queue.Empty:
recipientData = None

while recipientData:
message = prepareMessage()
sendMail(senderEmail, recipientEmail, message, smtpserver)

recipientQueue.task_done()
try:
recipientData = recipientQueue.get_nowait()
except Queue.Empty:
recipientData = None

smtpserver.quit()

if __name__ == '__main__':
THREADS = []
for i in range(CONCURRENCY):
THREADS.append(threading.Thread(target=threadProcessRecipient))
for thread in THREADS:
thread.run()
for thread in THREADS:
thread.join()

 

[Gabriel Genellina]

En Mon, 04 May 2009 04:19:21 -0300, Alex Jurkiewicz

Here's the relevant stuff from my (python2.6) code:


CONCURRENCY = 3

def threadProcessRecipient():
# Each thread has its own SMTP connection
smtpserver = smtplib.SMTP(SMTPSERVER)
# Each thread pulls a recipient entry from the queue to process
and loops until the queue is empty.
try:
recipientData = recipientQueue.get_nowait()
except Queue.Empty:
recipientData = None
while recipientData:
message = prepareMessage()
sendMail(senderEmail, recipientEmail, message, smtpserver)
recipientQueue.task_done()
try:
recipientData = recipientQueue.get_nowait()
except Queue.Empty:
recipientData = None
smtpserver.quit()

Try logging the start/stop of your threads. It may be that your threads
stop before you think. The above code works correctly only if you fill the
queue before starting any thread - because as soon as a thread sees the
queue empty, it finishes.
You could use the sample code in the Queue documentation at the end of
http://docs.python.org/library/queue.html

 

[Alex Jurkiewicz]

Try logging the start/stop of your threads. It may be that your
threads stop before you think. The above code works correctly only if
you fill the queue before starting any thread - because as soon as a
thread sees the queue empty, it finishes.
You could use the sample code in the Queue documentation at the end of
http://docs.python.org/library/queue.htm

The queue is actually filled before the threads are started, data is
only removed after they start.

Logging thread state (at least in debug mode) would be good though. How
would I go about it in this example? The docs suggest using thread.name,
but in my current setup I don't have any idea which thread is running.
Something like this:
for i in range(CONCURRENCY):
THREADS.append( threading.Thread(target=threadProcessRecipient,
args=i, name=i) )

And:

def threadProcessRecipient(name):
print "Name is %s" % name


AJ

 

[Diez B. Roggisch]

The queue is actually filled before the threads are started, data is
only removed after they start.

Logging thread state (at least in debug mode) would be good though. How
would I go about it in this example? The docs suggest using thread.name,
but in my current setup I don't have any idea which thread is running.
Something like this:
for i in range(CONCURRENCY):
THREADS.append( threading.Thread(target=threadProcessRecipient,
args=i, name=i) )

And:

def threadProcessRecipient(name):
print "Name is %s" % name

Each thread gets an unique name assigned anyway. Alternatively, you can use
partial to bind parameters to functions. And last but not least to subclass
Thread is also an option.

Diez

 

[Diez B. Roggisch]

Oh, and I forgot - threading.currentThread() gets you the actively running
thread of course.

Dize

 

[MRAB]

I doubt that there is such issue. The above discussion is not concluding
that there actually *are* threading-issues within smtplib. And looking
at the source of smtplib, it certainly doesn't use any
locking-primitives or (directly) any c-extension that might explain such
behavior.

Without more code, it's impossible to tell if there is anything peculiar
in your usage of the lib. Maybe you close your connections to fast to
see several open?

If all the threads are using the same server, couldn't the SMTP part be
put into its own thread and then fed the pre-composed emails via a
queue?

 

[Piet van Oostrum]

AJ> def threadProcessRecipient(): [snip]
AJ> if __name__ == '__main__':
AJ> THREADS = []
AJ> for i in range(CONCURRENCY):
AJ> THREADS.append(threading.Thread(target=threadProcessRecipient))
AJ> for thread in THREADS:
AJ> thread.run()

You should use thread.start(), not thread.run(). When you use run(), it
will be sequential execution, as you experience. With start() you get
concurrency