threads and memory

[Lee Leahu]

Hi List,

I am trying to write a simple threaded application which will simulate 1000 connections to a remote service in order to "stress test" that the remote service can handle that many connections.

However, I have encountered the following error after I have started my 381st thread:

---------------------------------------------------------------------------
Traceback (most recent call last):
File "./test.py", line 39, in ?
current.start()
File "/usr/lib/python2.4/threading.py", line 416, in start
_start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread
---------------------------------------------------------------------------

The command 'ps -eo pid,%cpu,%mem,rss,sz,vsz' gives at the point of error, the following information:
---------------------
pid: 5448
%cpu: 33.5
%mem: 0.6
rss: 9492
sz: 782729
vsz: 3130916
---------------------

I have 1.5GB of total ram in my computer, as evident by the command 'free':
---------------------
total used free shared buffers cached
Mem: 1553128 1314152 238976 0 0 357392
-/+ buffers/cache: 956760 596368
Swap: 977248 7964 969284
---------------------

An 'strace' of the script reveals the following:
NOTE the line that reads 'ENOMEM (Cannot allocate memory).
--------------------------------------------------------
futex(0x80b2b08, FUTEX_WAKE, 1) = 1
select(0, NULL, NULL, NULL, {0, 0}) = 0 (Timeout)
write(1, "381\n", 4381
) = 4
futex(0x80b2b08, FUTEX_WAKE, 1) = 0
futex(0x80b2b08, FUTEX_WAKE, 1) = 0
futex(0x80581f8, FUTEX_WAKE, 1) = 0
mmap2(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
write(2, "Traceback (most recent call last"..., 35Traceback (most recent call last):
) = 35
open("./test.py", O_RDONLY|O_LARGEFILE) = 383
write(2, " File \"./test.py\", line 40, in "..., 34 File "./test.py", line 40, in ?
) = 34
fstat64(383, {st_mode=S_IFREG|0755, st_size=1037, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f65000
read(383, "#!/usr/bin/python\n\nimport time\nf"..., 4096) = 1037
write(2, " ", 4 ) = 4
write(2, "current.start()\n", 16current.start()
--------------------------------------------------------

Is there a way to accomplish what I am trying to do, perhaps in a more memory-friendly way?

My source code is attached.

Please advise.

Thanks!

--
Lee Leahu RICIS, Inc.
Internet Technology Specialist 866-RICIS-77 Toll Free Voice (US)
(e-mail address removed) 708-444-2690 Voice (International)
http://www.ricis.com/ 866-99-RICIS Toll Free Fax (US)
708-444-2697 Fax (International)

RICIS, Inc. is a member of the Public Safety Alliance Group

This email and any attachments that are included in it have been scanned
for malicious or inappropriate content and are believed to be safe.

 

[Rene Pijlman]

Lee Leahu:

I am trying to write a simple threaded application which will simulate 1000
connections to a remote service in order to "stress test" that the remote
service can handle that many connections. [...]
Is there a way to accomplish what I am trying to do, perhaps in a more
memory-friendly way?

Yes. You don't need one thread per connection:
http://docs.python.org/lib/module-asyncore.html

 

[Lee Leahu]

Hi List,

I am trying to write a simple threaded application which will simulate 1000
connections to a remote service in order to "stress test" that the remote
service can handle that many connections. [...]
Is there a way to accomplish what I am trying to do, perhaps in a more
memory-friendly way?

Yes. You don't need one thread per connection:
http://docs.python.org/lib/module-asyncore.html

I am running into a problem where asyncore is through a filedescriptor error if I try to launch more that 1023 connections:
------------------------------------
Traceback (most recent call last):
File "./test.py", line 46, in ?
asyncore.loop()
File "/usr/lib/python2.4/asyncore.py", line 192, in loop
poll_fun(timeout, map)
File "/usr/lib/python2.4/asyncore.py", line 122, in poll
r, w, e = select.select(r, w, e, timeout)
ValueError: filedescriptor out of range in select()
------------------------------------

Is there a limitation on the number of simultaneous connects that can be made with asyncore ?

Attached is the updated source code.

Thanks!

--
Lee Leahu RICIS, Inc.
Internet Technology Specialist 866-RICIS-77 Toll Free Voice (US)
(e-mail address removed) 708-444-2690 Voice (International)
http://www.ricis.com/ 866-99-RICIS Toll Free Fax (US)
708-444-2697 Fax (International)

RICIS, Inc. is a member of the Public Safety Alliance Group

This email and any attachments that are included in it have been scanned
for malicious or inappropriate content and are believed to be safe.

 

[Bryan Olson]

I am trying to write a simple threaded application which will
> simulate 1000 connections to a remote service in order to
> stress test" that the remote service can handle that many
> connections.

That shouldn't be a problem on a modern OS, but there are
still quite a few not-up-to-speed systems in the field that
choke on just a few hundred threads. There are also many
systems with modern threading that choke because they are
poorly configured by default.

However, I have encountered the following error after I
> have started my 381st thread:
[...]
I have 1.5GB of total ram in my computer, as evident by the
> command 'free': [...]
NOTE the line that reads 'ENOMEM (Cannot allocate memory).

The error message may be misleading. The limiting factor on
the number of threads is usually address space, not physical
memory. You can get ENOMEM when the system cannon allocate
stack space for a thread. The space it needs is not RAM,
nor even RAM+swap; it's pure virtual address space.

Look up how to set the stack-space for threads on your
system. A reasonable size on a 32-bit system is 2MB to
4MB. Then you should be able to create threads numbering
in the low thousands. (On 64-bit systems, the issue just
goes away.)

 

[Fredrik Lundh]

I am running into a problem where asyncore is through a filedescriptor
error if I try to launch more that 1023 connections:
------------------------------------
Traceback (most recent call last):
File "./test.py", line 46, in ?
asyncore.loop()
File "/usr/lib/python2.4/asyncore.py", line 192, in loop
poll_fun(timeout, map)
File "/usr/lib/python2.4/asyncore.py", line 122, in poll
r, w, e = select.select(r, w, e, timeout)
ValueError: filedescriptor out of range in select()

what does "ulimit -n" (max number of file descriptors) say on your machine?

if it says 1024, that's what you're seeing. to increase this, you
have to reconfigure your OS.

if you get a larger value, you can try passing use_poll=True to the
asyncore.loop function.

if use_poll doesn't work, you may have to rebuild Python's select
module to support more handles. see the comments at the top of
Modules/selectmodule.c for details.

</F>

 

[Lee Leahu]

Hi List,

what does "ulimit -n" (max number of file descriptors) say on your machine?

As a normal user, I was limited to 1024. But su'ing to root allows me to set this higher, like 8196.

This error occurs after setting it higher (which resolves the socket.err(24, "too many open files") problem).

if you get a larger value, you can try passing use_poll=True to the
asyncore.loop function.

I will give this a try.

if use_poll doesn't work, you may have to rebuild Python's select
module to support more handles. see the comments at the top of
Modules/selectmodule.c for details.

Thanks!

--
Lee Leahu RICIS, Inc.
Internet Technology Specialist 866-RICIS-77 Toll Free Voice (US)
(e-mail address removed) 708-444-2690 Voice (International)
http://www.ricis.com/ 866-99-RICIS Toll Free Fax (US)
708-444-2697 Fax (International)

RICIS, Inc. is a member of the Public Safety Alliance Group

This email and any attachments that are included in it have been scanned
for malicious or inappropriate content and are believed to be safe.

 

[Ivan Voras]

However, I have encountered the following error after I have started my 381st thread:

This number (actually, ~380) is suspicious when seen with threads
because it almost always means running out of process address space. As
threads are part of a single process, and that process has (usually
today) 32bit address space, and modern Linux systems allocate huge
amounts of memory per-thread (8 MB for threads' stack!), and the kernel
reserves ~1MB in the address space for itself, you can easily reach >
4GB of total allocated memory.

This *doesn't* mean you are actually using 4GB of memory (i.e. storing
values to it), only that the process gets "assigned" this memory. Python
doesn't support specifying thread attributes AFAIK, so you should find a
way to specify default thread stack size, possibly with an environment
variable (see manual page for "pthread" library calls). Try 64k - your
task seems simple enough