Socket Speed

[mvanaswegen]

Hi All

I'm busy writing a python p2p program and would like some advice.

I'm pushing about 300k/s and would like to know if there are any python
tricks I could pull to speed things up. I'm thinking about unrolling
some of the loops and recuding calls to my custom socket class and
just calling recv and send directly. What is the overhead for using
try and exception blocks ? any recomendations for recv/send sizes ?
On the server side I'm using SocketServer. I was planning to use
twisted
but I don't have the time just get to get into it, SocketServer does
the job.

I'll post some code soon.

Regs

M

 

[Jeff Epler]

300KB/s sounds dreadfully low.

I simply ran "python /usr/lib/python2.3/SimpleHTTPServer.py &", then
"wget -O /dev/null http://0.0.0.0:8000/70megfile". On the best of 4
runs (when the file was cached) wget measured 225.20MB/s.

The hardware is a Pentium-M laptop with 768MB RAM runnng at 1.5GHz. The
OS is Fedora Core 2, kernel 2.6.12-rc5, Python 2.3.

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFCowVQJd01MZaTXX0RAk4BAKCN3fWTARRvrlvXngABtim+41syKQCdFeTY
I7JuLh6qr1mtQv73Jf17La8=
=0/P4
-----END PGP SIGNATURE-----

 

[Dave Brueck]

Hi All

I'm busy writing a python p2p program and would like some advice.

I'm pushing about 300k/s and would like to know if there are any python
tricks I could pull to speed things up. I'm thinking about unrolling
some of the loops and recuding calls to my custom socket class and
just calling recv and send directly. What is the overhead for using
try and exception blocks ? any recomendations for recv/send sizes ?
On the server side I'm using SocketServer. I was planning to use
twisted
but I don't have the time just get to get into it, SocketServer does
the job.

I'll post some code soon.

Yeah - do post some code as 300k/s is probably *way* below the potential. In one
project I'm working on, I do simple file transfers over a single socket with
Python on both ends and I routinely get above 500Mbps (often into the high
700's/low 800's - pretty close to what I could hope for for gig over copper with
TCP/IP) and the CPU utilization is way low - I assume disk on one end or the
other of the connection is a bottleneck now.

I use 64k send/receive buffers - above that it doesn't seem to help much for
what I'm doing.

In any case, before you make any local optimizations (loop unrolling, etc.), I'd
do some more investigation to find the problem because something major is
broken, and fixing it will probably do way more to improve performance than any
optimization "tricks".

-Dave

 

[marinus van aswegen]

Hi Jeff

Would you care to do that between two machines on a 100mb link ?

M

 

[marinus van aswegen]

Would you care to do that between two machines on a 100mb link ?

 

[Jeff Epler]

The machines with the 100mbps ethernet link are slightly
different---Pentium 4, 2.8GHz, Python 2.2, RedHat 9.

File size: 87490278

Best of 4 runs: 7.50 MB/s reported by "wget".

There was other network activity and system load at the time.

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFCpDtEJd01MZaTXX0RAtXyAJwLxN6ZwXmc3akcwJve0fIpKxND3QCgh7K+
0eRIxK4b13PvWEjgPzHbQmU=
=ImKe
-----END PGP SIGNATURE-----