Establishing a p2p connection in python

[gs]

Hi!

This is my first time posting to a newsgroup so please be gentle to me


Introduction to my problem:

I'm studying at the university and a lot of friends and I use MSN to
communicate. The problem is that MSN file sending/receiving
capabilities are terribly slow and has no resume option. We're also on
different subnets and can not access eachoters computers through smb
shares, nor can we have local ftp-servers.

My programming background:

I've been taking a few classes in JAVA, enough to get familiar with
sockets, threads and the most common sorting algortithms and
datastructures (binary trees, stacks etc). I'm also very comfortable
programming PHP on a webdeveloper basis.
I recently started out with Python and i immediatly fell in love with
it, and now I want to "master" it! Alas, my problem is also of
educational nature.

Setup:
We're about 30 friends who want the ability to share files.
We're spread out on about 3 different translated external IPs.
I have access to a public IP that has the ability to run python (but
not enough bandwith nor space to host an FTP).
I have limited programming experience.

Proposed solution:
I thought that one approach could be to write a small server for my
public server with the sole purpose of keeping track of connected
users and establishing p2p connections on demand.

Problems:
1) I've written a threaded server that stores information about
connected servers in a "container class". This class holds information
about username, a unique ID, the connecting adress and the actual
socket. I have not been able to find any information online on how i
could go about to connect two sockets. It would seem like the socket
is already "occupied" since it is connected to the server. Could one
duplicate, or otherwise get a dedicated socket for sending binary
data, to one connected socket from another? A few pointers and/or tips
would be greatly appreciated!

2) I've also failed to find python specific information on how abouts
one would do file resuming. I would think that you, somehow, use
md5-checksums to check file status and somehow skip the first part of
the datastream. Also, pointers and maybe an explanation in (short)
pseudo-code would be much appreciated.

3) Is this too big of a project for a novice programmer such as
myself? I do want to go through with this since it's a genuine
problem, but i do not know if i'm ready. I'm not a "quitter" per say,
but i'm afraid doing something TOO advanced in the beginning might be
overwhelming

Thanks for reading,
gs

 

[Alex Martelli]

We're about 30 friends who want the ability to share files.

You could look into BitTorrent, a popular p2p program which, I believe,
is implemented in Python. Maybe, if not usable as-is, it might be made
to fit your needs with some minor modifications...?


Alex

 

[Jeremy Jones]

Hi!

This is my first time posting to a newsgroup so please be gentle to me

Welcome. You have little to fear from this crowd.

Introduction to my problem:

<snip>
I think that your problem can be solved with Twisted. That is _a_
solution. Not the only, maybe not the best for everything you're
wanting to do (which it seems that learning a lot is important to you),
but _a_ solution. It's got a bit of a learning curve to wrap your head
around how everything fits together, but it's a powerful framework.
I've used it less than I plan to, but I've got some interest in how it
works.

Problems:
1) I've written a threaded server that stores information about
connected servers in a "container class". This class holds information
about username, a unique ID, the connecting adress and the actual
socket. I have not been able to find any information online on how i
could go about to connect two sockets. It would seem like the socket
is already "occupied" since it is connected to the server. Could one
duplicate, or otherwise get a dedicated socket for sending binary
data, to one connected socket from another? A few pointers and/or tips
would be greatly appreciated!

Are you talking about your server binding two different ports? If you
are using the Python standard SocketServer, you should be able to just
spin off a thread for each server you start and tell it to
serve_forever() (or what have you). If you're using Twisted, I think
all you would need to do is factory.listenTCP(<port number>, <factory>),
then reactor.run(). I'll leave getting data between the different ports
you've bound as an exercise for the reader.

2) I've also failed to find python specific information on how abouts
one would do file resuming. I would think that you, somehow, use
md5-checksums to check file status and somehow skip the first part of
the datastream. Also, pointers and maybe an explanation in (short)
pseudo-code would be much appreciated.

Can't help you here, sorry.

3) Is this too big of a project for a novice programmer such as
myself? I do want to go through with this since it's a genuine
problem, but i do not know if i'm ready. I'm not a "quitter" per say,
but i'm afraid doing something TOO advanced in the beginning might be
overwhelming

I don't think this is too big at all. But I do think you'll want to
break it down a little bit and set smaller goals for yourself. Try to
first get the smallest fairly usable thing working. Then you can start
adding functionality from there.

Thanks for reading,
gs

HTH,


Jeremy Jones

 

[Darren Kirby]

quoth the gs:

Hi!

This is my first time posting to a newsgroup so please be gentle to me



Introduction to my problem:

I'm studying at the university and a lot of friends and I use MSN to
communicate. The problem is that MSN file sending/receiving
capabilities are terribly slow and has no resume option. We're also on
different subnets and can not access eachoters computers through smb
shares, nor can we have local ftp-servers.

My programming background:

I've been taking a few classes in JAVA, enough to get familiar with
sockets, threads and the most common sorting algortithms and
datastructures (binary trees, stacks etc). I'm also very comfortable
programming PHP on a webdeveloper basis.
I recently started out with Python and i immediatly fell in love with
it, and now I want to "master" it! Alas, my problem is also of
educational nature.

Setup:
We're about 30 friends who want the ability to share files.
We're spread out on about 3 different translated external IPs.
I have access to a public IP that has the ability to run python (but
not enough bandwith nor space to host an FTP).
I have limited programming experience.

Proposed solution:
I thought that one approach could be to write a small server for my
public server with the sole purpose of keeping track of connected
users and establishing p2p connections on demand.

Problems:
1) I've written a threaded server that stores information about
connected servers in a "container class". This class holds information
about username, a unique ID, the connecting adress and the actual
socket. I have not been able to find any information online on how i
could go about to connect two sockets. It would seem like the socket
is already "occupied" since it is connected to the server. Could one
duplicate, or otherwise get a dedicated socket for sending binary
data, to one connected socket from another? A few pointers and/or tips
would be greatly appreciated!

2) I've also failed to find python specific information on how abouts
one would do file resuming. I would think that you, somehow, use
md5-checksums to check file status and somehow skip the first part of
the datastream. Also, pointers and maybe an explanation in (short)
pseudo-code would be much appreciated.

3) Is this too big of a project for a novice programmer such as
myself? I do want to go through with this since it's a genuine
problem, but i do not know if i'm ready. I'm not a "quitter" per say,
but i'm afraid doing something TOO advanced in the beginning might be
overwhelming

Thanks for reading,
gs

I can suggest taking a look at nicotine, which is a full fledged p2p app
written in python. I'm not saying it's your solution, but you may get some
ideas about how to implement what you want by reading the source code, as it
does support file resuming.

-d

--
Part of the problem since 1976
http://badcomputer.no-ip.com
Get my public key from
http://keyserver.linux.it/pks/lookup?op=index&search=bulliver
"...the number of UNIX installations has grown to 10, with more expected..."
- Dennis Ritchie and Ken Thompson, June 1972

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQBBZrDxOrzWcOwL7mwRAu9zAJ996iG8jAuNzdnVYCFTlc6TrAc/GgCffScs
FZK6DINDLa4YmWAHrc2OIo4=
=tR8D
-----END PGP SIGNATURE-----

 

[Ville Vainio]

gs> Proposed solution: I thought that one approach could be to
gs> write a small server for my public server with the sole
gs> purpose of keeping track of connected users and establishing
gs> p2p connections on demand.

One lightweight approach (assuming that a web server is available)
would be to put up a cgi script that adds the iport information to
onlineusers.txt. Then the clients can just use urllib to notify the
script when they become online, and also get the list of connected
clients by just retrieving onlineusers.txt. Of course onlineusers.txt
could contain the lists of shared files for each client as well...

If you use unique user ids, you can keep the file from growing up too
much. If not, you could clean up dead clients by trying to connect all
the specified addresses.

gs> 1) I've written a threaded server that stores information
gs> about connected servers in a "container class". This class
gs> holds information about username, a unique ID, the connecting
gs> adress and the actual socket. I have not been able to find any
gs> information online on how i could go about to connect two
gs> sockets. It would seem like the socket is already "occupied"
gs> since it is connected to the server. Could one duplicate, or
gs> otherwise get a dedicated socket for sending binary data, to
gs> one connected socket from another? A few pointers and/or tips
gs> would be greatly appreciated!

If you went on to "connect" the two sockets, you would just
instantiate a thread that forwards the data, and would use more server
bandwidth than an FTP server would have used.

The idea in p2p is to make clients connect to each other. The task of
the server is to tell the clients about other clients so they can
connect to each other.


gs> 2) I've also failed to find python specific information on how
gs> abouts one would do file resuming. I would think that you,
gs> somehow, use md5-checksums to check file status and somehow
gs> skip the first part of the datastream. Also, pointers and
gs> maybe an explanation in (short) pseudo-code would be much
gs> appreciated.

It's trivial since you are going to implement a custom protocol
anyway. Make the "get file" command that you pass to the file provider
such that in addition to the file name you pass the starting offset
within the file.


gs> 3) Is this too big of a project for a novice programmer such
gs> as myself? I do want to go through with this since it's a
gs> genuine problem, but i do not know if i'm ready. I'm not a
gs> "quitter" per say, but i'm afraid doing something TOO advanced
gs> in the beginning might be overwhelming

No, it's a great first project. Go ahead, I believe you will find it
surprisingly easy, educational and something for which Python is a
perfect fit.

It'll be a fun project to expand as well, to provide chunked file
transfer etc. You could also explore the alternative of making it a
bunch of wrapper scripts that just invoke bittorrent...

 

[gs]

You could look into BitTorrent, a popular p2p program which, I believe,
is implemented in Python. Maybe, if not usable as-is, it might be made
to fit your needs with some minor modifications...?


Alex

Ok, that sounds very interesting, Alex. BitTorrent seems to be able to
handle my needs at the moment, althoug i'd eventually want to write
something myself Thanks for the pointer!

Off to read up on this and try to persuade my isp to let me add
..torrent apptypes

sincerely,
gs

 

[Ville Vainio]

gs> Off to read up on this and try to persuade my isp to let me
gs> add .torrent apptypes

What does your ISP have to do with this? .torrents are just normal
files.

 

[gs]

gs> Proposed solution: I thought that one approach could be to
gs> write a small server for my public server with the sole
gs> purpose of keeping track of connected users and establishing
gs> p2p connections on demand.

One lightweight approach (assuming that a web server is available)
would be to put up a cgi script that adds the iport information to
onlineusers.txt. Then the clients can just use urllib to notify the
script when they become online, and also get the list of connected
clients by just retrieving onlineusers.txt. Of course onlineusers.txt
could contain the lists of shared files for each client as well...

If you use unique user ids, you can keep the file from growing up too
much. If not, you could clean up dead clients by trying to connect all
the specified addresses.

gs> 1) I've written a threaded server that stores information
gs> about connected servers in a "container class". This class
gs> h

olds information about username, a unique ID, the connecting

gs> adress and the actual socket. I have not been able to find any
gs> information online on how i could go about to connect two
gs> sockets. It would seem like the socket is already "occupied"
gs> since it is connected to the server. Could one duplicate, or
gs> otherwise get a dedicated socket for sending binary data, to
gs> one connected socket from another? A few pointers and/or tips
gs> would be greatly appreciated!

If you went on to "connect" the two sockets, you would just
instantiate a thread that forwards the data, and would use more server
bandwidth than an FTP server would have used.

The idea in p2p is to make clients connect to each other. The task of
the server is to tell the clients about other clients so they can
connect to each other.


gs> 2) I've also failed to find python specific information on how
gs> abouts one would do file resuming. I would think that you,
gs> somehow, use md5-checksums to check file status and somehow
gs> skip the first part of the datastream. Also, pointers and
gs> maybe an explanation in (short) pseudo-code would be much
gs> appreciated.

It's trivial since you are going to implement a custom protocol
anyway. Make the "get file" command that you pass to the file provider
such that in addition to the file name you pass the starting offset
within the file.


gs> 3) Is this too big of a project for a novice programmer such
gs> as myself? I do want to go through with this since it's a
gs> genuine problem, but i do not know if i'm ready. I'm not a
gs> "quitter" per say, but i'm afraid doing something TOO advanced
gs> in the beginning might be overwhelming

No, it's a great first project. Go ahead, I believe you will find it
surprisingly easy, educational and something for which Python is a
perfect fit.

It'll be a fun project to expand as well, to provide chunked file
transfer etc. You could also explore the alternative of making it a
bunch of wrapper scripts that just invoke bittorrent...

Thanks all of you for the nice feedback. I'm spending all of my
sparetime just reading up on all technlogies and suggestions!

cheers,
gs

 

[Carl Scharenberg]

Hi!

This is my first time posting to a newsgroup so please be gentle to me


Introduction to my problem:

I'm studying at the university and a lot of friends and I use MSN to
communicate. The problem is that MSN file sending/receiving
capabilities are terribly slow and has no resume option. We're also on
different subnets and can not access eachoters computers through smb
shares, nor can we have local ftp-servers.

<snip>

You might look into the Waste p2p software. It is a p2p program that
uses RSA encryption and requires manual addition of nodes, I believe.
So you can use it to create a private p2p network. It might work for
you, or at least give you some ideas in terms of security features and
such.