How to do UDP calls

[Luis G.]

Hey guys...

I have a question here and maybe someone can help me.

I wrote a script to get information from some websites. Basically I call
an url and I get a file as response. I'm using Mechanize to do that,
like this:

@agent = Mechanize.new do |a|
a.history.max_size = 1
a.read_timeout = 60
end

res = @agent.get(url).save_as(path)

After I get the file, I just need to open and extract the information I
need. Everything is working good.

My problem now is that I have some UDP urls and Mechanize does not
support UDP urls...

So, my question is, how can I get the files from UDP urls?
I looked online for gems to do that, but I didn't find anything...
Any of you can help me?

Thanks,

Luis

 

[Robert Klemme]

Hey guys...

I have a question here and maybe someone can help me.

I wrote a script to get information from some websites. Basically I call
an url and I get a file as response. I'm using Mechanize to do that,
like this:

@agent =3D Mechanize.new do |a|
=A0 =A0 =A0a.history.max_size =3D 1
=A0 =A0 =A0a.read_timeout =3D 60
end

res =3D @agent.get(url).save_as(path)

After I get the file, I just need to open and extract the information I
need. Everything is working good.

My problem now is that I have some UDP urls and Mechanize does not
support UDP urls...

So, my question is, how can I get the files from UDP urls?
I looked online for gems to do that, but I didn't find anything...
Any of you can help me?

The standard library has basic UDP functionality already.

http://www.ruby-doc.org/stdlib/libdoc/socket/rdoc/classes/UDPSocket.html

Kind regards

robert

--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

 

[Brian Candler]

Luis G. wrote in post #968518:

So, my question is, how can I get the files from UDP urls?
I looked online for gems to do that, but I didn't find anything...
Any of you can help me?

Can you give an example of what you mean by a "UDP url"? I don't know of
any standard for doing HTTP over UDP. Is this something like a SIP url
perhaps? TFTP? In that case you could look for a SIP or TFTP client
library.

Otherwise, as has already been said, the code for doing UDP sockets is
there. Have a look at resolv.rb in the standard library for some example
code (this is a pure-ruby DNS resolver)

 

[Luis G.]

Maybe I have not explained well...

I'm basically sending requests to torrent trackers to get the number of
seeds and peers of a torrent in a specific tracker.
As you might know, you can have this:

http://9.rarbg.com:2710/announce
udp://tracker.openbittorrent.com:80/announce

so, you can have http and udp trackers.

For the first one (HTTP) I just use this:

url = tracker_url +
'?info_hash=%19%89B%0AL%87k%9A%2B%28%3A%B36%9D%93%9A%8D%FA%97I'

agent = Mechanize.new do |a|
a.history.max_size = 1
a.read_timeout = 60
end

res = agent.get(url).save_as(path)

with this I get a file with the number of seeds and peers on it. I just
need to open the file in 'path' and extract the information.

But for the second one (UDP) I can't use Mechanize, so I need to work
with the standard library as you told before.

I never used socket library before, so I'm kinda lost.
I was trying something like:

host = 'tracker.openbittorrent.com'
port = 80
data = '%19%89B%0AL%87k%9A%2B%28%3A%B36%9D%93%9A%8D%FA%97I'

sock = UDPSocket.open
sock.connect(host, port)
sock.send("?info_hash=" + data, 0)

But there are some things here that I don't know:

1- The main idea is to create a socket, connect it to the remote host,
send the request and get the response, right?

2- do I need to include the 'udp://' in the host?

3- is ok to include the name of the field I want to send (?info_hash=)
in the data variable? I need to include that as you can see here:
http://wiki.theory.org/BitTorrent_Tracker_Protocol

4- how can I get the response from the server? something like:
sock.recvfrom(1024)?

I tried several options and I had no success.

Any clues?

Thanks,

Luis

 

[Brian Candler]

Luis G. wrote in post #968857:

As you might know, you can have this:

http://9.rarbg.com:2710/announce
udp://tracker.openbittorrent.com:80/announce

No, I haven't seen the udp:// URL format at all. Is there an RFC which
describes it?

For the first one (HTTP) I just use this: ...
But for the second one (UDP) I can't use Mechanize, so I need to work
with the standard library as you told before.

And not only that, you need to know what format data to put *inside* the
UDP packet. That's what I was getting at. You need some documentation of
the application-layer protocol, which sits inside the transport layer
protocol.

I guess it *might* be HTTP, complete with a full set of HTTP headers.
Worth a try, if you have a valid URL to point at.

You can use netcat (nc) to test:

echo "GET /announce" | nc -u tracker.openbittorrent.com 80

and run

sudo tcpdump -i eth0 -nn -s0 udp port 80

in another window to check for any replies. When I try that particular
example, I get nothing back.

1- The main idea is to create a socket, connect it to the remote host,
send the request and get the response, right?

I guess so...

2- do I need to include the 'udp://' in the host?

Nope. The socket functions take hostnames/IP addresses, not URLs. There
is a 'URI' class you can use to parse it out of the URL though.

3- is ok to include the name of the field I want to send (?info_hash=)
in the data variable?

That I have no idea, because I've not seen any documentation for this
protocol.

4- how can I get the response from the server? something like:
sock.recvfrom(1024)?

Yes, recvfrom returns a tuple of [data, address_info]. Or I believe you
could 'connect' the socket first, send, recv.

I tried several options and I had no success.

Any clues?

Look at a C implementation of this protocol, and port it to ruby?
tcpdump -X an existing working client, and see what it sends/receives?
Ask on a bittorrent development mailing list?

 

[Brian Candler]

OK, there's a spec linked from here:
http://en.wikipedia.org/wiki/UDP_tracker

It's not HTTP. It's bittorrent-specific (so using "udp" as the URL label
was silly). You'll find Array#pack will be your friend.

 

[Luis G.]

Hey man, thanks for your reply.

I read the documentation but this is hard for me... But actually I don't
have enough skills to do this without help...

So, as far as I understood in the structures section of the UDP tracker
protocol documentation, the first thing I need to do is:

1. Choose a (random) transaction ID.
2. Fill the connect input structure.
3. Send the packet.

connect input
64-bit integer connection_id 0x41727101980
32-bit integer action 0
32-bit integer transaction_id 16

You told me that Array#pack will help me. This means that I need to put
the the content of the connect input in an array and pack it so I can
send it, right?
Actually I thought we need to have something like:
connection_id =>0x41727101980
action => 0
...

Anyway, see this example:

data = Array.new
data << 0x41727101980 #connection_id
data << 0 #action
data << 16 #transaction_id

host = 'tracker.openbittorrent.com'
port = 80

s = data.pack('q') #q?!

sock = UDPSocket.open
sock.connect(host, port)
sock.send(s, 0)
response = sock.recvfrom(1024)

After this I should:
1. Receive the packet (response).
2. Check whether the packet is at least 16 bytes.
3. Check whether the transaction ID is equal to the one you chose.
4. Check whether the action is connect.
5. Store the connection ID for future use.

You can see something wrong with this code?

BTW, I tried

echo "GET /announce" | nc -u tracker.openbittorrent.com 80
sudo tcpdump -i eth0 -nn -s0 udp port 80

and I got results:
18:09:14.719638 IP 192.168.182.123.57232 > 95.215.62.26.80: UDP, length
14

Thanks a lot,

Luis

 

[Brian Candler]

Luis G. wrote in post #968897:

connect input
64-bit integer connection_id 0x41727101980
32-bit integer action 0
32-bit integer transaction_id 16

You told me that Array#pack will help me. This means that I need to put
the the content of the connect input in an array and pack it so I can
send it, right?
Actually I thought we need to have something like:
connection_id =>0x41727101980
action => 0
...

Anyway, see this example:

data = Array.new
data << 0x41727101980 #connection_id
data << 0 #action
data << 16 #transaction_id

host = 'tracker.openbittorrent.com'
port = 80

s = data.pack('q') #q?!

There are three elements in the array 'data', so:
s = data.pack('qNN')

However 'q' may be wrong here, if you're on a little-endian machine.
Unfortunately, Array#pack doesn't have a 64-bit network-order flag.

I think you should use something like this:

conn_id = 0x41727101780
s = [conn_id >> 32, conn_id & 0xffffffff, 0, 16].pack('NNNN')

Then for debugging, to check you're sending what you think:

STDERR.puts s.unpack("H*").first

BTW, I tried

echo "GET /announce" | nc -u tracker.openbittorrent.com 80
sudo tcpdump -i eth0 -nn -s0 udp port 80

and I got results:
18:09:14.719638 IP 192.168.182.123.57232 > 95.215.62.26.80: UDP, length
14

That's the outbound packet (from 192.168.182.123 local port 57232, to
95.215.62.26 port 80), and you got no response.

 

[Luis G.]

Hello again.

Man, thanks a lot for you time. You've been a really great help.

first question, shouldn't the array just contain 3 element?
64-bit integer connection_id 0x41727101980
32-bit integer action 0
32-bit integer transaction_id 16

I tried to run the script with the information you gave me. So far I =

have this:

host =3D 'tracker.openbittorrent.com'
port =3D 80
connection_id =3D 0x41727101980

s =3D [connection_id >> 32, connection_id & 0xffffffff, 0, =

16].pack('NNNN')

sock =3D UDPSocket.open
sock.connect(host, port)
sock.send(s, 0)
response =3D sock.recvfrom(1024)

And actually now I have something in the 'response' variable. If I print =

the content I got something like

["\000\000\000\000\000\000\000\020#B\005\027=EF=BF=BDAP\377",
["AF_INET", 80, "tracker.openbittorrent.com", "95.215.62.5"]]

And STDERR.puts response.unpack("H*").first will return
In the documentation says that 'Check whether the transaction ID is =

equal to the one you chose.'. So basically I should get the same value I =

sent, right?

Thanks,

Luis

-- =

Posted via http://www.ruby-forum.com/.=

 

[Brian Candler]

Luis G. wrote in post #969040:

first question, shouldn't the array just contain 3 element?
64-bit integer connection_id 0x41727101980
32-bit integer action 0
32-bit integer transaction_id 16

A 64-bit integer is 8 bytes, and two 32-bit integers are 8 bytes.

The problem is you want to pack 0x123456789abcdef as 0123456789abcdef, =

but the 'q' option to pack gives something else (on Intel-machines =

anyway):

n =3D 0x123456789abcdef =3D> 81985529216486895
[n].pack("q") =3D> "\357\315\253\211gE#\001"
[n].pack("q").unpack("H*").first

=3D> "efcdab8967452301"

You want to send it in network-byte order, MSB first. There's no =

function in pack to do this, but "N" packs 32-bits in network-order. So:

[n>>32,n&((1<<32)-1)].pack("NN") =3D> "\001#Eg\211\253\315\357"
[n>>32,n&((1<<32)-1)].pack("NN").unpack("H*").first

=3D> "0123456789abcdef"

Mind you, if the only thing to need to do is check the conn_id is the =

same, maybe you will get away with "q" if you use it the same for =

packing and unpacking.

I tried to run the script with the information you gave me. ...
And actually now I have something in the 'response' variable. If I prin= t
the content I got something like

["\000\000\000\000\000\000\000\020#B\005\027=EF=BF=BDAP\377",
["AF_INET", 80, "tracker.openbittorrent.com", "95.215.62.5"]]

And STDERR.puts response.unpack("H*").first will return
In the documentation says that 'Check whether the transaction ID is
equal to the one you chose.'. So basically I should get the same value = I
sent, right?

The result is an array of two elements, the data and the address info.

Take the first element, then look at data.unpack("NNq") or =

data.unpack("NNNN"). To handle the byte-ordering problem again:

action, trans_id, c0, c1 =3D data.unpack("NNNN")
conn_id =3D (c0 << 32) | c1

(note that they come in a different order in the response than in the =

request)

-- =

Posted via http://www.ruby-forum.com/.=

 

[Luis G.]

Hello again.

Thanks for your explanation.
I understood that we need to split a 64bit integer in two 32 bit integer
and pack them in network-byte order.
So, as you said:

connection_id = 0x41727101980
pp connection_id
s = [connection_id >> 32, connection_id & 0xffffffff, 0,
16].pack('NNNN')
c0, c1, action, trans_id = s.unpack("NNNN")
conn = (c0 << 32) | c1

pp conn
pp action
pp trans_id
I did this small test like you explained and I get the same values after
pack and unpack the information(connection_id = conn).

So, after I pack the data, I send it and I get a response:

sock = UDPSocket.open
sock.connect(host, port)
sock.send(s, 0)
response = sock.recvfrom(1024)

so, I extract the content of the response like this (using a different
order of the request):

action, trans_id, c0, c1 = response[0].unpack("NNNN")
conn_id = (c0 << 32) | c1

pp action
pp trans_id
pp conn_id
In the documentation says:
- Check whether the transaction ID is equal to the one you choose - I
got the same value '16', so I think is ok;
- Check whether the action is connect - I got a '0'... Shouldn't I get a
'1'?

The next step is to fill the announce input and send it, so I can get
the seeds and peers (my goal ).
I have one question here: All the fields are required, right? Because I
don't really know some fields, like: peer_id, downloaded, left,
uploaded, event, key. the field key, for example don't even appear in
BitTorrent Tracker Protocol
(http://wiki.theory.org/BitTorrent_Tracker_Protocol)

Just one more question... The info_hash is a 20-byte string. To pack
this variable, I can use the 'm' flag, right?

Thanks,

Luis Goncalves

 

[Brian Candler]

Luis G. wrote in post #969537:

In the documentation says:
- Check whether the transaction ID is equal to the one you choose - I
got the same value '16', so I think is ok;
- Check whether the action is connect - I got a '0'... Shouldn't I get a
'1'?

Not from my reading of the description at
http://xbtt.sourceforge.net/udp_tracker_protocol.html

It says the 'connect input' packet is

Offset Size Name Value
0 64-bit integer connection_id 0x41727101980
8 32-bit integer action 0
12 32-bit integer transaction_id
16

and the 'connect output' packet is
Offset Size Name Value
0 32-bit integer action 0
4 32-bit integer transaction_id
8 64-bit integer connection_id
16

This is admittedly a pretty rubbish protocol document, but if you
understand the semantics of each of these messages from some other
reference, it might make sense to you.

The next step is to fill the announce input and send it, so I can get
the seeds and peers (my goal ).
I have one question here: All the fields are required, right? Because I
don't really know some fields, like: peer_id, downloaded, left,
uploaded, event, key. the field key, for example don't even appear in
BitTorrent Tracker Protocol
(http://wiki.theory.org/BitTorrent_Tracker_Protocol)

I'd say this is no longer anything to do with Ruby, but all about how to
write a BitTorrent protocol implementation, and so this would be best
answered on a BitTorrent mailing list. Sorry.

Just one more question... The info_hash is a 20-byte string. To pack
this variable, I can use the 'm' flag, right?

I doubt it, that would MIME-encode it. You probably want 'a20'

[-1,"foo",-1].pack("Na20N")

=>
"\377\377\377\377foo\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\377\377\377\377"

[-1,"foobarfoobarfoobarfoobar",-1].pack("Na20N")

=> "\377\377\377\377foobarfoobarfoobarfo\377\377\377\377"