NetworkFacade 0.4

[Florent Solt]

NetworkFacade is an alternative to DRb, XML/RPC and other REST api,
all in one gem !

http://network-facade.rubyforge.org/

The easiest way to describe it, is through examples.

A simple tcp client/server :

# -- Server --
class Foo
def bar; 42; end
end
server = NetworkFacade::Server.newport => 5042)
server << Foo.new
server.start

# -- Client --
class Foo < NetworkFacade::Client 'nf://localhost:5042'
end
f = Foo.new
puts f.bar

A REST client, for Digg :

class Digg < NetworkFacade::REST::Client 'http://services.digg.com'
mapping :apple_stories, '/stories/topic/apple'
mapping :me, '/user/pyros'
end

d = Digg.new
puts d.apple_storiesappkey => 'http://network-facade.rubyforge.org',
:count => 3, :type => :json)
puts d.meappkey => 'http://network-facade.rubyforge.org')

Many other examples (Unix socket, SSL, Compression, Flickr,
Remember the milk, Netvibes) are explained here :

http://network-facade.rubyforge.org/doc

 

[Felipe Contreras]

NetworkFacade is an alternative to DRb, XML/RPC and other REST api,
all in one gem !

http://network-facade.rubyforge.org/

The easiest way to describe it, is through examples.

A simple tcp client/server :

# -- Server --
class Foo
def bar; 42; end
end
server = NetworkFacade::Server.newport => 5042)
server << Foo.new
server.start

# -- Client --
class Foo < NetworkFacade::Client 'nf://localhost:5042'
end
f = Foo.new
puts f.bar

A REST client, for Digg :

class Digg < NetworkFacade::REST::Client 'http://services.digg.com'
mapping :apple_stories, '/stories/topic/apple'
mapping :me, '/user/pyros'
end

d = Digg.new
puts d.apple_storiesappkey => 'http://network-facade.rubyforge.org',
:count => 3, :type => :json)
puts d.meappkey => 'http://network-facade.rubyforge.org')

Many other examples (Unix socket, SSL, Compression, Flickr,
Remember the milk, Netvibes) are explained here :

http://network-facade.rubyforge.org/doc

Sweet!

I did something similar:
http://snipplr.com/view/2476/ruby-rest/

But I definitely like your :mapping thing much better.

I'm still not sure if I would like the whole Network Facade just for
REST, but maybe it makes sense for XML/RPC or DRb.

Good job.

 

[Florent Solt]

I'm still not sure if I would like the whole Network Facade just for
REST, but maybe it makes sense for XML/RPC or DRb.

You know, network-facade is just a 15k lib

Good job.

Thanks

 

[Brian Candler]

Many other examples (Unix socket, SSL, Compression, Flickr,
Remember the milk, Netvibes) are explained here :

http://network-facade.rubyforge.org/doc

I can't find any documentation of the TCP protocol itself.

Looking at the code, the call appears to be a four-byte length followed by
a Marshal dump of
[@uri.path[1..-1], name, args]
and the response is a 4-byte length followed by a Marshal dump of a single
object. This means that the protocol is similar to DRb, but different, i.e.
not compatible :-(

I can see a problem with this protocol: it's impossible to distinguish
between a method returning an Exception object and a method raising an
exception.

There are certainly problems which DRb has which could do with solving.
Chief in my mind these are:

(1) Doesn't run easily over FastCGI, IO.popen and stdin/stdout, or Unix
domain sockets.

(2) Doesn't allow bi-directional method calls through a single connection,
e.g. for use through a firewall and/or NAT and port forwarding.

(3) Doesn't interoperate with non-Ruby applications. I would like to be able
to carry strings and integers as plain strings. This would make it easy to
interoperate with Perl, say.

(4) Not easy to secure.

(5) Large and complex implementation, not easy to understand, especially
with regards to connection caching and what happens if connections are
dropped.

Your library addresses (4), (5) and part of (1). But I think (2) and (3)
need careful thinking to produce a suitable protocol. As for (2), I'm not
sure your protocol supports proxy objects at all, a.k.a. DRbUndumped, but
please correct me if I'm wrong.

I think a replacement protocol for DRb should have the following
characteristics:

* 'Requests' and 'Responses' explicitly marked as such, so that calls can be
made in both directions down the same socket.

* Each Request and Response to be tagged with an ID, so multiple overlapping
calls can be multiplexed onto the same socket, meaning that only a single
TCP connection needs to be kept open between a particular pair of hosts.

* Ability to pass Undumped objects in such a way that the callback is either
made back down the same socket, or to a global URI (the latter is what DRb
does). The latter also allows the callback to be made independently, and
avoids forming long chains of tunnels. (+)

* Basic protocol uses only string encoding, allowing tiny implementation
suitable for embedded devices etc, and cross-language interoperability.

* Optionally enable other native marshalling protocols (e.g. Marshall, YAML,
Perl Storable etc) if understood by both sides. Ideally negotiated.

* Optional SASL authentication as well as, or instead of, SSL. This allows
basic username/password authentication (AUTH PLAIN) which is simpler to
configure than client certificates.

* Optional synchronous mode (one request -> one response) for tunneling over
HTTP or FastCGI and for simplistic implementations in embedded devices

* Automatic establishment or re-establishment of dropped connections

I also think that implementations should consider using an opaque key to
identify each object, rather than it's object_id. For one thing, object_id's
can be recycled; for another, it prevents people probing for random objects
which have not been explicitly shared. Keeping a mapping table of opaque_key
=> object will prevent DRbUndumped objects from being garbage collected
(although this may or may not be desirable)

Just a few thoughts.

Regards,

Brian.

(+) It would be useful to pass a different global URI depending on where you
are connecting to. For example, if you are connecting outbound through the
Internet, the global URI you expose may be your firewalls' outside IP or
hostname and a port on the firewall which forwards inbound connections.

To make this more transparent, it could be useful to auto-detect NAT. This
could be done by passing your *actual* source IP and port inside the
connection message; the far end would compare these to the *seen* source IP
and port. IPSEC NAT traversal uses a similar mechanism.

 

[Florent Solt]

Before anything else, thanks for your long and interesting reply.

I can't find any documentation of the TCP protocol itself.

The documentation is still incomplete, I will wait some
versions to improve it.

Looking at the code, the call appears to be a four-byte length followed
by
a Marshal dump of
[@uri.path[1..-1], name, args]
and the response is a 4-byte length followed by a Marshal dump of a
single
object. This means that the protocol is similar to DRb, but different,
i.e.
not compatible :-(

It's true. I don't try to do a compatible protocol, because I think it's
difficult to add some new ideas with the same protocol.

I can see a problem with this protocol: it's impossible to distinguish
between a method returning an Exception object and a method raising an
exception.

Also true. I do not realize that. I will try to fix it.

There are certainly problems which DRb has which could do with solving.
Chief in my mind these are:

(1) Doesn't run easily over FastCGI, IO.popen and stdin/stdout, or Unix
domain sockets.

(2) Doesn't allow bi-directional method calls through a single
connection,e.g. for use through a firewall and/or NAT and port forwarding.

(3) Doesn't interoperate with non-Ruby applications. I would like to be
able to carry strings and integers as plain strings. This would make it
easy to interoperate with Perl, say.

(4) Not easy to secure.

(5) Large and complex implementation, not easy to understand, especially
with regards to connection caching and what happens if connections are
dropped.

Your library addresses (4), (5) and part of (1). But I think (2) and (3)
need careful thinking to produce a suitable protocol. As for (2), I'm
not sure your protocol supports proxy objects at all, a.k.a.
DRbUndumped, but please correct me if I'm wrong.

I think a replacement protocol for DRb should have the following
characteristics:

* 'Requests' and 'Responses' explicitly marked as such, so that calls
can be made in both directions down the same socket.

I will do this, you are right.

* Each Request and Response to be tagged with an ID, so multiple
overlapping calls can be multiplexed onto the same socket, meaning
that only a single TCP connection needs to be kept open
between a particular pair of hosts.

Very interesting too, I will implement this too.

* Ability to pass Undumped objects in such a way that the callback
is either made back down the same socket, or to a global URI (the
latter is what DRb does). The latter also allows the callback to
be made independently, and avoids forming long chains of tunnels. (+)

I don't realy understand this point, my english is not perfect
Could you explain it, thanks.

* Basic protocol uses only string encoding, allowing tiny implementation
suitable for embedded devices etc, and cross-language interoperability.

* Optionally enable other native marshalling protocols (e.g. Marshall,
YAML, Perl Storable etc) if understood by both sides. Ideally negotiated.

I already thought about that, and I would like to release this in the
next
version (0.5). I would like to have these serialization methods :
- YAML
- JSON
- XML
- Marshall-Ruby

And in a second time :
- PHP
- Perl
- String encoding

I already don't know some problem, for example, how to serialize a ruby
exception in php format

* Optional SASL authentication as well as, or instead of, SSL. This
allows basic username/password authentication (AUTH PLAIN) which is
simpler to configure than client certificates.

It could be a great idea but I'm not familiar with SASL, I will take a
look
at some docs.

* Optional synchronous mode (one request -> one response) for tunneling
over HTTP or FastCGI and for simplistic implementations in embedded devices

I will think about this too, I could be cool, but I don't know if I will
be
able to merge this with the current code.

* Automatic establishment or re-establishment of dropped connections

Already planned

I also think that implementations should consider using an opaque key to
identify each object, rather than it's object_id. For one thing,
object_id's can be recycled; for another, it prevents people probing
for random objects which have not been explicitly shared. Keeping a
mapping table of opaque_key => object will prevent DRbUndumped objects
from being garbage collected (although this may or may not be desirable)

object_id is just used with the logger (through client_id method) to
print
something revelant to the user, and for example, it's overloaded by
TCP::Server. So, if I'm not wrong, it's not necessary.

Just a few thoughts.

Thanks again for your thoughts.

Regards,

Brian.

(+) It would be useful to pass a different global URI depending on where
you are connecting to. For example, if you are connecting outbound through
the Internet, the global URI you expose may be your firewalls' outside IP or
hostname and a port on the firewall which forwards inbound connections.

To make this more transparent, it could be useful to auto-detect NAT.
This could be done by passing your *actual* source IP and port inside the
connection message; the far end would compare these to the *seen* source
IP and port. IPSEC NAT traversal uses a similar mechanism.

This could be great, but not for the moment. I will wait a stable and
clean
API before doing this kind of feature.

Best regards.

 

[Brian Candler]

I don't realy understand this point, my english is not perfect
Could you explain it, thanks.

Firstly, do you understand DRbUndumped? It took me a while to work this out,
and I ended up writing it up on the web. See
http://wiki.rubygarden.org/Ruby/page/show/DRbTutorial
and see if that makes sense.

In particular, skip down to section headed "Why does the client run
'DRb.start_service'?"

I already thought about that, and I would like to release this in the
next
version (0.5). I would like to have these serialization methods :
- YAML
- JSON
- XML
- Marshall-Ruby

And in a second time :
- PHP
- Perl
- String encoding

I already don't know some problem, for example, how to serialize a ruby
exception in php format

As the lowest common denominator, I would encode an exception as a string.
It's pretty much impossible to map arbitary PHP exceptions to Ruby
exceptions and vice versa, so on receiving one of these exceptions you'd
create a generic exception, say NetworkFacade::RemoteError, with the message
containing the string.

Or maybe you could return an array of [class, message, backtrace].

If the two ends agree on Ruby Marshal, then the exception can be sent in
this format. You might still have to turn this into a generic RemoteError,
e.g. if the exception class contained in the marshalled object is not known
at the client side.

It could be a great idea but I'm not familiar with SASL, I will take a
look
at some docs.

RFC 2222 is the base SASL document, but it's not very helpful. A good
starting point is to see how CRAM-MD5 is done in IMAP (RFC 2195) and then
look at the PLAIN method (RFC 2595). This brings SASL back down to the level
of a simple username and password login

It doesn't have to be SASL of course, but this is the official "extensible"
authentication mechanism for protocols. In the end, having a simple
username/password or shared secret authentication on top of TLS is what I'd
like to see.

object_id is just used with the logger (through client_id method) to
print
something revelant to the user, and for example, it's overloaded by
TCP::Server. So, if I'm not wrong, it's not necessary.

Some sort of object_id is needed when making callbacks - but we are back to
the issue of DRbUndumped.

BTW I think if you're going to use your existing library as a starting point
and start modifying it, then you'll need at least a protocol version field
on connection

I think many or all the options - SSL, Zlib, Marshal, Pipeline (overlapping
requests down the same socket) etc - could be offered as bits in a bitmap.
Some of them are conflicting, e.g. the server will not choose both SSL and
Zlib simultaneously.

Another thought: when using SSL, the client should negotiate SSL so that the
same port can be used for SSL and non-SSL connections. Also it should send
the hostname it connected to in the initial connection message, before SSL
is turned on. This enables 'virtual hosting' where the server can choose the
correct one out of many certificates. You might not think this is important,
but it's something which can easily be gotten right if you design it in at
the start.

RFC 2817 describes a HTTP extension for this. Unfortunately, because it came
along late, no-one implements it so we're stuck with having separate ports
for HTTP and HTTPS :-(

Regards,

Brian.

 

[Tomas Pospisek's Mailing Lists]

I think many or all the options - SSL, Zlib, Marshal, Pipeline (overlapping
requests down the same socket) etc - could be offered as bits in a bitmap.
Some of them are conflicting, e.g. the server will not choose both SSL and
Zlib simultaneously.

If you, Florent, choose to implement this, and implement this as a
bitmap, then take care not to make the bitmap too short or fixed length
since the "negotiable features" will be limited then.
*t

 

[Brian Candler]

If you, Florent, choose to implement this, and implement this as a
bitmap, then take care not to make the bitmap too short or fixed length
since the "negotiable features" will be limited then.

Nah. You just make bit 31 be the "extended options" option

I'm keen to see the protocol be lightweight and simple to parse.

 

[Florent Solt]

On Wed, 9 May 2007, Brian Candler wrote:

If you, Florent, choose to implement this, and implement this as a
bitmap, then take care not to make the bitmap too short or fixed length
since the "negotiable features" will be limited then.
*t

For the next release (0.5) I will focus on these features :
- Exception confusion
- Auto reconnection
- Request/Response container with uniq ID
- Pipe support (not something over a socket, but popen)
- Serialization abstraction
- JSON support (via json gem)
- YAML support (via stdlib)
- XML support (via a new gem I will wrote)

After that, I will come back and talk here about SASL, NAT traversal,
pipeline and tunnel over HTTP features.

Thanks again for your comments. If you want to contribute or patch the
lib, you are welcome.

 

[Florent Solt]

Nah. You just make bit 31 be the "extended options" option

I'm keen to see the protocol be lightweight and simple to parse.

I would like to keep the protocol K.I.S.S as much as possible.