Real project -- general application structure

[Ricardo Jasinski]

Hi everyone,

I'm supposed to build a test application where a PC would communicate
with many other devices (say, temperature sensors) via TCP/IP sockets,
and display their current status on a web page. Furthermore, all
gathered information should be stored on a database.

I just started learning ruby/rails but I'm already hooked on it. In a
week I could do two things:
- communicate with the sensors, and show incoming messages on a
text/command line window. This could be easily extended to store the
information on the database;
- access the database and present formatted data on a dynamically
generated web page.

What I really can't imagine right now is how to put these two things
together, in a way that whenever a sensor sends new status
information, the corresponding line in the webpage gets updated
asynchronously.

My first guess would be having two separate processes or applications,
one in charge of communicating with the devices, and the other would
be the web server itself. These two should communicate with each other
as well, though it's not clear to me how this would happen (maybe a
socket between the two applications? I would rather not use the
database for the kind of transitory message they will be exchanging).

I'd be glad to hear your thoughts, and implement as many suggestions
as possible. I'll give you feedback as I go on with this; I was hoping
to have something working in a month or so.

I'm very new to this (I'm more of a hardware designer), so even a
short comment could be extremely valuable to me. Thanks in advance!

Ricardo Jasinski.

 

[Joel VanderWerf]

Hi everyone,

I'm supposed to build a test application where a PC would communicate
with many other devices (say, temperature sensors) via TCP/IP sockets,
and display their current status on a web page. Furthermore, all
gathered information should be stored on a database.

Do you know about RRDTool? There are ruby bindings for it, too.

http://oss.oetiker.ch/rrdtool/

I'm using it to do more or less what you say: remote systems measure
temperature and other variables, and fire the data to a server (UDP, not
TCP, though). The server collects it into RRDs (Round Robin Databases),
and there are some simple CGI scripts (not ruby, but RRDTool's own mini
scripting language) to display the data according to user inputs (time
ranges etc).

Of course, if you're doing this to learn Rails, maybe you want to do it
the hard way ....

 

[ChrisH]

Hi everyone,

I'm supposed to build a test application where a PC would communicate
with many other devices (say, temperature sensors) via TCP/IP sockets,
and display their current status on a web page. Furthermore, all
gathered information should be stored on a database.

So the data collection app stores it in the DB, and the web page just
retrieves the latest data
from the DB when a user navigates to the page. Piece of cake! 9^)

 

[Ricardo Jasinski]

So the data collection app stores it in the DB, and the web page just

retrieves the latest data
from the DB when a user navigates to the page. Piece of cake! 9^)

Hi Chris,

thanks for your input! Yes, it is really almost a piece of cake,
except that the data displayed on the page must be updated whenever
new information is sent by one of the sensors (without having to click
'refresh' on the web browser).

Of course a periodic page reload could take care of this issue, but I
believe we can do better than that... =)

Cheers,

Ricardo

 

[snacktime]

Hi Chris,

thanks for your input! Yes, it is really almost a piece of cake,
except that the data displayed on the page must be updated whenever
new information is sent by one of the sensors (without having to click
'refresh' on the web browser).

Of course a periodic page reload could take care of this issue, but I
believe we can do better than that... =)

Have a status table that gets updated whenever an update comes in, and
on the webpage use periodically_call_remote to check the table to see
if there are new updates. Or you can just do an auto refresh of the
page every so often also.

 

[Ricardo Jasinski]

Have a status table that gets updated whenever an update comes in, and

on the webpage use periodically_call_remote to check the table to see
if there are new updates. Or you can just do an auto refresh of the
page every so often also.

so there's no other way of making the two applications talk to each
other except via the database? Since these messages are completely
transitory and don't have any real need to be in the database, I was
willing to avoid using the db for that...

Thanks for your comments!

Ricardo.

 

[Florian Aßmann]

=20
so there's no other way of making the two applications talk to each
other except via the database? Since these messages are completely
transitory and don't have any real need to be in the database, I was
willing to avoid using the db for that...
=20
Thanks for your comments!
=20
Ricardo.
=20
=20
=20

Hi Ricardo,

I haven't done anything with EventMachine for the time being but this
could be a solution for you.

Maybe you want to write a daemon that retrieves sensor data and provides
it through EM (maybe as a webservice) to its clients who'll drive a
simple GUI (something in .NET for example).

-> No polling, no Weblication Overkill, no Database, ...

Just a way I'd go for this.

Regards
Florian

 

[Ricardo Jasinski]

Hi Florian,

thanks a lot for pointing me out the existence of Eventmachine; I had
missed that one completely.

I was amazed to see how easy it is to structure the terminal
connections using EM. In less than 5 minutes after installing the gem
I could already handle connections and interpret the sensors data.

However, I realized that it was essentially the same that my original
code was already doing (excpet it is much cleaner and way cooler!). I
can easily talk to the remote units, but I still need to find a way to
display their momentary status on a web page (which should always
display up-to-date information, and be independent of full page
reloads on the browser).

What would you suggest; can I use EM to talk to my web server which
would be another application? Or perhaps open a pipe to it somehow?

Thanks a lot!

Ricardo.

 

[snacktime]

Hi Florian,

thanks a lot for pointing me out the existence of Eventmachine; I had
missed that one completely.

I was amazed to see how easy it is to structure the terminal
connections using EM. In less than 5 minutes after installing the gem
I could already handle connections and interpret the sensors data.

However, I realized that it was essentially the same that my original
code was already doing (excpet it is much cleaner and way cooler!). I
can easily talk to the remote units, but I still need to find a way to
display their momentary status on a web page (which should always
display up-to-date information, and be independent of full page
reloads on the browser).

What would you suggest; can I use EM to talk to my web server which
would be another application? Or perhaps open a pipe to it somehow?

You might take a look at using mongrel (with the swiftiply patch to
have mongrel use EM) and create your own handler. That gives you a
custom http server with the EM engine running. Look at the mongrel
examples/simpletest.rb file. Towards the bottom it calls run, at the
point EM is running. You could for example put an EM periodic timer
there to fire off your checks on the remote units, and create a class
variable (singleton) to hold the latest data that your handler can
access whenever an http request comes in.

Now you just have one program to handle everything.

Chris

 

[Florian Aßmann]

Hi Florian,
=20
thanks a lot for pointing me out the existence of Eventmachine; I had
missed that one completely.
=20
I was amazed to see how easy it is to structure the terminal
connections using EM. In less than 5 minutes after installing the gem
I could already handle connections and interpret the sensors data.
=20
However, I realized that it was essentially the same that my original
code was already doing (excpet it is much cleaner and way cooler!). I
can easily talk to the remote units, but I still need to find a way to
display their momentary status on a web page (which should always
display up-to-date information, and be independent of full page
reloads on the browser).
=20
What would you suggest; can I use EM to talk to my web server which
would be another application? Or perhaps open a pipe to it somehow?
=20
Thanks a lot!
=20
Ricardo.
=20
=20
=20

Hi Ricardo,

I wouldn't either do polling nor use ajax because this method requires
polling. There's no way around this since webpages are not meant this way=
=2E

Instead I'd embed, if you really need so, a java applet that accepts
messages from your event machine telling it to refresh or even the new
information.

Don't know if Flash can do, too... or maybe Silverlight... *chuuu*

Better, imho, is a little spectating app that's easily build upon .NET
framework.

You may glue them together through soap.

That leaves the frontend data be handled by the spectating app and the
daemon do almost nothing except telling the EM to send and reading the
sensors.

What's the big idea behind the webpage thing?

Regards
Florian

P.S.:
Maybe you can use an already written snmp client...

 

[Ricardo Jasinski]

Hi Florian,

this is starting to look good. I'd better ellaborate on the
application a little bit more:

The remote units will monitor a series of environmental variables on
the ambient where they are installed. Whenever there's a significant
change, or periodically, the units will open a socket to the server
and send it some updated data.

This is the remoute units <-> server half of our application; in
addition to that, we need to put up a web server so that the person
who installed the remote units on his locations can easily see the
updated status of all sites. Basically, that's it.

This was my first idea for the application structure:


TCP/IP sockets
+---------------------+ +------------+
[Sensor#1] <-------> | | | |
[Sensor#2] <-------> | Sensors interfacing | <-------> | Web server |
[Sensor#3] <-------> | application | | (App#2) |
... | (App#1) | +------------+
+---------------------+ ^
^ /
\ ________ v
v/ \
\________/
| |
| DB |
\________/


App#1 needs to access the DB mainly for validation purposes. App#2
needs DB access to provide nicely formatted web pages with sensor data
and other stuff to the guy on the couch who will be checking out the
sensors status on his web browser.

Ah, and I can't install any software on the guy's PC.

Regarding the sensors, they are somewhat limited embedded devices, so
I can count on them having a decent TCP/IP stack, but I don't think I
could squeeze anything like RPC or XML into them.

Instead I'd embed, if you really need so, a java applet that accepts
messages from your event machine telling it to refresh or even the new
information.

This sounds nice, does it fit the above description?

Thanks again!

Mfg,

Ricardo.

 

[Florian Aßmann]

hi Ricardo

_____ sends sensor data to connected _____
TCP/IP sockets ^ v
+---------------------+ +-------------------+ +------------------+
[Sensor#1] --------> | EventMachine | | Weblication | | Browser |
[Sensor#2] --------> | (sensors interface) | | beautiful HTML + |-------->| displays HTML + |
[Sensor#3] --------> | | | Java applet | | runs Java applet |
... | | +-------------------+ +------------------+
+---------------------+ ^
/
________ v
/ \
\________/
| |
| DB |
\________/

I'm not very in ASCII...

Regards
Florian

 

[Florian Aßmann]

Hi Wayne,

Use periodically_call_remote to have the client poll (unless you want t= o
code a Java applet, but do you really want to get into that?).

Ok, don't think this would take more than 2 days and you have a proper
application/gui that doesn't stress your server when you have more than
10 clients simulatiously fetching data from the rails balloon, he
possibly wants it realtime, doesn't he?

_He can even spend several thousand of dollars having a great connection
and hardware, then it doesn't need to scale - it just performs (hopefully=
)._

AJAX is a great framework to disburden your webserver and polling is,
imho, the Reverse=E2=84=A2...

Regards
Florian

 

[Florian Aßmann]

In addition to the comet plugin there is also juggernaut

http://juggernaut.rubyforge.org/

I found a writeup on rails spikes
http://www.railspikes.com/tags/comet

Cheers

Something like this I had in mind when I said Flash but it uses - aehrm
- Shockwave files?

Sincerely
Florian

 

[Ricardo Jasinski]

Daniel,

this is a lot of useful information. You got straight to the point:
how can I push data to the clients based on the occurrence of a server-
side event? Now I understand that this is no standard browser
behavior.

I read the article you pointed about Juggernaut, that's really cool.
I'm sure going to try it in some personal projects, but we were told
in advance that we couldn't use Flash on this one.

Also, today I had a meeting with the guys who will be doing the real
application (mine is just a proof of concept/demonstration) and they
told us that all information will have to go through the DB (unlike my
previous ascii art diagram). I believe they will opt for a polled
solution from the web browser, since they said that the sensors status
only needs to be updated once every minute or so (much less real-time
than I was thinking...)

I already have a simple server working with EventMachine. Now I feel
that I can go ahead on my own, with all the advice you guys have
provided me.

This second week on RoR has been so productive!

Thanks everyone! Cheers,

Ricardo.

 

[Ry]

Hi Ricardo,

I have a javascript class, Ajax.Pull, which, I believe, does exactly
what you need: allows a server to push data to the client. By using a
novel XMLHttpRequest technique the mechanism becomes rather more lucid
than other methods.

The server provides a URL which streams JSON content. The client
connects with Ajax.Pull to this URL to receive updates. For example,
your server might stream the following JSON objects like this

{"temperature": {"time":1184676153,"value":25.54},
"pressure": {"time":1184676154,"value":1.2}
};
{"temperature": {"time":1184676156,"value":25.53},
"pressure": {"time":1184676158,"value":1.1}
};

The handler you provided in the client might look like

function handler(json) {
$('temperature').innerHTML = json['temperature']['value'];
$('pressure').innerHTML = json['pressure']['value'];
}

Unfortunately you need to go around Rails in order to do stream data
like this. It's not so hard to write a small Mongrel handler to do
this - I've done this to implement an upload progress bar. The my code
from the upload progress bar handler contains an example Rails
application. It shouldn't be so hard to modify this upload progress
bar to fit your needs.

http://four.livejournal.com/777996.html
info for the progress bar mongrel handler
http://four.livejournal.com/777996.html
explicit instructs for starting the handler
http://tinyclouds.org/svn/ajax_pull/trunk/
svn repo for the javascript class. it contains another example of
how to write a mongrel handler.

Let me know if you use this - I'd like to see it in action!

ry

p.s. hopefully i haven't sent this message twice - google groups seems
to not be working properly