Application architecture (long post - sorry)

[limeydrink]

Hi all,

I want to create a mobile field worker data solution.

Let me explain...

I work for a company that has some software used by call takers to
enter information into a database about faults with electrical
appliances they manufacture, sell to customers, and then provide
maintenance contracts for.

The company has a number of field workers (growing number) servicing
these appliances, and information about the faults and customer
locations is fed to them by printing off job sheets from the call
taking system and then either faxing them to the engineers or the
engineers drop in to the office and collect them.

There are lots of problems with this, such as lost job information,
incomplete forms, incorrect non validated data, the cost of printing
off job sheets and also the engineers having to collect, regular phone
calls to the engineers to update them on call informtion, and then the
fact that all this data then has to be inputted back into another
system manully.

Basically I want to create a means of getting this data to them
electronically.

I know there are a few companies who could provide this solution but
they are very expensive and possibly overkill at the moment, we could
start developing our own basic system then it can grow over time.

I have looked at the options for developing the client for these
"electronic job sheets" and have decided upon Microsoft Pocket PC and
the .net compact framework. It seems the easiest environment for
developing and the PDA's can be obtained very cheaply as a package from
a GSM/GPRS data carrier.

The server side is where cost savings can be made and so i have decided
on a linux based server side using a database backend such as
PostgreSQL.

It would be usefull (and preferable) to make the server side of the
applicaiton portable, I have looked at Java and quite frankly I have
gone off the idea, it seems too complicated and I don't need to have my
application tick all the buzzword boxes - basically it doesn't need to
be a fashionable solution and if it did I would run it all under
Windows and .net from client to server. I have decided on Python
(hopefully your advice will confirm to me it's the correct decision)

I have not developed any software of this type before and the trouble I
am having is deciding on how to interface between the client apps and
the server side.

Problem 1 - Physical connectivity
how would I make the actual network connection in to the server from
the client
Running a VPN connection over GPRS ?
Connecting to the web on the client then using web services ?

The above decisions probably depend on the answer to problem 2...
How to connect to the data
If I have a VPN connection to the network as described above could it
be just as simple as using ODBC to connect to the server and access the
data like just any other database application (not knowing how network
heavy this would be over a much slower connection than a LAN)

Or even simpler I could just export data for the engineers into XML or
..csv files and expose them using a web server the client then just
connects to the server and then downloads the job data , and uploads
completed job information

Or (this is where the bulk of the python part comes into it), I create
a network server in Python that the clients connect to this could be a
two way thing then, as the client app could connect first thing in the
morning and collect jobs but I could also push data to the client such
as job updates and notifications making it much smarter and
interactive.

There are currently around 60 engineers and they will be requesting
data at specific times, first thing in morning and probably midday
being the most busy times where probably all engineers will be
connecting at once.

So if I go down the network server route it will have to handle
multiple connections from clients and then connect to a database to
retrieve job information for each specific engineer and then send over
the connection to the client.

The number of engineers is increasing steadily and so this application
needs to be fairly scaleable.

I have a grander plan for this, as much of the work will be done in my
own time, as sort of a pet project, and if possible I would like to
make the application customizable and maybe even sell to other
companies although lets not run before we can walk eh


I know this post isn't Python specific but there are some smart guys on
here that could point me in the right direction or at least give me
something to think about

Thanks in advance

 

[Mike Meyer]

I have looked at the options for developing the client for these
"electronic job sheets" and have decided upon Microsoft Pocket PC and
the .net compact framework. It seems the easiest environment for
developing and the PDA's can be obtained very cheaply as a package from
a GSM/GPRS data carrier.

That seems like overkill for this application.

Problem 1 - Physical connectivity
how would I make the actual network connection in to the server from
the client
Running a VPN connection over GPRS ?
Connecting to the web on the client then using web services ?

The physical connectivity isn't your worry. How you access it from
software depends on your chosen platform.

The above decisions probably depend on the answer to problem 2...
How to connect to the data
Or even simpler I could just export data for the engineers into XML or
.csv files and expose them using a web server the client then just
connects to the server and then downloads the job data , and uploads
completed job information

I've gone this route before, and this is why I say your .net solution
looks like overkill. The phones that GSM providers *give away* have
web browsers built into them, and it's been that way for years. No
need for .net or Pocket PC or whatever on the mobile device - just get
a web-enabled phone. This makes half the development trivial - you
don't have to do *any* development on the mobile side of things.

The hard part for you will be finding a competent web designer who
understands the web well enough to write for something other than
WinIE - in particular, who can produce pages that will work properly
on your target platform, no matter what it is.

The second hardest part will be picking the mobile platform. That will
depend on what you want to display, as low-end devices any not have a
big enough display, and on what kind of data entry has to be done in
the field: if cursor motion and select is enough, any thing will do. A
phone keypad is ok for short text. For longer things, you may want
something like Graffiti or even a real keypad. This latter takes you
out of phones and back into Pocket PC land - but only to get the UI,
not for development.

Once you've chosen the platform, you'll know what browser it has
available, and that will provide the requirements for the web
developer. The range is from simple (no frames, limited table nesting,
no CSS, no JS, and similar things) to very complete (Opera runs on
some of these devices. They call the Pocket PC browser IE, but I've
never dealt with it). You can design web apps to work around any/all
of these things being missing; all it takes is that competent web
designer.

I normally recommend picking the software first, and lettinng that
drive the choice of hardware. But in this case, the hardware is a
crucial part of the user interface, and you need to make sure that
that's acceptable. And the mobile hardware choice - assuming you go
the web route - isn't going to have a critical effect on the server
software.

Or (this is where the bulk of the python part comes into it), I create
a network server in Python that the clients connect to this could be a
two way thing then, as the client app could connect first thing in the
morning and collect jobs but I could also push data to the client such
as job updates and notifications making it much smarter and
interactive.

Web servers generally don't do push. There are technologies that do -
but then you're limiting your mobile platform choices to things that
support it, or having to port it yourself. Unless there's a real
reason for wanting to push the data, I'd avoid it. That doesn't mean
the your field engineers can't get updates whenever they want - they
just have to ask for them.

There are lots of python choices for building web applications. Maybe
to many. You could use just about any of them.

There are currently around 60 engineers and they will be requesting
data at specific times, first thing in morning and probably midday
being the most busy times where probably all engineers will be
connecting at once.

You'll need to make sure you have a fast enough machine for that. A
33MHz 386 ought to do the trick .

So if I go down the network server route it will have to handle
multiple connections from clients and then connect to a database to
retrieve job information for each specific engineer and then send over
the connection to the client.

This is a pretty standard architecture for a web application. Any of
the python web application tools can handle this.

<mike

 

[Ernst Noch]

Hi all,

I want to create a mobile field worker data solution.

Let me explain...

I work for a company that has some software used by call takers to
enter information into a database about faults with electrical
appliances they manufacture, sell to customers, and then provide
maintenance contracts for.

The company has a number of field workers (growing number) servicing
these appliances, and information about the faults and customer
locations is fed to them by printing off job sheets from the call
taking system and then either faxing them to the engineers or the
engineers drop in to the office and collect them.

There are lots of problems with this, such as lost job information,
incomplete forms, incorrect non validated data, the cost of printing
off job sheets and also the engineers having to collect, regular phone
calls to the engineers to update them on call informtion, and then the
fact that all this data then has to be inputted back into another
system manully.

Basically I want to create a means of getting this data to them
electronically.

I know there are a few companies who could provide this solution but
they are very expensive and possibly overkill at the moment, we could
start developing our own basic system then it can grow over time.

Just one advise from my personal experience:

Don't build it yourself.

This part of your company seems to be growing, and a number of 60 field
engineers is already considerable.
So, processes will change, managers will ask for specific KPIs, the need
will come up to dispatch engineers based on their location/skillset etc.
etc.
You might run into technical problems with the client appliances
(compability) if you don't do extensive testing.
So, soon a commercial application might not be overkill anymore, but you
already have a Big Ball of Mud sitting there.
( http://www.laputan.org/mud/mud.html#PiecemealGrowth )

All in all you should at least do a careful cost comparison between
commercial off-the-shelf software (COTS) and inhouse development for
the next 3 to 5 years.
Don't understimate testing and training costs and the change of your
business.

I could see two alternatives:
1. Buy a COTS package which has the big additional benefit of being
implemented for some tried and tested processes, so that your company
doesn't have to reeinvent their processes from scratch (it worked with
SAP, didn't it )
2. Go the super cheap route, like buying a blackberry package from a
local mobile provider and send the jobs to the engineers via email.
The neat thing is that it pushes out the mails to the clients.
Maybe there are also some ready made packages to implement some forms on
the blackberry's email client.
That way, you have the alternative of cheaply learning how things go in
practice, and refine the requirements.

 

[Martin P. Hellwig]

That seems like overkill for this application.


The physical connectivity isn't your worry. How you access it from
software depends on your chosen platform.


I've gone this route before, and this is why I say your .net solution
looks like overkill. The phones that GSM providers *give away* have
web browsers built into them, and it's been that way for years. No
need for .net or Pocket PC or whatever on the mobile device - just get
a web-enabled phone. This makes half the development trivial - you
don't have to do *any* development on the mobile side of things.

<cut>
Somehow I immediately thought of the s60 platform, it even runs some
form of python if you really need it to ;-)
http://www.s60.com/

 

[limeydrink]

In response to Mike's post...

I know exactly where you're coming from and you are right a web based
solution is the simplest and would be the fastest to develop and
rollout etc. but..

The cost is in the data, in the uk you get charged for the amount of
data you send/receive by GPRS and this data you are paying for would
ideally be useful data and not the HTML to present it.

Engineers generally get the job information then fill in bits as they
go along, and also reveiew what has been done on the job at various
times throughout their work, this could mean alot of traffic moving
through data screens in a web based application.

The push capability of a full client could also be very useful.

The pocket PC hardware is very cheap, and this platform would provide a
good user experience with regards to screen size and means of inputting
the data.



In response to Ernst...

There hasn't been a full decision by management to spend lots of cash
on this as yet hence why I am making this sort of a pet project for my
self and spending mostly my own time, we will then hopefully trial it
with a few engineers and then see where it goes, you are right though,
the company ideally should invest in a full solution as in my opinion
there could be massive savings to the company both in money and time
and also improved service to customers, problem is management can take
a lot of convincing and they don't like spending money.

It would give me great satisfaction though to roll my own solution to
this and then at least prove the benefits to the company, even if just
to convince them to purchase a system from a vendor, and even if that
happens i will have learnt a lot in the process.

Thanks for your replies, they are much appreciated.

 

[Mike Meyer]

In response to Mike's post...

I know exactly where you're coming from and you are right a web based
solution is the simplest and would be the fastest to develop and
rollout etc. but..

The cost is in the data, in the uk you get charged for the amount of
data you send/receive by GPRS and this data you are paying for would
ideally be useful data and not the HTML to present it.

I'll take your word for it that you don't have GPRS providers in the
UK that have an all-you-can eat plan. They all do in the US.

However, HTML doesn't *have* to be the bloated crap that seems to be
produced by most designers - more more accurately, by the GUI tools
they use to produce HTML. Used properly, it can be a very sparse
encoding system. Not as good something designed for the data at hand,
and not as good a binary format, but not awful, either.

Further, you may be able to save bandwidth by putting some of the HTML
on the mobile device. For example, the form your field engineers fill
in to note that a job has been done could be kept on the PDA (assuming
it's static), and just opened in the browser, filled in, and
submitted. Palm did this with their PQA technology - a PQA was a set
of HTML files that were "compiled" into a representation their browser
understood.

Finally, if the data bandwidth is really a serious problem, you might
want to look into a solution at a different level. For instance, if
you're using a VPN to get to your internal network, some VPN
technologies include facilities to compress the data traveling across
the network. That will reduce the difference in bandwidth usage
between the various encoding formats you might consider.

<mike

 

[limeydrink]

Bump

 

[Frithiof Andreas Jensen]

It would give me great satisfaction though to roll my own solution to
this

Ahh - a bright young employee ready to meet the realities of corporate life,
much like a bike rider meets the concrete he drives on ;-)

and then at least prove the benefits to the company, even if just
to convince them to purchase a system from a vendor, and even if that
happens i will have learnt a lot in the process.

"Company" will not give a rats arse - really, they will not!!

Benefits to *management* is what this game is about; If the implementation
provides "management" with good material for powerpoint slides that they can
use to impress their betters and move up a notch then, obviously, some of
the blessings will shine upon you too - if it causes management grief, then
you will be made to suffer.

"What can the project provide in the way of opportunity to Management?" is
what you should look into. Opportunity for "company" is in an entirely
different stratosfhere. Mere employees shall not try to second-guess the
Gods.

Bragging rights are allways good - but keeping the level of trouble below
the upper layers of management attention level is a *much, much* better
career move. *I* would certainly go for the primitive, but almost
certain-to-work, Web solution rather than being clever and having to deal
with the weirdness on how f.ex. sockets happen to be implemented on the
mobile device and the 2 year's to obsolensence of the hardware+os (Those
mobile devices *are* in general proprietary and deviant from RFC standards
in unpublished but painful ways).

I would also have some people that knows how to integrate with the existing
Management Information System do it too. Usual business rules apply: ssume
responsibility if it goes well, blame contractors if it does not. That sort
of thing.

.... Ignore the "cost issue" - if management think it is worth it to them
they buy the project whatever the actual price or actual benefit - after
all, it is only money for shareholders and tax people they spend anyway!!!
And if they do not, then the work you do not have to do is the easiest.

Thanks for your replies, they are much appreciated.

Nah - probably not (I am getting too old for this business and I *still*
have 20 years to go ;-)
But in 20 years you will say the same. And whoever you say it to will go
ahead with his own thing anyway.

 

[Cameron Laird]

I'll take your word for it that you don't have GPRS providers in the
UK that have an all-you-can eat plan. They all do in the US.

.
.
.
Also, in at least some parts of Europe, carriers nominally charge
for the Web, but it's possible to browse effectively--to reach
nearly all the same pages--with an alternative protocol to "http:"
(the list of which I've now utterly, if perhaps temporarily,
forgotten).

 

[limeydrink]

Hello Mike,

Thanks for the response.

And yeah, you're right, after a bit of investigation I found you can
pay for an unlimited mobile data usage and it's not that expensive so
thanks for prompting me.

Ok then, web it is, just wondering how you can test the web application
while developing and how do you find out the capabilities of the
browser, I now have browsed WAP pages on my Windows smartphone, is
there any difference to developing a WAP/WML solution to a HTML -
obviously it's in a different markup language, but is there anything
else that is needed between the client and the webserver ?

I have only ever played at web development in the past and I used MS
Active Server Pages and this seemed pretty straightforward, I know you
can use Python as the scripting language for ASP but I want something
platform neutral, so I was wondering if anyone could suggest some
similar framework for Python on both Linux and Windows.

Thanks again.

 

[Mike Meyer]

Ok then, web it is, just wondering how you can test the web application
while developing and how do you find out the capabilities of the
browser, I now have browsed WAP pages on my Windows smartphone, is
there any difference to developing a WAP/WML solution to a HTML -
obviously it's in a different markup language, but is there anything
else that is needed between the client and the webserver ?

I wouldn't bother with WAP/WML these days. It's designed for *really*
small devices, like pagers and 2nd millenium cell phones. Unless, of
course, something else dictates it's use.

You found out the capabilities of the browser by reading the browser
documentation. That should specify which version of
HTML/XHTML/CSS/DOM/etc. it supports. I wouldn't worry to much about
finding a browser that supports exactly the same set of
documents. Find one that supports at least those, then use that. I
like FireFox, because the available extensions include lots of things
that are useful for web developers. Make sure you only use features
that are documented as supported by the target browser. You'll have
problems with bugs in FireFox for some things, and then when you get
around to deployment you'll have problems with bugs in your target
browser. The further you stay away from advanced/esoteric features,
the less trouble you'll have.

I have only ever played at web development in the past and I used MS
Active Server Pages and this seemed pretty straightforward, I know you
can use Python as the scripting language for ASP but I want something
platform neutral, so I was wondering if anyone could suggest some
similar framework for Python on both Linux and Windows.

And with that, you open a can of worms. There are so many choices
available for Python that people complain there are to many. The
closest thing to ASP is PSP. There are a number of version of that;
I'd use the one bundled with recent versions of mod_python for Apache
if you want that. Personally, I prefer to Cheetah Templates to PSP -
it plays better in the OO world, and I can use intelligent
HTML/XML/SGML editors on the files. You can also "precompile" cheetah
templates to python, and just put on the server; I'm not sure you can
do that with mod_python's PSP. There are other choices at that level,
and there are also far more integrated solutions available as
well. You may have to byte the bullet and evaluate them.

<mike

 

[Dave Benjamin]

... Personally, I prefer to Cheetah Templates to PSP - it plays better
in the OO world, and I can use intelligent HTML/XML/SGML editors on the
files. ...

Thanks for reminding me of Cheetah. I am trying out TurboGears on a
small project, and haven't gotten to the HTML part yet... I was going to
try and learn Kid, but I'm in a bit of a hurry, and Cheetah's much closer
to what I'm used to. So, I decided to search Google for TurboGears and
cheetah, and guess what I found:

TurboGears now supports Cheetah and Stan
http://www.blueskyonmars.com/2006/01/01/turbogears-now-supports-cheetah-and-stan/

Happy New Year!
Dave

 

[David Nemeth]

Ok then, web it is,

One weakness in the web architecture is retrieving and sending data
when there is no wireless signal (will your techs be working in
basements, for example?) Some browsers do support "working offline",
but I don't know if that's adequate for this application.

There may be an easy solution to this with a web based architecture,
but it's an aspect of the system design to keep in mind.

 

[limeydrink]

All these web technologies I don't know where to start, have you got
any suggestions for getting started in the world of web development,
books maybe ?

Once again thanks for your help

 

[Paul Rubin]

All these web technologies I don't know where to start, have you got
any suggestions for getting started in the world of web development,
books maybe ?

This is out of date but may help get started.
http://philip.greenspun.com/panda/

 

[Mike Meyer]

All these web technologies I don't know where to start, have you got
any suggestions for getting started in the world of web development,
books maybe ?

Sorry, but I can't recommend books on this. I learned it before the
books existed, so what I learned it from won't help.

Since you're approching this as a programmer, I think it'd be useful
to understand the underlying mechanims. Get a vanilla apache install,
and try writinng a few simple cgi scripts. Do at least one or two
without the cgi module, so you know how the data actually gets to your
program.

With that knowledge in hand, you'll be able to better evaluate the
other solutions. You'll have some understanding of what needs to
happen, and will be able to see how those packages help - or hinder -
your doing it. It may be that what you're doing is straightforward
enough that the cgi module will do the job. Some things don't even
need that, but it doesn't sound like what you have in mind is that
simple. But having done a few things, you'll know enough to decide
that yourself.

<mike