Request for suggesstions and comments

[Vivek Kumar]

Hi all,

I have to write a network server (sort of) and I am
looking for your valuable comments. Currently I have
written a prototype in VB6 but it can only handle up
to 30 or so clients. I need to upgrade the application
so that it can handle up to 1000-1500 clients at a
time (later if every thing works fine then hoping for
5000-6000 client). The scenario is as follows.

1. An server application (above mentioned app) will be
listening on a particular port for client connection.
The client will be connecting to it using TCP
protocol.
2. The server will be continuously sending a trigger
pulse to all the clients connected on this port (say
every 3 sec).
3. Each client (in my case another app written in VB6)
will be sending a string of approx. 70-75 chars every
1 sec.
4. Each string will contain a start and an end char.

5. Each string will contain some data eg. first 4 char
acc. id, next 10 char user name etc.

6. As the data may come in small chunks (due to the
nature of network and buffering), it will be the
server's responsibility to buffer incoming data and
look for valid strings in buffer.

7. Once the server software find a valid string in
buffer, it should extract the valid string from the
buffer and parse it (extract the data in the string)
and insert into a database server with timestamps (in
my case SQL Server 2k).

8. The server will also have to send the valid string
to two other ports (one for showing status in GUI) and
second to a remote client which will further parse the
data and do some processing on the data (currently
written in VB). There can be around 20-30 such
clients. For GUI there will be around 5-10 clients.

9. The remote clients to whom the valid string is
being sent can also send some data to be sent back to
the clients from whom we are receiving data per sec.
For this the remote client will send data to the
server and the servers responsibility will be again to
parse this data and send it to the appropriate client
on the other side. This string can also be intended
for broadcasting, in that case we may need to send
data to all the clients on other side.

This is a general view for the application. Now here
are the issues on which I am requesting your comments
and suggestions.

Q. Is it possible to write an application for this
kind of server activity in Python? I mean whether
Python will be suitable for this kind of high activity
load, real time app?

Q. How much time it can take (approx.) to develop such
kind of server?

Q. Do we need multi threading? If so, then whether the
Socket modules will be safe to use with multi
threading?

Q. What modules for networking (socket ?), database
access etc should I use?

Currently as I am just planning about the solutions so
I don't have much of questions in my mind. But if it
is possible to write this kind of app in Python easily
(relative to C family of language , then surely I
will have some more questions for experts out there
..

As for me, I have written some small scripts in
Python. I have used Tk for GUI, PostgreSQL and Mysql
for my Database, some CGI programming with apache,
reporting using reportlab. But that was a long time
back (some 2 yrs back). After that I joined one
organization working on M$ platforms. So, since then I
am working in VB6 only. But I am confident that after
7-10 days of study I can again start working with
Python (Which is of course my favorite language .

Eagerly waiting for your comments and suggestions.

TIA,
Vivek Kumar




__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

 

[akameswaran]

Hi, the advice is free, so tkae it for what it's worth.
Q. Is it possible to write an application for this
kind of server activity in Python? I mean whether
Python will be suitable for this kind of high activity
load, real time app? - Absolutely, Look at Zope or Cheetah for
examples of fairly large
python based app servers. That doesn't mean something written quickly
from scratch will work well.

Q. How much time it can take (approx.) to develop such
kind of server?
Very hard to say, I've written similar server code for a peer to peer
tool - haven't touched the server portion in a while - and I learned a
lot more about how python threads really work doing it. complete
half-assed estimate - min 1 week, max 1 month.

Q. Do we need multi threading? If so, then whether the
Socket modules will be safe to use with multi
threading?
Q. What modules for networking (socket ?), database
access etc should I use?
This is an answer for both questions above, yes you'll need
multithreading. Best idiom, each client gets a thread. and whatever
else you need(your pinger thread) The socket modules are not entirely
thread safe. I don't remember the problems, but they were there, and
relatively solvable, if I recall corretly, most issues were
non-critical in that they resulted in a bad shutdown, but didn't impact
the server while running.. As for modules, I worked most with
TCPSocketServer(?? is this the right name? bad bad memory - don't have
the code here But I think you for this it may be easier to use the
XMLRPC server. A lot more is done for you than the more primative
SocketServers. This is a particularly good route to go, if you know
XMLRPC already, or plan to add browser only clients in the future.

And on another note, welcome to Python - I switched from java, as my
most productive language, after visiting a client. They needed a tool
built in a hurry. Some ugly XML parsing that I knew would take about a
week to write. Long story short, my first day with Python I wrote a
functional version of the XML code in 3 hours, and got a deliverable
version of the code written the next day. And one more hint- I just
switched from the TK based tKinter to wxPython - and I don't think I'm
going back.

Hope this helps.

 

[Magnus Lycka]

Hi all,

I have to write a network server (sort of) and I am
looking for your valuable comments.

You might want to look at twisted for this.
http://twistedmatrix.com/