Implementing a database engine in javascript?

[aaronwmail-usenet]

Hi folks. I'd really like some comments or flames please .

I've ported my xsdb database engine to javascript. I put up
a demo page here:

http://www.xfeedme.com/xsdbXML/xsdbjs/runqueries.html

The xsdb framework provides a flexible and well defined
infrastructure to allow tabular data to be published,
retrieved, and combined over the Internet.

This demo page allows you to load example xsdb
queries from example files, optionally edit them, run
the queries and examine the results of the query
execution. Both the queries and the responses are
encoded in the xsdb data language -- a dialect of XML.

All live processing is done on the client using javascript.
The server provides static data files only. The ultimate goal
of this effort is to provide powerful and easy data manipulation
capabilities on the client. I've tested on IE and Firefox/win only.

For more information on xsdb please go to
http://xsdb.sourceforge.net
The javascript implementation is technically not released yet,
but you can see the current version at
http://www.xfeedme.com/xsdbXML/xsdbjs/xsdball.js
if you are interested.

Comments?
-- Aaron Watters

====
I'm out of my mind at the moment; please leave a message.
(from a teeshirt)

 

[Bart Van der Donck]

I've ported my xsdb database engine to javascript. I put up
a demo page here:

http://www.xfeedme.com/xsdbXML/xsdbjs/runqueries.html

The xsdb framework provides a flexible and well defined
infrastructure to allow tabular data to be published,
retrieved, and combined over the Internet.

This demo page allows you to load example xsdb
queries from example files, optionally edit them, run
the queries and examine the results of the query
execution. Both the queries and the responses are
encoded in the xsdb data language -- a dialect of XML.

All live processing is done on the client using javascript.
The server provides static data files only. The ultimate goal
of this effort is to provide powerful and easy data manipulation
capabilities on the client. I've tested on IE and Firefox/win only.

For more information on xsdb please go to
http://xsdb.sourceforge.net
The javascript implementation is technically not released yet,
but you can see the current version at
http://www.xfeedme.com/xsdbXML/xsdbjs/xsdball.js
if you are interested.

There are inherent problems with this kind of client side applications.

They require a vast amount of memory from the client. Though your
software may work at acceptable speed on small data, I do not believe
it will be suitable for somewhat larger stuff. XML dialects such as XSL
suffer from the same shortcomings. It's all just too slow and too
limited, and there's not much that can be done about that. Benchmark
your results and compare those too classic RDBMS engines - you'll see a
huge performance difference.

Your software requires huge network traffic too. Queries need their
tables at their finger tips before they can even begin. So every table
needs to be fully transferred first. What about relational stuff like
SQL joins ? Your application would need to get every data file first,
parse it, query it and then output the result... Again, not much can be
done about this either. It's just the way client side queries work.

I think your javascript programme is okay to present/sort/group/parse
small data on the client side. At its best after some (server side)
data filtering method took place. It's like Microsoft's TDC - a pretty
browser thingie but not much more than that. For that purpose, I think
your application could be okay.

 

[aaronwmail-usenet]

re: http://www.xfeedme.com/xsdbXML/xsdbjs/runqueries.html
Thanks. I agree. This approach is not good for large data
sets using javascript on the client.

The intention is to allow processing of larger data sets
on the server if needed, either using one of the other implementations
(java, C#, python) or using a gateway to another archiving
methodology. The client side component is best for simple
manipulations before presentation.

Thanks for the comment! -- Aaron Watters

===
% if I had a ( for every $ Bush spent how many ('s would I have?
too many ('s.
%

 

[Bart Van der Donck]

re: http://www.xfeedme.com/xsdbXML/xsdbjs/runqueries.html
Thanks. I agree. This approach is not good for large data
sets using javascript on the client.

The intention is to allow processing of larger data sets
on the server if needed, either using one of the other implementations
(java, C#, python) or using a gateway to another archiving
methodology. The client side component is best for simple
manipulations before presentation.

Then I wouldn't call it a 'javascript database engine'. Microsoft TDC
doesn't call itself a database engine either, simply because it is no
such thing. Also the word 'powerful' is IMO not at its place.

Not that I don't make them, but the javascript seems to have some bugs.
Quick test on MS IE 6.0.2900 XP SP2:

Load ~/xsdbXML/examples/SupplierParts/query_frame_html > hit 'load
query from file' > hit 'run query' > JS error 'Could not get document
element bad xml?' at line 6875.

Load ~/xsdbXML/examples/SupplierParts/remote_parts_query.xsdb > hit
'load query from file' > hit 'run query' > JS error 'Could not retrieve
' at line 6849. Same error for
~/xsdbXML/examples/SupplierParts/rquery.xsdb.

If the Python/C#/Java implementations have acceptable CPU benchmarks,
then that will make a big difference in their applicability compared to
the javascript version. I've no idea how xsdb compares to, let's say,
some SQL database... xsdb will most likely be much slower - of course
this has also to do with the over-and-back XML conversions.

It would be a good idea to investigate how various (Open Source) data
processing software optimizes the read and query actions, to get it as
sharp as possible. Also, csv2xml or xml2csv optimizers should exist.

Good luck with it.

 

[aaronwmail-usenet]

Thanks again.

Sorry about the 3 buggy data files. I have removed them from the page
http://www.xfeedme.com/xsdbXML/xsdbjs/runqueries.html.
One file was simply in the wrong format (html) and the other 2 tried to
get remote files from another server, which violates javascript
security limitations (but is supported by the other implementations).
These were bugs in the page, not bugs in xsdb.

I will continue to call xsdb a database engine, however, because it has
all of the query functionality of sql. It may be that if you need more
speed you will want to use something else, but for many projects the
xsdb approach may be very convenient and fast enough.

Thanks for the other comments. I will be looking into them.

-- Aaron Watters

===
The best performance enhancement is the transition from
"not working" to "working". -- John Ousterhout.

 

[aaronwmail-usenet]

Thanks! I've fixed the problems you mentioned and added
a bunch of other stuff, including an explanatory essay
http://www.xfeedme.com/why.html and an AJAX zip code completion
example http://www.xfeedme.com/zip/zipcompletions_async.html.
Still under construction of course.

Please let me know if you have more comments!
-- Aaron Watters

===
February is a horrible month which should be avoided whenever possible.
(from a cookie program somewhere)