Designing a Pythonic search DSL for SQL and NoSQL databases

[Alec Taylor]

Dear Python community,

I am analysing designing an abstraction layer over a select few NoSQL
and SQL databases.

Specifically:

- Redis, Neo4j, MongoDB, CouchDB
- PostgreSQL

Being inexperienced; it is hard to know a nice way of abstracting search.

For conciseness in my explanation, think of `Table` as being table,
object, entity or key; and `name` as being name or type.

Maybe `res = Table.name.search(<attr_list with conditions>)`

Or on multiple `Table`:
`res = AbstractDB().AbstractSearch(<Table.attr_list with conditions>
<conditions> <Table.attr_list with conditions>)`

Then: `res.paginate(limit=25, offset=5)`

Or if you want all: `res.all()`

And additionally borrow/alias from a relevant subset of PEP249, e.g.:
`fetchone` and `fetchmany`

Will open-source this once it's of sufficient functionality.

What do you think?

Advice on better design; or just feedback on what I've mentioned?

Thanks,

Alec Taylor

PS: I am using Bottle. You'll note that some of the syntax I've
references is inspired from web2py's DAL and ODMG.

 

[Roy Smith]

Dear Python community,

I am analysing designing an abstraction layer over a select few NoSQL
and SQL databases.

Specifically:

- Redis, Neo4j, MongoDB, CouchDB
- PostgreSQL

This isn't really a Python question.

Before you even begin to think about "how to do this in Python", you
need to think about "how to do this at all". You've got a huge range of
storage paradigms there. Redis is basically a key-value store. Mongo
does documents. Postgres does relations. I'm not familiar with Neo and
Couch, but I assume they also have their own interesting ways of storing
things.

You need to figure out what it means to abstract search over such a
diverse range of technologies. I honestly don't think it's possible,
but maybe you've got some good ideas. In any case, you need to figure
that part out before you even begin to think about how to implement it
in any particular language.

 

[Alec Taylor]

Hmm, looking at the capabilities of Redis; you're likely right.

Would welcome discussion on the implementability and usefulness of a
central search abstraction to the other stores mentioned.

One thing could be that using the paradigm the database was made for
is better than any advantages abstracting their differences creates.