Python database of plain text editable by notepad or vi

[James Harris]

I am looking to store named pieces of text in a form that can be
edited by a standard editor such as notepad (under Windows) or vi
(under Unix) and then pulled into Python as needed. The usual record
locking and transactions of databases are not required.

Another way to look at it is to treat the separate files as entries in
a dictionary. The file name would be the key and the lines of the file
the value.

Anyone know of a database (with a Python interface) which will allow
text files to be treated as database fields? If not I can just write
it but I thought it best to ask if there was an existing solution
first.

James

 

[jkn]

Kirbybase is one possibility.

http://pypi.python.org/pypi/KirbyBase/1.9


J^n

 

[Jon Clements]

I am looking to store named pieces of text in a form that can be
edited by a standard editor such as notepad (under Windows) or vi
(under Unix) and then pulled into Python as needed. The usual record
locking and transactions of databases are not required.

Another way to look at it is to treat the separate files as entries in
a dictionary. The file name would be the key and the lines of the file
the value.

Anyone know of a database (with a Python interface) which will allow
text files to be treated as database fields? If not I can just write
it but I thought it best to ask if there was an existing solution
first.

James

I could be missing something here, but aren't you basically just
talking about an OS's filesystem?

glob or listdir somewhere, then create a dict using the file contents
would meet your criteria, with very little lines of code -- but I
would be interested to know what the use-case was for this... Is it
read completely at start up time, or if each file contains a large
amount of lines and aren't fixed width (or has no other indexing
support without maintenance), then is a complete sequential-scan
required each time, or do you just tell the user to not update when
running (unless I s'pose something along the lines of a SIGHUP for
config files is applicable).

Sorry, just don't understand why you'd want this.

Jon.

 

[James Harris]

....
I could be missing something here, but aren't you basically just
talking about an OS's filesystem?

For storage, yes. The files would be marked-up text stored in the
filesystem. The "dbms" (I use the term loosely!) would provide access
to them by some full or partial key mechanism yet to be determined.
Read-only access would do - at least for now.

glob or listdir somewhere, then create a dict using the file contents
would meet your criteria, with very little lines of code -- but I
would be interested to know what the use-case was for this... Is it
read completely at start up time, or if each file contains a large
amount of lines and aren't fixed width (or has no other indexing
support without maintenance), then is a complete sequential-scan
required each time, or do you just tell the user to not update when
running (unless I s'pose something along the lines of a SIGHUP for
config files is applicable).

All good questions. For now, at least, the files can be read-only and
I'd want those on disk to be the master copies at all times. If I was
writing it myself I'd probably 'cache' some files in memory and stat
them before use. If newer I would reread the file.

Sorry, just don't understand why you'd want this.

I tried to avoid boring folks with the details. I'm toying with some
ideas for a way to help generate source code (in various languages,
not just Python). If it goes ahead the text files would be mainly
marked-up code snippets - with or without symbols that need to be
replaced.

Rather than write one single monolithic app I thought to split it into
reusable components. One part being data access could perhaps be an
existing database (and I'll take a look at jkn's suggestion).

Think of the database as similar to an associative array stored on
disk. The only difference is I may want to play fast and loose with
the keys in some ways - e.g. check for partial key matches or return a
list of part-matched keys. The language name could be part of the key
but I'd also need to store variants for specific language versions.
I'm not sure yet how it will all pan out. As I say, just throwing
around some ideas.

James

 

[Harishankar]

For storage, yes. The files would be marked-up text stored in the
filesystem. The "dbms" (I use the term loosely!) would provide access
to them by some full or partial key mechanism yet to be determined.
Read-only access would do - at least for now.


All good questions. For now, at least, the files can be read-only and
I'd want those on disk to be the master copies at all times. If I was
writing it myself I'd probably 'cache' some files in memory and stat
them before use. If newer I would reread the file.


I tried to avoid boring folks with the details. I'm toying with some
ideas for a way to help generate source code (in various languages,
not just Python). If it goes ahead the text files would be mainly
marked-up code snippets - with or without symbols that need to be
replaced.

Rather than write one single monolithic app I thought to split it into
reusable components. One part being data access could perhaps be an
existing database (and I'll take a look at jkn's suggestion).

Think of the database as similar to an associative array stored on
disk. The only difference is I may want to play fast and loose with
the keys in some ways - e.g. check for partial key matches or return a
list of part-matched keys. The language name could be part of the key
but I'd also need to store variants for specific language versions.
I'm not sure yet how it will all pan out. As I say, just throwing
around some ideas.

James

I am not sure exactly what you need, but would you consider using
something like ConfigParser module provided by Python? It appears to be
something similar to what you need.

 

[Jon Clements]

For storage, yes. The files would be marked-up text stored in the
filesystem. The "dbms" (I use the term loosely!) would provide access
to them by some full or partial key mechanism yet to be determined.
Read-only access would do - at least for now.


All good questions. For now, at least, the files can be read-only and
I'd want those on disk to be the master copies at all times. If I was
writing it myself I'd probably 'cache' some files in memory and stat
them before use. If newer I would reread the file.

It's hard to bore this group

I tried to avoid boring folks with the details. I'm toying with some
ideas for a way to help generate source code (in various languages,
not just Python). If it goes ahead the text files would be mainly
marked-up code snippets - with or without symbols that need to be
replaced.

Rather than write one single monolithic app I thought to split it into
reusable components. One part being data access could perhaps be an
existing database (and I'll take a look at jkn's suggestion).

Think of the database as similar to an associative array stored on
disk. The only difference is I may want to play fast and loose with
the keys in some ways - e.g. check for partial key matches or return a
list of part-matched keys. The language name could be part of the key
but I'd also need to store variants for specific language versions.
I'm not sure yet how it will all pan out. As I say, just throwing
around some ideas.

James

Thanks for the explanation.

Who admins and, who's editing this data?

I couldn't 100% guarantee that I could modify a text file and always
put the right
delimiter in the right place and remember to escape the relevant chars
(and I'm
probably not the 'average' user).

Any opposition to just putting it in a 'proper' DB, then 'blobbing'
the values?
(or just integrate a procedure/script/function whatever your chosen
RDBMS calls to choose it).
Or in some systems, 'externally referencing'... loads of DB's have
free front-ends,
and there are lots of Python libraries.

I think perhaps, all I'm saying is, I'd choose a different approach.
I'd provide a front-end, rather than choose to re-write the wheel over
DB's.

Be nice to know how you get on, if you'd be so kind?

Cheers,

Jon.

 

[Ethan Furman]

It's hard to bore this group




Thanks for the explanation.

Who admins and, who's editing this data?

I couldn't 100% guarantee that I could modify a text file and always
put the right
delimiter in the right place and remember to escape the relevant chars
(and I'm
probably not the 'average' user).

Any opposition to just putting it in a 'proper' DB, then 'blobbing'
the values?
(or just integrate a procedure/script/function whatever your chosen
RDBMS calls to choose it).
Or in some systems, 'externally referencing'... loads of DB's have
free front-ends,
and there are lots of Python libraries.

I think perhaps, all I'm saying is, I'd choose a different approach.
I'd provide a front-end, rather than choose to re-write the wheel over
DB's.

Just to provide a counter-viewpoint:

The (one) system I have worked with that was like that (program source
files kept in blobs in a database) I absolutely hated. I was stuck with
using the tools provided by the app (which, amazingly enough, I also
hated , and unable to use my own tools because because *my* source
files were _not_ saved as *files*. Okay, venting over.

My point is, if what you are storing is plain ol' source files,
providing a way to directly access them is a good thing. If what you
are storing is a mangled version, the ability to let the user choose any
editor to use is a good thing.

My $0.02.

~Ethan~

 

[James Harris]

....
Thanks for the explanation.

No problem. Thanks for taking an interest!

Who admins and, who's editing this data?

For the app I have in mind a programmer would admin and edit the
files.

I couldn't 100% guarantee that I could modify a text file and always
put the right
delimiter in the right place and remember to escape the relevant chars
(and I'm
probably not the 'average' user).

Apart from any markup each file would be just source code or, maybe,
individual fragments of source code so there would be few delimiters
to get right. And any that were wrong could be detected either by the
code writer or the compiler when used.

Any opposition to just putting it in a 'proper' DB, then 'blobbing'
the values?
(or just integrate a procedure/script/function whatever your chosen
RDBMS calls to choose it).
Or in some systems, 'externally referencing'... loads of DB's have
free front-ends,
and there are lots of Python libraries.

I've thought of that. Like Ethan, though, I'd prefer simple text for
this. Databases, for good reasons, tend to obscure text. For one thing
it makes them more secure but it also makes them less transparent and
harder to examine and edit. For this app simple text files seem to be
the best option at the moment.

I think perhaps, all I'm saying is, I'd choose a different approach.
I'd provide a front-end, rather than choose to re-write the wheel over
DB's.

Agreed. That was my reason for asking the question initially.

Be nice to know how you get on, if you'd be so kind?

I don't know yet if it will be feasible but if I do eventually write
something I'll report back.

James