ANN: pythonfs - create linux filesystems in python

[David McNab]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

I've just built a framework which allows mountable linux filesystems to
be implemented in pure python.

It consists of a back-end to LUFS (lufs.sf.net), and is available from
http://www.freenet.org.nz/python/lufs-python

Simple to use - after adding the pythonfs code to LUFS, and
building/installing, one only needs to copy the template.py python
module and flesh out the methods in the Filesystem class. Generous
docstrings spell out exactly what is needed.

Have phun

Cheers
David

- --

leave this line intact so your email gets through my junk mail filter
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQE/1GkxUuEuev/PjrcRArSzAJ4gbHZtRN6Di3I/sD/9cCk0m/86igCeLFvi
grZoxZILdpugnNiuwb2fW5E=
=fTSI
-----END PGP SIGNATURE-----

 

[Rene Pijlman]

David McNab:

a framework which allows mountable linux filesystems to be
implemented in pure python.

Serious question: why?

 

[David McNab]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Rene Pijlman wrote:
|>a framework which allows mountable linux filesystems to be
|>implemented in pure python.
|
| Serious question: why?

Serious answer - why *not*?

Basically, the 'file/directory' tree paradigm comes in handy in a lot of
different ways. In fact, one of the original Unix claims to fame was and
is the 'everything is a file' concept.

If you look at the LUFS site (lufs.sf.net), you'll see a bunch of
experimental filesystems, including (of all things!) gnutellafs.

So once someone has decided to offer a filesystem interface for
something, it can be an immense boost to be able to implement the thing
in Python, which on average takes a fraction of the time it takes to
write similar in C.

When first learning Python, I had a dramatic experience of Python's
leverage, when I had to implement the same set of functions in Python
and C. They took about 20 minutes to implement and debug in Python, but
over 2 hours in C (even though I worked as a senior C developer for 10
years).

- --

Kind regards
David

- --

leave this line intact so your email gets through my junk mail filter
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQE/1HoWUuEuev/PjrcRAiwwAJ9+7hRZsKvZyuAj8dXZe6iyEA0d5ACeLNwb
zKw6xalCBihdocAHtZeQi3s=
=mBdk
-----END PGP SIGNATURE-----

 

[Alan Kennedy]

[David McNab]
[Rene Pijlman]

Serious question: why?

One very serious usage that springs to mind is to implement a
versioned virtual file system, in combination with a version control
system like subversion (subversion.tigris.org).

ClearCase (the versioning system) has the concept of such virtual file
systems. You specify a view of the "Versioned Object Base" (VOB): the
view is specified using dates/labels/revision numbers/etc, and is
specified inside a "config spec" file. [1]

Once you have specified your configspec, ClearCase creates a virtual
file system, in which every file and directory actually resides inside
the ClearCase VOB, so you are actually working directly on versioned
files, even though your filesystem just looks like a plain ordinary
file system: no special tools or techniques required.

Subversion supports python fantastically well:-

o Python scripts can be executed pre- and post-commit[2]
o Python interfaces to all the important subversion libraries are
available, through SWIG[3]

It seems to me that David's work could be the last link in the chain
to creating a complete VFS-based versioning system, using python and
subversion.

And that's just one idea that occurred to me: I'm sure that there are
*plenty* of others.

Nice one David!

regards,

--
alan kennedy
------------------------------------------------------
check http headers here: http://xhaus.com/headers
email alan: http://xhaus.com/contact/alan

[1]
http://www.agsrhichome.bnl.gov/Controls/doc/ClearCaseEnv/v4.0doc/cpf_4.0/ccase_ux/ccdev/ccdev-6.html

[2]
http://svnbook.red-bean.com/html-chunk/ch05s02.html#svn-ch-5-sect-2.1

[3]
http://svnbook.red-bean.com/html-chunk/ch07s02.html#svn-ch-7-sect-2.3

 

[Paul McGuire]

Here's another idea, piggybacking on the /proc filesystem:

Use this pseudo-filesystem to create a query front-end to a database, using
a file directory application metaphor. The database could contain
information on say patients in a hospital, or lots or equipment in a
factory, or active claims in an insurance office. Then under the main
directory, each claim could have a directory of its own, with sub-files
containing a summary text description, a subdirectory of claim event
"files," and so on. You could also have directories that organize claims by
category or status, with "links" to the actual claim directories in the
associated category directories. All of this with the GUI of a simple file
browser (or a real browser!), and very flexible. If you find out that users
often want to know about claims over a certain amount, or that have remained
uncleared longer than a certain threshold period, just add another
"directory."

In a directory tree, it might look like:
/claims
/claimsByStatus
/active
/pendingEstimates
/pendingMedical
/pendingAdjuster
/pendingLegal
/closed
/contested
/claimsByAge
/claimsByAmount
/allClaimsByClaimant
/A_G
/H_M
...

and so on. In reality there are no "files" or "directories," but this
filesystem utility makes it look like there are.

Or maybe just use this as a flexible rapid prototyping or requirements
gathering technique - then once the structure stabilizes, build into an
application GUI with menus (maybe leave the filesystem around for "expert"
mode).

-- Paul

 

[Skip Montanaro]

Paul> Here's another idea, piggybacking on the /proc filesystem:
...

Here's another idea. Provide a /proc filesystem on systems which don't
support one out of the box.

Skip

 

[Tim Hoffman]

You could access ZODB as a filesystem ;-)

David McNab:



Serious question: why?

 

[Andrew Wilkinson]

David McNab:

Serious question: why?

Or how about a cvs backend to the file system?

*drools* ;-)

 

[Terry Reedy]

David McNab:

Serious question: why?

If I wanted to learn about or experiment with linux filesystems, I might do
better to start with this than track down the actual Linux C sources.

TJR

 

[Peter Hansen]

David McNab:

Serious question: why?

Just thought of another possibility: might be useful for doing
unit testing where one could use the Python-based filesystem to
mock a real filesystem, but simulate certain particularly difficult
kinds of errors (corruption, files that go missing, invalid file names)
that might happen in extreme situations.

-Peter

 

[Nick Vargish]

Serious question: why?

How about providing a selection of ways of accessing an mp3
collection? At the top level you could have /artist, /album, /genre,
/year...

Nick

 

[Christos TZOTZIOY Georgiou]

How about providing a selection of ways of accessing an mp3
collection? At the top level you could have /artist, /album, /genre,
/year...

And suppose the mp3 files are spread over many computers in the local
network... perhaps the vfs could substitute a file of momentary silence
for every song not available at the moment, so that you never have to
change the playlist...

 

[Mike Rovner]

How about providing a selection of ways of accessing an mp3
collection? At the top level you could have /artist, /album, /genre,
/year...

It reminds me of Microsoft's promise to combine RDBMS and file system in a
new Longhorn OS. Several years ago that idea was highly popular but nothing
came out.

That approach may be very interesting to allow user to use some plug-ins for
the file system
(access, security, notification, etc.).

Mike

 

[Tim Roberts]

Paul> Here's another idea, piggybacking on the /proc filesystem:
...

Here's another idea. Provide a /proc filesystem on systems which don't
support one out of the box.

Yes, but his pythonfs product is extremely Linux-centric, where the /proc
filesystem is ALWAYS available.

 

[Paul Boddie]

It reminds me of Microsoft's promise to combine RDBMS and file system in a
new Longhorn OS. Several years ago that idea was highly popular but nothing
came out.

Longhorn isn't supposed to be out before 2006, unless you live in
Malaysia (apparently). Is this a message from the future? ;-)

Actually, I can imagine that it isn't quite equivalent to the
much-hyped Longhorn features and the GNOME Storage stuff, since a
filesystem typically provides a fairly rigid naming interface,
although I'd admit that the same kinds of naming issues superficially
exist with Web applications, and you can always disregard the
hierarchical "assumptions" there and interpret the query string in
your own way. One thing I dislike about various Web frameworks is that
they don't really let you disregard the directory-like nature of the
namespace, forcing you into mod_rewrite and other magic to "name
things your own way".

That approach may be very interesting to allow user to use some plug-ins for
the file system (access, security, notification, etc.).

What about doing the kinds of things that KDE's IOSlaves and GNOME's
VFS do, but at a lower layer in the system (and in a sane language, of
course)? Python lets you do HTTP, FTP and loads of other protocols
fairly trivially - imagine hooking that up to a filesystem interface.

Paul

 

[David M. Wilson]

It reminds me of Microsoft's promise to combine RDBMS and file system in a
new Longhorn OS. Several years ago that idea was highly popular but nothing
came out.

That approach may be very interesting to allow user to use some plug-ins for
the file system
(access, security, notification, etc.).

Quite a few people have attempted work in this area, but as with
Microsoft, the plan to combine a traditional database and a
traditional filesystem never seems to come to fruition.

If you are interested in this sort of stuff, the only really serious
attempt I have seen in the recent years is the work of Hans Reiser, of
reiserfs fame. Take a read through the pages at:

http://www.namesys.com/


And possibly pick the thread up over on one of the database or
filesystem news groups.


David.

 

[John J. Lee]

Just thought of another possibility: might be useful for doing
unit testing where one could use the Python-based filesystem to
mock a real filesystem, but simulate certain particularly difficult
kinds of errors (corruption, files that go missing, invalid file names)
that might happen in extreme situations.

Better to clobber open() than to risk crashing your kernel, though!

Though maybe rebinding open won't remain legal forever...


John

 

[Skip Montanaro]

Paul> Here's another idea, piggybacking on the /proc filesystem:

Tim> Yes, but his pythonfs product is extremely Linux-centric, where the
Tim> /proc filesystem is ALWAYS available.

Whoops, yeah...

Skip

 

[Mike Rovner]

You misquoted me a bit
I ment (expanding the previous author) it can be used to provide "different
views
to the same data" (database) in contrast to the "same view on the different
data" aka filesystem.

Longhorn isn't supposed to be out before 2006, unless you live in
Malaysia (apparently). Is this a message from the future? ;-)

I red 2005. So I live on the same planet at least.

Actually, I can imagine that it isn't quite equivalent to the
much-hyped Longhorn features and the GNOME Storage stuff, since a
filesystem typically provides a fairly rigid naming interface,

No. The whole idea was to invert that. You use familiar approach (and api)
to get to dynamic data (as in /proc).

course)? Python lets you do HTTP, FTP and loads of other protocols
fairly trivially - imagine hooking that up to a filesystem interface.

Right, the whole universe. Don't fix up on my list.

Mike

 

[Mike Rovner]

Microsoft, the plan to combine a traditional database and a
traditional filesystem never seems to come to fruition.

AFAIK Microdoft never come out with revolutionary ideas,
it makes mass-production of already proved concepts and excel in it.
The idea of implementing filesystem interface on top of proved database
pluses like transaction isolation, integrity, changes propagation and
backup/restore
mechanisms may come to something usefull due to them.

If you are interested in this sort of stuff, the only really serious

Not really but thank you for the link.

Mike