python24.zip

[Scott David Daniels]

I'll bet this means that the 'zope.zip', 'python24.zip' would drop
you to "about 12500 - 10000 = 2500" failing opens. That should be
an easy test: sys.path.insert(0, 'zope.zip') or whatever.
If that works and you want to drop even more, make a copy of zope.zip,
update it with python24.zip, and call the result python24.zip.

--Scott David Daniels
(e-mail address removed)

 

[Dieter Maurer]

... 10.000 failing opens -- a cause for significant IO during startup ? ...

So I would agree that IO makes a significant part of startup, but
I doubt it is directory reading (unless perhaps you have an
absent NFS server or some such).

We noticed the large difference between warm and cold start even when
we run from a zip archive. We expected that the only relevant IO would
go to the zip archives and therefore, we preloaded them to the
OS cache (by reading them sequentially) before the Python start.
To our great surprise, this did not significantly reduced Python's
(cold) startup time. We concluded that there must be other IO
not directed to the zip archives, started investigating and found
the 10.000 opens to non-existing files as the only other
significant IO contingent....


Dieter

 

[Dieter Maurer]

...
I'll bet this means that the 'zope.zip', 'python24.zip' would drop
you to "about 12500 - 10000 = 2500" failing opens. That should be
an easy test: sys.path.insert(0, 'zope.zip') or whatever.
If that works and you want to drop even more, make a copy of zope.zip,
update it with python24.zip, and call the result python24.zip.

We can not significantly reduce the amount of opens further:

Each module import from a zip archive opens the archive.
As we have about 2.500 modules, we will get this order of opens
(as long as we use Python's "zipimporter").

The "zipimporter" uses a sequence of "stat"s to determine
whether it can handle a path item: it drops the last
component until it gets an existing file object and then
checks that it is indeed a zip archive.
Adding a cache for this check could save an additional few
hundreds of opens.


Dieter

 

[Steve Holden]

...
Indeed I have written PEP 302-based code to import from a relational
database, but I still don't believe there's any satisfactory way to
have [such a hooked import mechanism] be a first-class component of an
architecture that specifically requires an os.py to exist in the file
store during initialization.


I wasn't asking for an import hook mechanism (since I already knew
these to exist), but for a way to allow such mechanisms to be the sole
import support for certain implementations.

We do not have "os.py" (directly) on the file system.
It lives (like everything else) in a zip archive.

This works because the "zipimporter" is put on
"sys.path_hook" before the interpreter starts executing Python code.

Thus, all you have to do: use a different Python startup
and ensure that you special importer (able to import e.g. "os")
is already set up, before you start executing Python code.

It might help others like me if you were to document this setup, as I
was unable to persuade the interpreter to start without producing the
dire-sounding warning messages I mentioned in the bug report.

regards
Steve

 

[Steve Holden]

Now I remember what makes this stuff really difficult: PEP 302
introduces path hooks (sys.path_hooks), allowing imports from
other sources than files. So the items on sys.path don't have
to be directory or file names at all, and importing from them
may still succeed if though stat fails.

This new feature also makes the strategy of looking in the filestore for
"os.py" somewhat dubious, hence my bug report.

regards
Steve