Python AppStore / Marketplace

[Marcel Luethi]

Dear Community

Now I'm standing here, having this great idea for a brand new rocking
app...
But where do I start? I want it to be multi-platform (Linux, Mac OS X,
Windows). It should be easy to install and upgrade. It should be self-
contained, independent of an already installed Python. And of course -
the world should be able to find it!

So what do I do? I'm studying all the possibilities to make it self-
contained (virtualenv, InstantDjango, PortablePython...), searching
for installers (PyInstaller, EasyInstall, ...), looking into making it
executable on every platform (py2exe, py2app, cx_Freeze, ...),
analyzing all GUI frameworks (wxPython, PyGTK, PyQt, ...),
investigating all hosting providers (Google Code, SourceForge, ...)
and so on and so forth.

This is not easy at all!

Using my iPhone I suddenly realize how easy it is to find applications
in Apple's AppStore. How easy and fast it is to install or de-install
an app. My iPhone even checks in the background if there is an upgrade
which could be installed painlessly.

Then I see VMware's Virtual Appliance Marketplace, where you can
download already pre-configured "appliances" for all kind of stuff.
You can even download a basic appliance, install and configure your
servers and tools - and upload it again, so that others can profit of
your work.

Unfortunately there's nothing like this in the Python world...

My idea: what about having a beefed up Cheeseshop for Python apps and
an accompanying client app kind of iTunes for Python apps?

The server side could be easily implemented with any of the current
web frameworks. It's simply a management platform for the app packages
(descriptions, versions, platforms, licenses, user's comments, number
of downloads, ...) and the package files themselves.
It should be definitely hosted by the PSF I think.

The multi-platform client should be intuitively and elegantly allow
app browsing, downloading and installing those Python apps. In that
respect it is sort of a Linux package manager (Synaptic, YUM, ...).
But this is only the end-user related stuff. Furthermore it should
allow to create new apps (probably based on a previously downloaded
base app), package and upload them again (kind of Google AppEngine
Launcher). Those base packages should include some basic management
framework (for installation and configuration) and hopefully soon
after the release of this platform they will be created in a broad
variety to allow app developers to choose among many Python-version/
platform/GUI/...-combinations.

IMHO an architecture like this would greatly enhance the productivity
of the whole Python community by capitalizing the knowledge of many
Python specialists. I don't have to find/install/configure all the
basic stuff myself (see above) to build my app. But I can concentrate
on my app because I'm standing on the shoulders of giants.

I believe it also would make Python as a great platform more visible
to the world - because there is only one place you have to go...


What do you think???
Is this crazy??? Or simply stupid?
Or is this the way to world domination...? ;-)


Unfortunately I'm not expert enough to build such a system - but if
there is enough interest in the community I gladly would like to help.

Marcel


PS:
Naming is important! "Python AppStore" or "Python App Marketplace" is
nice - but I would prefer something like "Python Bazaar" or "Python
Souq" to emphasize the community aspect.

 

[Michael Torrie]

Unfortunately I have no idea what a "souq" is, so I suspect this may be
linguistically biased against English speakers. Or perhaps I'm just
ignorant.

Nah. Not biased against English speakers. Just biased against the
un-traveled.

 

[Steve Holden]

Nah. Not biased against English speakers. Just biased against the
un-traveled.

Don't think so. I've visited roughly twenty-five difference countries
and traveled I-don't-know-how-many miles, but certainly over a million.
Unless you'd like to redefine "un-traveled" ... ?

But I do (still) speak English (of a kind).

regards
Steve

 

[Rhodri James]

Don't think so. I've visited roughly twenty-five difference countries
and traveled I-don't-know-how-many miles, but certainly over a million.
Unless you'd like to redefine "un-traveled" ... ?

But I do (still) speak English (of a kind).

A souq is a bazaar

Well, close enough anyway. It's another arabic word that translates as
"market" in both the mercantile and financial senses, I believe. Maybe
I've just read too much arabic-themed fiction, but I was surprised not
to find the word in my trusty Chambers.

 

[Richard Brodie]

A souq is a bazaar

Maybe I've just read too much arabic-themed fiction, but I was surprised not
to find the word in my trusty Chambers.

Try under 'souk'. Transliterating to the Roman 'q' seems to have become
popular relatively recently.

 

[Paul Boddie]

Using my iPhone I suddenly realize how easy it is to find applications
in Apple's AppStore. How easy and fast it is to install or de-install
an app. My iPhone even checks in the background if there is an upgrade
which could be installed painlessly.

Yes, but this is just one platform. Can you download these iPhone
applications onto Android devices and use them? How would you package
your Objective-C (or whatever Apple has mandated) software for Android
or Debian or anything else out there?

Then I see VMware's Virtual Appliance Marketplace, where you can
download already pre-configured "appliances" for all kind of stuff.
You can even download a basic appliance, install and configure your
servers and tools - and upload it again, so that others can profit of
your work.

Unfortunately there's nothing like this in the Python world...

That's because you first wrote about platform-specific packaging and
now you're writing about language-specific packaging. And it's not as
if the former area hasn't been well addressed by Free Software,
either, but I suppose that's another discussion to be had.

My idea: what about having a beefed up Cheeseshop for Python apps and
an accompanying client app kind of iTunes for Python apps?

The server side could be easily implemented with any of the current
web frameworks. It's simply a management platform for the app packages
(descriptions, versions, platforms, licenses, user's comments, number
of downloads, ...) and the package files themselves.
It should be definitely hosted by the PSF I think.

Well, what you're proposing risks exactly the same problems that many
other language-specific packaging solutions suffer from: it starts off
with specifying platforms, dependencies (which you didn't mention),
download files, and ends up with the user downloading huge amounts of
stuff that may already be on their system, mostly because such
solutions assume that the system doesn't really know anything about
what's already installed. Consequences of this may include a lack of
decent system-related tools to manage the result of installing stuff:
it would be like your iPhone application being installable on Android
by being dumped into the filesystem with little regard for existing
content and with no obvious strategy for safely removing the
application later.

It's true that there are other application repositories similar to the
VMware service you mention which don't really care about dependencies
and just offer self-contained applications for reasons of catering to
the lowest common denominator, and perhaps there's a desire to have
such services out there, but then again, doing such stuff specifically
for Python will lead you into making self-contained applications using
those "installer" technologies, or at least mapping and providing non-
Python library dependencies across multiple platforms. Or you could
just make it easier for Python projects to provide self-contained
applications based on VMware or Free Software alternatives - at least
that would allow you to focus on one platform, although the result
wouldn't be regarded as a "fully integrated" cross-platform approach
because a virtual machine would be involved.

The multi-platform client should be intuitively and elegantly allow
app browsing, downloading and installing those Python apps. In that
respect it is sort of a Linux package manager (Synaptic, YUM, ...).

Why wouldn't you make use of existing system package management, then?

But this is only the end-user related stuff. Furthermore it should
allow to create new apps (probably based on a previously downloaded
base app), package and upload them again (kind of Google AppEngine
Launcher). Those base packages should include some basic management
framework (for installation and configuration) and hopefully soon
after the release of this platform they will be created in a broad
variety to allow app developers to choose among many Python-version/
platform/GUI/...-combinations.

Well, system packages of Python software typically make some use of
distutils to put files in the right places before bundling those files
together. The resulting system packages then work within a "basic
management framework", although I accept that there typically isn't
much support for trivial repackaging, although it isn't that difficult
to repackage, say, a Debian package and to publish it in your own
repository.

IMHO an architecture like this would greatly enhance the productivity
of the whole Python community by capitalizing the knowledge of many
Python specialists. I don't have to find/install/configure all the
basic stuff myself (see above) to build my app. But I can concentrate
on my app because I'm standing on the shoulders of giants.

It isn't hard to find/install/configure the basic stuff in most GNU/
Linux distributions, either.

I believe it also would make Python as a great platform more visible
to the world - because there is only one place you have to go...

What do you think???
Is this crazy??? Or simply stupid?
Or is this the way to world domination...? ;-)

Some people - usually Windows users - seem to like the idea of a
Python-specific solution, whereas others look to be able to administer
packages of many different kinds of software on their own system, and
in some environments people look to be able to administer packages for
many different kinds of system, at which point the Python-specific
interest becomes very small indeed. Generally, one has to choose one
of these levels and work mostly at that level when dealing with
packages, and I would argue that the best level is that of the system,
making language-specific solutions fit inside the facilities (and
restrictions) of the framework provided at the system level.

I think there is some merit in allowing people with a Python interest
to follow new developments (as can be done somewhat with the Package
Index) and for references to system-specific packages to be
incorporated into such a service. At one point I started writing a Web
application to do this, but instead of focusing on manual editing of
package details, it would be better to link the package records to
other services which make the actual system packages available. There
could also be unofficial build systems which provide such system
packages ahead of the usual operating system distribution release
schedule that people often complain about (in that they don't get the
latest stuff quickly enough).

For example, you'd have a package record like this:

Name: XYZ
Platforms:
Debian: http://debian.package.server/python-xyz
Windows: http://windows.package.server/python-xyz

Maybe you'd delegate the versioning and architecture availability to
the specific platforms.

Unfortunately I'm not expert enough to build such a system - but if
there is enough interest in the community I gladly would like to help.

There is ongoing work on packaging-related matters. Perhaps you should
subscribe to the catalog-sig or distutils-sig mailing lists (see
mail.python.org for details). That said, I'm skeptical about language-
specific packaging precisely because of the reasons given above, and
to make my own Python libraries and applications work within the
Debian mechanisms is something I've learned to do. Since that is all
that really matters to me, I don't see that much benefit in learning
something more sophisticated for Python which does less in my own
environment, which I consider to be the operating system and not the
language technology.

Paul

 

[ajaksu]

And the "multiplatform client" that should "easily and elegantly allow
app browsing, downloading and installing those apps" would presumably
have to account for the many differences in package formats and install
requirements between the different platforms.

And then you'd also need to re-download the app if you need it on a
(sometimes slightly) different platform.

Unless the Souq can beat Apple in its own game, by offering
Multiversal Binaries that work across hardware platforms, OSes and
their versions, GUI frameworks, Python versions and universes.
Settling for Downright Obese Binaries if supporting different
universes is considered overkill would be OK too.

shoulda-include-a-single-asset-8-by-8-shotgun-cross-compiler-ly y'rs
Daniel

 

[Steve Holden]

Hi Steve

[I've probably replied to this once already, but here's another try]

I really appreciate your feedback!
Certainly I'm no expert "for the many differences in package formats and
install requirements between the different platforms".

But let me explain a bit more:
I don't have the idea that the multi-platform client should handle all
the different package formats - instead it should define a new one: a
simple zip-file.

OK, but have you asked yourself why the "simple zip file" isn't the
current basis of any Python distribution format?

For me "self-contained, independent of an already installed Python"
means that there is a complete, isolated Python environment for each app
(see virtualenv). So Python would go into APP/lib/..., APP/bin/... and
the app itself into APP/app/... Certainly there would be also some more
files like the app icons and a description file APP/app.xml
The packaged app is a zip/gzip of the whole APP directory. Installing is
simply unpacking to a defined directory (into something like
PythonApps/APP) and setting it up as an os app.

This might work. I have for a while now advocated that each Python
application should carry its own interpreter and all other dependencies
with it. This doesn't solve all problems, though: imagine two apps with
different versions of the same library using incompatible media formats ...

Because there is a complete Python environment for each app (disk space
and bandwidth are cheap nowadays!) and all 3rd party libs are
"integrated" there shouldn't be any other local dependencies. So there
could be APP1 with a Python 2.5.2 + a wxPython/wxWidgets and another
APP2 with Python 2.6.1 + PyQt4.
(This is something I'm not really sure about: is it possible to setup
all binary 3rd party libs «*.so/*.pyo/*.dll/*.pyd/...» into an isolated
Python environment?)

Time, perhaps, for a little research into what the "simple zip file"
actually needs to contain and where it gets dropped?

Of course there can't be a single APP1 for all possible platforms. But
there could be one setup for each: APP1_Linux, APP1_MacOsX10.5, APP1_WinXP
The developer has to setup/test every supported platform anyway.

The recently announced Snakebite project, when it gets off the ground,
might be able to offer helpful resources. Mike Driscoll has done some
work on building Windows installers for Python apps, though they were to
integrate with an existing Python installation.

The "bootstrap" (standing on the shoulders of giants) could work like this:
- first there is a base app "Python 2.5.1 for Mac OS X 10.5" built by
developer A
- next developer B creates "Django 1.0.2 for Mac OS X 10.5" based on the
one above
- then developer C integrates Pinax into B's app and creates "Pinax
0.5.1 for Mac OS X 10.5"

Why do you need a "Python x.y.z for Mac OS 10.5" if every app brings its
own Python with it?

All those "apps" could be found in Python AppStore and a developer could
search for the one which matches his requirements best. His
implementation cost is therefore minimized.

A developer could, perhaps, but we need to think of the end-users who
will not have the sophistication to make such decisions.

A iTunes-like client would allow the end-user to find apps - assumed
that all (or most) developers upload their apps into the one and only
Python AppStore.

Or that the AppStore is suitable distributed. This, I think, is possibly
the most fanciful part of the scheme. What does this "iTunes-like"
client do for the user?

The client (there should be one for every supported platform) is a
front-end application for the (PSF-) hosted "super-cheeseshop", which
end-users are not expected to access directly.


Is my nirvana really that far away? ;-)

Well it doesn't seem to be getting closer very fast ... but I wish you
luck, and hope to see progress.

regards
Steve

 

[David Lyon]

Hi all,

I only just noticed this thread... :-(

At the moment, I am working on a Python Package Manager project on
Sourceforge.

http://sourceforge.net/projects/pythonpkgmgr/

Firstly, I would welcome any developers who are willing to assist.

Secondly, I think there is much work to be done with python and it's
handling
of packages. To me it seems to need a great deal of streamlining. But I am
working mainly on windows.. and i know it is really easy with ubuntu and
so forth because the os seems to do the job really well.

That's kindof what we're working on...

David

 

[David Lyon]

I think Marcel has a point...

Much can be done and should be done to improve packaging and applications
for python.

That's why I for one am working on the python package manager project. On
sourceforge.

It uses the pypi interface to search.

Actually we haven't made a release yet. Still many.. many things to fix.

Using my iPhone I suddenly realize how easy it is to find applications
in Apple's AppStore. How easy and fast it is to install or de-install
an app. My iPhone even checks in the background if there is an upgrade
which could be installed painlessly. [...]
Unfortunately there's nothing like this in the Python world...

It is being worked on...

I'd be content
to hear your projection for a mere 18 years out (the amount of time
for which Python apps have been in production), or even 10 years
(which takes us back to Python 1.5).

Just a GUI for package management that lets you seperate what is available
for the python platform that you are running on. Install, deinstall, and
get package information.

https://sourceforge.net/projects/pythonpkgmgr/

We only have source at the moment. Only for windows... and only for
python 2.5. Pretty limited... but we'll get there.

David

 

[Daniel Fetchinson]

I think Marcel has a point...

Much can be done and should be done to improve packaging and applications
for python.

That's why I for one am working on the python package manager project. On
sourceforge.

It uses the pypi interface to search.

Actually we haven't made a release yet. Still many.. many things to fix.

Using my iPhone I suddenly realize how easy it is to find applications
in Apple's AppStore. How easy and fast it is to install or de-install
an app. My iPhone even checks in the background if there is an upgrade
which could be installed painlessly. [...]
Unfortunately there's nothing like this in the Python world...

It is being worked on...

I'd be content
to hear your projection for a mere 18 years out (the amount of time
for which Python apps have been in production), or even 10 years
(which takes us back to Python 1.5).

Just a GUI for package management that lets you seperate what is available
for the python platform that you are running on. Install, deinstall, and
get package information.

https://sourceforge.net/projects/pythonpkgmgr/

We only have source at the moment. Only for windows... and only for
python 2.5. Pretty limited... but we'll get there.

How will your solution be different from distutils, setuptools, pip,
zc.buildout and a couple other similar packages I don't recall now?
Have you considered joining one of these efforts in order to not
fragment the "packaging and distribution" arena? Have you evaluated
these solutions in detail and have found that they are not appropriate
for your needs? If yes, what is the most problematic features of these
already existing solutions that you don't like?

Cheers,
Daniel

 

[Kushal Kumaran]

How will your solution be different from distutils, setuptools, pip,
zc.buildout and a couple other similar packages I don't recall now?
Have you considered joining one of these efforts in order to not
fragment the "packaging and distribution" arena? Have you evaluated
these solutions in detail and have found that they are not appropriate
for your needs? If yes, what is the most problematic features of these
already existing solutions that you don't like?

From the web page:
"Coded in Python, and using wxWidgets, this program is a GUI that
drives easy_install and/or pip."

Hopefully, it could grow to handle the other tools you mention as
well.

Note: all I know about this project is the text on the page linked
above. I haven't used it myself.

 

[Paul Boddie]

Just a GUI for package management that lets you seperate what is available
for the python platform that you are running on. Install, deinstall, and
get package information.

https://sourceforge.net/projects/pythonpkgmgr/

We only have source at the moment. Only for windows... and only for
python 2.5. Pretty limited... but we'll get there.

See this page for details of a similar (cross-platform) tool for
browsing the Package Index:

http://www.boddie.org.uk/david/Projects/Python/PyPI-Browser/index.html

For a more ambitious project, take a look at the Smart Package
Manager:

http://labix.org/smart

Paul

 

[David Lyon]

How will your solution be different from distutils, setuptools, pip,
zc.buildout and a couple other similar packages I don't recall now?

For a start.. it doesn't replace those.. it drives them...

Have you considered joining one of these efforts in order to not
fragment the "packaging and distribution" arena?

They seem pretty comprehensive and well developed as they are. I am not
sure what I could do to improve the under-the-hood functionality at this
point in time.

My GUI just drives their back end code..

Have you evaluated these solutions in detail and have found that they
are not appropriate for your needs?

Yes. They don't have a GUI.

If yes, what is the most problematic features of these

already existing solutions that you don't like?

Not having a GUI is the biggest problem. Since I am swapping
languages (Python/Perl/+1 more) in any given day I'm just after
something that is "simple". I can never remember the commandlines..

Regards

David

 

[Daniel Fetchinson]

How will your solution be different from distutils, setuptools, pip,

For a start.. it doesn't replace those.. it drives them...


They seem pretty comprehensive and well developed as they are. I am not
sure what I could do to improve the under-the-hood functionality at this
point in time.

My GUI just drives their back end code..


Yes. They don't have a GUI.

If yes, what is the most problematic features of these

Not having a GUI is the biggest problem. Since I am swapping
languages (Python/Perl/+1 more) in any given day I'm just after
something that is "simple". I can never remember the commandlines..

Okay, I see your point. You might find it useful to go through the
following [1] thread on python-dev that talks about various packaging
and distribution options. Tarek Ziade led a discussion on this topic
at pycon, you might find that helpful too [2] and he also conducted a
survey on packaging [3].

[1] http://mail.python.org/pipermail/python-dev/2009-March/087590.html
[2] http://mail.python.org/pipermail/python-dev/2009-March/087834.html
[3] http://mail.python.org/pipermail/python-dev/2009-March/087656.html

Cheers,
Daniel