why can't Ruby load .gem files directly?

[Eivind Eklund]

Eivind Eklund wrote:
Okay (and thanks for the reply). That's weird. If they don't paticularly
care about Ruby, what are they doing installing Ruby gems (usually
libraries)? If they're trying to install an application that happens to
be written in Ruby, then does it not suffice for the .deb to list ruby
and rubygems as dependences, and then for the install step to be `gem
install the-app-dependencies` or whatever?

(Note that some of these are projections from problems I know about it
in the context of FreeBSD, which I'm a developer for, and RPA, which
I've been an advisor for. I'm just projecting to Debian, based on
having read various Debian policy documents and talked to Debian users
and developers.)

There is several reasons, including the deinstall problem that was
mentioned, that Gems start writing places they shouldn't (so write
protection of some areas of the disk don't work), and that Debian
generally do maintenance on single versions, fixing security holes as
they pop up etc so you can install a package, then run apt-get update
and get everything updated. This cascades to the rest of the package
organization.

Also, people expect to be able to find documentation etc in the
"normal places" even for langauges they're not used to, and
applications sometimes depend on executables.

There's a whole host of complexities that come up here when you
actually do packaging and maintenance.

Eivind.

 

[Joshua Haberman]

They're trying to repackage software that the author assume to be
Gem-distributed as Debian packages (.deb, IIRC). The reason for this
is simple: To provide Ruby software on Debian for Debian users in the
way Debian users are used to. These users do not particularly care
about Ruby, and definately do not want to have to know one packaging
system for handling Ruby software, and another for handling Perl, and
another for handling Java, and another for handling Haskell, and
another for handling Python, and another way for handling C code, and
...

Instead, they want they software to Just Work Like All Other Software,
the way they are used to, and to not have to care about the language
the software is written in at all.

May I request that we talk about technical issues like grown-up
engineers, not like guests on "Jerry Springer?"

Tell me if this is an accurate list of the problems packaging gems:

1. Gems provide more guarantees than Debian packages can

2. Gems let you install more than one version of the same software,
but Debian packages don't

3. something about Debian wanting software to be in a single directory?

I've tried to bring myself up to speed on these issues by reading
recent threads, but it's very difficult to separate the technical
information from the arguing.

Josh

 

[Devin Mullins]

Nope, because in other debian-ish situations with libraries (be they written
in C and compiled) you can put them in and pull them out seperate of the
application by using apt-get, which makes a LOT of sense.

But why would someone who doesn't know or care about Ruby be doing that?

What would happen when you uninstall the program if packaged as you mention?
You can't remove the gems, and you can't really leave them there either...

`gem uninstall thegems`

(Yeah, there are probably things you can do with the debian pkg manager
that you can't with gems, and I think it's worth fixing that.)

I guess my concern with repackaging is that it introduces duplication.

Devin

 

[Devin Mullins]

That was my first thought too. No, because if you distribute the debians as
part of a distro, you're requiring them to have an internet connection to get
them, and in fact there is no way at that point to include said software as
part of a distro that you can install without an internet connection.

RubyGems can install local gem files quite easily. That was what I was
referring to.

Also, my mom (who just unplugged her network cable) wants to know what this
funky error about WGET FAILED FOR GEM INSTALL FROM DEB PACKAGE means.

Really? My mom wants to know why it takes 6 minutes to boot up Windows
Me.

Devin
(I'm joking. Note the "".)

 

[Joshua Haberman]

--Apple-Mail-3--357959784
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=US-ASCII;
delsp=yes;
format=flowed

[...] For example, can gems support these use cases?

- I want to try out a gem on a system where I don't have root. [...]

Yes.

How?

- I want to have more than one local gem repository [...]

Yes.

How?

- I want to install a gem from a gemfile I already have [...]

Yes.

How?

- I want to "unpack" a gem into a directory structure that can be
deployed to another machine, without having to run "gem" on the
remote machine

Yes.

How?

- same as the previous one, but I also want to "unpack" the gem into
a different directory root [...]

Yes.

How?

(References to documentation are fine, I just haven't been able to
find much information about how to to do these things).

Josh

--Apple-Mail-3--357959784--

 

[Kevin Brown]

On Oct 2, 2005, at 9:10 AM, Eivind Eklund wrote:
[*snip*]
May I request that we talk about technical issues like grown-up
engineers, not like guests on "Jerry Springer?"

Unnecessary insult, but sure.

Tell me if this is an accurate list of the problems packaging gems:

1. Gems provide more guarantees than Debian packages can

Gems provide guarantees that turn out to be wrong with the way Debian installs
software with its packaging scheme (out of the way of the user, so that you
can install your own stuff in /usr/bin etc).

2. Gems let you install more than one version of the same software,
but Debian packages don't

Debian packages do, but will maintain a symlink to the most recently installed
version. I have python2.2 and 2.4 installed right now, but the symlink
points to 2.4, thus causing problems if python2.2 software assumes it can get
to python with /usr/bin/python.

3. something about Debian wanting software to be in a single directory?

Not quite, but in an out of the way place.

I've tried to bring myself up to speed on these issues by reading
recent threads, but it's very difficult to separate the technical
information from the arguing.

Hope I've helped.

 

[Devin Mullins]

no, since gems make layout assumptions that do not match the debian
policy (nor the policy of most package systems on *nix, it seem).
Anyway, there was a huge thread on this on ruby-core that people may
like to read, since some of this.

Yeah, I figured that I should just be not-lazy and go read ruby-core.

I have faith in the rubygems dev crew, anyway, so that the problems
can be addressed before merging it in ruby's HEAD.

Ditto.

(References to documentation are fine, I just haven't been able to
find much information about how to to do these things).

http://docs.rubygems.org/

Devin

 

[Devin Mullins]

Okay. Well, those are reasonable.

Devin

 

[Eivind Eklund]

May I request that we talk about technical issues like grown-up
engineers, not like guests on "Jerry Springer?"

These are user interface issues, and I am trying to explain them in a
way that get the overall top level user issues communicated.=20
Ruby-talk is the wrong list for the detailed technical discussion.=20
Also, the technical issues are reasonably easy to solve, as long as
there is acceptance that they need to be solved, and that they need to
be discussed at a technical level.

Tell me if this is an accurate list of the problems packaging gems:

1. Gems provide more guarantees than Debian packages can

*Different* guarantees, and being unable to support the guarantees
that Debian packages give. Specifically, Gems do not support using
the directory layout that Debian users expect. Gems generally go in a
single directory. Some ruby developers end up relying on that single
directory because Gems lack the infrastructure for them to access
things any other way.

I've posted more exact details of the stuff that's necessary to
support to rubygems developer list as references a year and half or so
ago. In the last couple of days, I've posted this as extracted and
minimized text with very concrete lists, to be able to start building
the abstractions necessary. That's under the subject of "Gems and
repackaging, hopefully helpful".

There's also the problem that gems are hard to patch, though that's
resolvable by using the original source as long as the authors are not
led to rely on the RubyGems layout, and end up being just rude to
those trying to give them patches to make this work.

Eivind.

 

[Eivind Eklund]

`gem uninstall thegems`

What if other stuff rely on those gems? Like user-written code?

This stuff is tricky to get right, even more tricky than the single
package manager situation.

(Yeah, there are probably things you can do with the debian pkg manager
that you can't with gems, and I think it's worth fixing that.)

I guess my concern with repackaging is that it introduces duplication.

In what way are you thinking of?

I think not repackaging introduce duplication, in the form of
duplicated package managers handling the same machine, and thus
duplicated interfaces the user needs to know.

Having native packages avoid duplication as long as you use only
native packages; using both is of course a problem, and really
ideally, they'd be able to hook together nicely - unfortunately, I'm
uncertain if that is doable in practice (for combined technical and
social reasons.)

Eivind.

 

[Joshua Haberman]

These are user interface issues, and I am trying to explain them in a
way that get the overall top level user issues communicated.

I apologize -- I misread your tone. It sounded like you were trying
to make Debian users sound unreasonable, and their wishes unimportant.

I just read the "gems and repackaging" thread on ruby-core@, and I
agree with your position 100%.

Josh

 

[Joshua Haberman]

--Apple-Mail-4--347340683
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=US-ASCII;
delsp=yes;
format=flowed

If you have a shared library embedded in your gem file, how do you
point your
LD_LIBRARY_PATH environment variable at it so that the OS knows
about it.
Not only is a hard problem to solve, it is a hard problem that must
(potentially) be solved differently on each platform.

Yes, I can definitely see that loading a shared library packaged
inside a gem is not going to work.

So why not have shared libraries live in their own files in the gem
directory?

my-gem-dir/
my-gem-dir/foo-1.2.gem
my-gem-dir/foo-1.2-libfoo.so
my-gem-dir/foo-1.2-libbar.so
my-gem-dir/baz-0.3.gem

The gem could specify what shared libraries need to be loaded (or you
could pattern match on file names -- eg. load "foo-1.2*.so")

Yes, that would mean that packages containing C extensions would no
longer be one single gem file, but luckily we have tar to help with
that.

Josh

--Apple-Mail-4--347340683--

 

[Devin Mullins]

What if other stuff rely on those gems? Like user-written code?

OK, point. What does apt-get (or ports) do natively to fix this?

In what way are you thinking of?

For a given libraries, you would have multiple packages (RubyGems,
ports, apt-get, portage, rpm, fink, darwinports...) all that contain the
same meaning in them -- what to install and how to install it. Much of
the same logic comprises each of those packages. I would rather efforts
be taken to put as much of the logic in a single package system, and let
the other ones be "adapters" to that one.

Further, this causes one of two problems for the library author:
1. He's got to publish in 10 different package formats
2. His library has got to be famous enough that the major packaging
systems have put maintainers in charge of his library, and he's got to
not mind the delay.

I think not repackaging introduce duplication, in the form of
duplicated package managers handling the same machine, and thus
duplicated interfaces the user needs to know.

Yeah, I see your point here. I would think that an "adapter" technique
would solve the UI problem better than static repackaging would. As far
as multiple package managers being duplication, well, that's true, but
there are, like, twenty web frameworks for Python, and that's likely not
going to change. Duplication of effort is a fact of life.

really
ideally, they'd be able to hook together nicely - unfortunately, I'm
uncertain if that is doable in practice (for combined technical and
social reasons.)

That's too bad. Imagine if there existed a single piece of code that
allowed apt-get to interact with .gem files and gems.rubyforge.com.
That would allow apt-get users to control Ruby libs, and wouldn't
require each lib to be repackaged for apt-get specifically.

(It would't obviate all those differences you've pointed out, of course.)

Devin

 

[Jim Weirich]

[...] For example, can gems support these use cases?

[... yes to all use cases ...]
How?

Ok, I was lazy with my responces. Here's the details.

- I want to try out a gem on a system where I don't have root. [...]

http://docs.rubygems.org/read/chapter/10#page33 -- See the --install-dir
option.

- I want to have more than one local gem repository [...]

http://docs.rubygems.org/read/chapter/12 -- See the GEM_PATH and GEM_HOME
environment options. Also see http://docs.rubygems.org/read/chapter/11 for
config file variations.

- I want to install a gem from a gemfile I already have [...]

http://docs.rubygems.org/read/chapter/10#page33 -- see --local option.

http://docs.rubygems.org/read/chapter/10#page39 -- The "unpack" command. Copy
the resulting directory to whereever you want on any machine and (a) run the
packages native install/setup scripts or (b) arrange for the proper
directories to be included in the LOAD_PATH when ruby is run. You can
determine the "proper directories" by examining the gemspec for that gem (see
http://docs.rubygems.org/read/chapter/10#page37 on how to extract the gem
spec).

If you don't want to install the gem first (unpack works on installed gems),
you can just untar the gem file and then untar the data embedded there

Sorry, there's more manual steps in this one, but the information is there and
is public. If you don't want to do all these steps by hand, we do include a
script that does it for you (see http://docs.rubygems.org/read/book/2).

- same as the previous one, but I also want to "unpack" the gem into
a different directory root [...]

Same answer as above, note the "whereever you want" in the answer.

 

[Jim Weirich]

Yes, that would mean that packages containing C extensions would no =A0
longer be one single gem file, but luckily we have tar to help with =A0
that.

Exactly. A .gem file is just a tar file.=20

Once you have to start unpacking gem files for a portion of the payload, =
you=20
might as well go all the way and unpack the entire thing.

--=20
-- Jim Weirich (e-mail address removed) http://onestepback.org

 

[Austin Ziegler]

my-gem-dir/
my-gem-dir/foo-1.2.gem
my-gem-dir/foo-1.2-libfoo.so
my-gem-dir/foo-1.2-libbar.so
my-gem-dir/baz-0.3.gem

Except that my-gem-dir/foo-1.2-libfoo.so would *actually* be:
my-gem-dir/i386-linux/foo-1.2-libfoo.so or something like that.

Ruby and RubyGems solves a lot of this already.

-austin

 

[Austin Ziegler]

OK, point. What does apt-get (or ports) do natively to fix this?

Nothing. Nothing at all, which makes it something of a nonsense
question. If something isn't installed by the package management system,
then the package management system can't be responsible for it.

I'm in the middle of discussions so that we can possibly solve this in a
way that at *least* makes the most complex case feasible without adding
all sorts of nonsense requirements to the developer or RubyGems. I
maintain that RubyGems is the *right* solution for Ruby because it both
helps with package management and with API versioning. I also believe
that the *right* way for package management systems to work with
RubyGems is to *use* RubyGems and the facilities it offers (or will
offer) to install gems. RubyGems does a stand-up job of managing its
files -- and with a singular exception (binary stubs), those files can
*never* be missed when the package is uninstalled.

[...]

That's too bad. Imagine if there existed a single piece of code that
allowed apt-get to interact with .gem files and gems.rubyforge.com.
That would allow apt-get users to control Ruby libs, and wouldn't
require each lib to be repackaged for apt-get specifically.

(It would't obviate all those differences you've pointed out, of
course.)

Heck, since Debian is a binary distribution, there's nothing that stops
Debian repackagers from creating .debs that contain customised .gem
files that are compiled for particular platforms. It's that simple.

-austin

 

[Joshua Haberman]

Except that my-gem-dir/foo-1.2-libfoo.so would *actually* be:
my-gem-dir/i386-linux/foo-1.2-libfoo.so or something like that.

Ruby and RubyGems solves a lot of this already.

Right, but the problem with RubyGems that I am trying to solve is
that the interpreter can't load them directly. (see the subject of
this email). I don't want to perform an "install" step.

If this functionality was supported, I believe a lot of the problems
currently being debated would melt away.

Josh

P.S. Incidentally, I don't see the point of the extra "i386-linux"
directory. What does it buy you? I don't keep my C libraries in /
usr/lib/i386-linux. If shared objects are on my machine, it is
assumed that they match the architecture of my machine.

 

[Joshua Haberman]

I also believe
that the *right* way for package management systems to work with
RubyGems is to *use* RubyGems and the facilities it offers (or will
offer) to install gems.

Why?

What compelling reason is there to use "gem install" over "tar xf"?

There are many cases where "gem install" is not suitable, but "tar
xf" is.

Josh

 

[Austin Ziegler]

P.S. Incidentally, I don't see the point of the extra "i386-linux"
directory. What does it buy you? I don't keep my C libraries in /
usr/lib/i386-linux. If shared objects are on my machine, it is assumed
that they match the architecture of my machine.

But do they? What about on a modern AMD or Intel? You'll have i686-linux
and x86_64-linux on the same box. On 64-bit Windows, this is represented
as "Program Files" (64-bit) and "Program Files-x86" (64-bit).

-austin