Rubygems now integrated in the FreeBSD ports tree

[Jonathan Weiss]

Cheers,


I integrated rubygems into the FreeBSD ports tree.

You now can do a

# cd /usr/ports/www/rubygem-rails
# make install clean

That means that the user is now able to manage gems through the ports tree
and does not have to also use the gem command.

See more here:

http://blog.innerewut.de/articles/2005/07/21/rubygems-now-integrated-in-the-
freebsd-ports-tree


Greets,
Jonathan

 

[Mauricio Fernández]

Hi,

Cheers,

I integrated rubygems into the FreeBSD ports tree.

You now can do a

# cd /usr/ports/www/rubygem-rails
# make install clean

That means that the user is now able to manage gems through the ports tree
and does not have to also use the gem command.

I have been reading http://www.freebsd.org/cgi/query-pr.cgi?pr=82397 and
have a number of questions, esp. about how this would work in practice
from the FreeBSD developer's point of view. Also, please feel free to
correct me if any of the following info is inaccurate.

If I understand

do-install:
${RUBYGEMBIN} install --install-dir ${PREFIX}/lib/ruby/gems/${REV} \
${PORTSDIR}/distfiles/rubygem/${PORTNAME}-${PORTVERSION}${EXTRACT_SUFX}

correctly, ports essentially end up calling gem to install a .gem file placed
in ${PORTSDIR}/distfiles/rubygem/.

If so,
* Is it possible to apply patches to RubyGems packages? What if the
RubyGems package differs substantially from the pristine sources (e.g.
if there's a complex build process using for instance a Rakefile,
probably with some source code massaging)... wouldn't that case
complicate patching?
* Don't you have to create the Makefile, distinfo, pkg-descr and
pkg-list (at least in part) manually for every single version released
through RubyGems?
* If the answer to the previous question is affirmative, this means that
newly released RubyGems packages won't be available through the ports system
until a FreeBSD developer generates the pertinent data, right?
* RubyGems packages installed via ports interfere with those installed
directly with gem install, don't they?
* User scripts need to require 'rubygems' or set RUBYOPT to load
RubyGems packages installed via ports, right?

Thanks in advance for the answers,

 

[Jonathan Weiss]

I have been reading http://www.freebsd.org/cgi/query-pr.cgi?pr=82397 and
have a number of questions, esp. about how this would work in practice
from the FreeBSD developer's point of view. Also, please feel free to
correct me if any of the following info is inaccurate.

If I understand

do-install:
${RUBYGEMBIN} install --install-dir ${PREFIX}/lib/ruby/gems/${REV} \
${PORTSDIR}/distfiles/rubygem/${PORTNAME}-${PORTVERSION}${EXTRACT_SUFX}

correctly, ports essentially end up calling gem to install a .gem file placed
in ${PORTSDIR}/distfiles/rubygem/.

Yes, the port infrastructure will download the gem and install it for you in
${PREFIX}/lib/ruby/gem/1.8 that happens to be the default
/usr/local/lib/ruby/gem/1.8 if you did not change something.

If so,
* Is it possible to apply patches to RubyGems packages? What if the
RubyGems package differs substantially from the pristine sources (e.g.
if there's a complex build process using for instance a Rakefile,
probably with some source code massaging)... wouldn't that case
complicate patching?

Patching is possible as the .gem file is just an archive. The maintainer
could specify patches that would be applied to the files inside the gem. At
the moment there is no routine for extracting the gem and re-archiving it
with help from the ports system. But it would not be difficult to extend it
to do it for you.

The best way to add patches to the gem is to contact the gem's author and
the patches should be applied/included in the original software.

Normally patching should only be needed for C-extension gems.

* Don't you have to create the Makefile, distinfo, pkg-descr and
pkg-list (at least in part) manually for every single version released
through RubyGems?

For a new port you have to create Makefile, distinfo, pkg-descr and
pkg-plist.

The Makefile is not very complicated. Rake's looks like this:

PORTNAME= rake
PORTVERSION= 0.5.4
CATEGORIES= devel ruby
MASTER_SITES= http://rubyforge.org/frs/download.php/4258/

MAINTAINER= (e-mail address removed)
COMMENT= Ruby Make

include <bsd.port.pre.mk>
include "${PORTSDIR}/devel/ruby-gems/Makefile.common"
include <bsd.port.post.mk>

The pkg-plist can be easily generated automatically with the port
sysutils/pkg_trackinst. Distinfo is created by doing a `make makesum` in the
ports directory.

When a new release is out, the maintainer has to edit the Makefile, distinfo
and pkg-plist in order to reflect the new version.

I worked on an automatic way to extrac pkg-plist from the gem-specification
but it turned out to be not trivial.

* If the answer to the previous question is affirmative, this means that
newly released RubyGems packages won't be available through the ports system
until a FreeBSD developer generates the pertinent data, right?

Yes, as with any program avaliable through the ports system. Normally this
will happen soon after the programs release.

* RubyGems packages installed via ports interfere with those installed
directly with gem install, don't they?

For the `gem` command they are not distinguishable but this should not harm
you if you do not try to manipulate the gems installed through the ports
system with the `gem` command.

* User scripts need to require 'rubygems' or set RUBYOPT to load
RubyGems packages installed via ports, right?

For user scripts nothing changes from the manual way of installing gems. The
ports system ist just a wrapper around the gem command for installing
rubygems.

The idea is that a FreeBSD user should only have to use the ports/package
system in order to install software. Only one command to manage installed
software and not `gem` for rubygems `pear` for PHP-Pear extension `perl
-MCPAN` for Perl libs and so on.

Just do cd `/usr/ports/my/program && make install clean` or `portupgrade
-a` and you are done.

The system is not perfect but I hope that it helps FreeBSD users.

Thanks in advance for the answers,

Greets,
Jonathan

 

[Mauricio Fernández]

Thanks for the *quick* reply

Patching is possible as the .gem file is just an archive. The maintainer
could specify patches that would be applied to the files inside the gem. At
the moment there is no routine for extracting the gem and re-archiving it
with help from the ports system. But it would not be difficult to extend it
to do it for you.

I see. It'd be necessary to modify the do-install target to patch the
extracted files, right?

The best way to add patches to the gem is to contact the gem's author and
the patches should be applied/included in the original software.

Yes, this makes sense anyway in most cases, since we're often dealing with
(potential) upstream patches. And I guess FreeBSD-specific (if any) patches
would be handled as you said. I believe it's still good to be able to patch
at that level because the original author could be MIA, or even disappear
completely after a while.

Normally patching should only be needed for C-extension gems.

They're often tricky to build, aren't they?

For a new port you have to create Makefile, distinfo, pkg-descr and
pkg-plist.

The Makefile is not very complicated. Rake's looks like this: [very clean Makefile]
MASTER_SITES= http://rubyforge.org/frs/download.php/4258/

The pkg-plist can be easily generated automatically with the port
sysutils/pkg_trackinst. Distinfo is created by doing a `make makesum` in the
ports directory.

Would it be possible to point to the original .gem (also specifying the
checksum) instead of putting it under ${PORTSDIR}/distfiles/rubygem/ ?
I don't really know the FreeBSD infrastructure but I suspect the latter
is more costly (?). But I guess it'd require the sort of specific code
you talked about before. Also, is distfiles/rubygem/ updated if one uses
CVSup to track the ports hierarchy? Wouldn't distfiles/rubygem/ become quite
large/heavy eventually?

When a new release is out, the maintainer has to edit the Makefile, distinfo
and pkg-plist in order to reflect the new version.

I worked on an automatic way to extrac pkg-plist from the gem-specification
but it turned out to be not trivial.

IIRC the metadata includes a complete file list, but it's marshalled with YAML
and inside the "tar(tar + metadata.gz)" .gem structure. Maybe a Ruby script
could handle this (I might try to write it if I find the time).

Yes, as with any program avaliable through the ports system. Normally this
will happen soon after the programs release.
OK


For the `gem` command they are not distinguishable but this should not harm
you if you do not try to manipulate the gems installed through the ports
system with the `gem` command.

I see, so one shouldn't gem uninstall anything installed through the ports
system

The idea is that a FreeBSD user should only have to use the ports/package
system in order to install software. Only one command to manage installed
software and not `gem` for rubygems `pear` for PHP-Pear extension `perl
-MCPAN` for Perl libs and so on.

Just do cd `/usr/ports/my/program && make install clean` or `portupgrade
-a` and you are done.

The system is not perfect but I hope that it helps FreeBSD users.

You're too modest It seems to me the system adds:
* atomic installs (although the system cannot yet be protected from
breakeage through direct invocation of gem)
* native dependencies (like needed typically by C extensions)

These alone (especially the second IMHO) look very well from here

I have some concerns regarding upstream source compatibility (the
"require 'rubygems'" and require_gem issues in RubyGems packages
vs. everything else, DATADIR, etc.) and the ability to patch at the
repackager level, but these are intrinsic to RubyGems and not due to
your wrapper.

cheers,

 

[Jonathan Weiss]

Thanks for the *quick* reply

I see. It'd be necessary to modify the do-install target to patch the
extracted files, right?

Or you could define a pre-install target, patch the source and rebuild the
gem.

Yes, this makes sense anyway in most cases, since we're often dealing with
(potential) upstream patches. And I guess FreeBSD-specific (if any) patches
would be handled as you said. I believe it's still good to be able to patch
at that level because the original author could be MIA, or even disappear
completely after a while.

This is true, I will think about what's the best way to support patches.

For a new port you have to create Makefile, distinfo, pkg-descr and
pkg-plist.

The Makefile is not very complicated. Rake's looks like this: [very clean Makefile]
MASTER_SITES= http://rubyforge.org/frs/download.php/4258/

The pkg-plist can be easily generated automatically with the port
sysutils/pkg_trackinst. Distinfo is created by doing a `make makesum` in the
ports directory.

Would it be possible to point to the original .gem (also specifying the
checksum) instead of putting it under ${PORTSDIR}/distfiles/rubygem/ ?
I don't really know the FreeBSD infrastructure but I suspect the latter
is more costly (?). But I guess it'd require the sort of specific code
you talked about before. Also, is distfiles/rubygem/ updated if one uses
CVSup to track the ports hierarchy? Wouldn't distfiles/rubygem/ become quite
large/heavy eventually?

The gem is not put into ${PORTSDIR}/distfiles/rubygem/ by the maintainer, it
is downloaded by the ports infrastructure and put there when you do the
`make install`. So it does not clutter the CVS repository. The maintainer
just specifies from where to get the gem like done here:

MASTER_SITES= http://rubyforge.org/frs/download.php/4258/

The common Makefile will then try to download

http://rubyforge.org/frs/download.php/4258/{$PORTNAME}-{$PORTVERSION}.${
EXTRACT_SUFX}

Which should be in most cases exaclty what you want (like rake-0.5.4.gem).
Of course you can override these values in the port's Makefile.

IIRC the metadata includes a complete file list, but it's marshalled with YAML
and inside the "tar(tar + metadata.gz)" .gem structure. Maybe a Ruby script
could handle this (I might try to write it if I find the time).

The problem is that the file list can be in YAML and in pure Ruby. Further,
some C-extensions do not have a complete file list for the generated code.
So I picked the easier solution of having the maintainer specifying the file
list.

Generating the list out of the spec would be a very nice feature.

You're too modest It seems to me the system adds:
* atomic installs (although the system cannot yet be protected from
breakeage through direct invocation of gem)
* native dependencies (like needed typically by C extensions)

These alone (especially the second IMHO) look very well from here

Thanks.

Now the C-extensions for MySQL or PostgreSQL can actually depend on the
databases in FreeBSD.

This was why I wrote this extension in the first way. I wanted to install
Typo through the ports system but it depends on the Rails-gems. You could
not specify this dependency.

Now applications that depend on rubygems can be installed (if the rubygem is
present in the ports tree).

Greets,
Jonathan

 

[lists]

Sweet, thanks!

-Ryan