How to download file from web site

[Timothy Hunter]

Okay, it's easy to download a file from an FTP server with Net::FTP.
Given a URL like this:
"http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz" is
there a way using pure Ruby to download this file?

 

[ara.t.howard]

Okay, it's easy to download a file from an FTP server with Net::FTP. Given a
URL like this:
"http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz" is there
a way using pure Ruby to download this file?

harp:~ > cat a.rb
require 'open-uri'
uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'

open(uri) do |fin|
open(File.basename(uri), 'w') do |fout|
while(buf = fin.read(8192))
fout.write buf
end
end
end

harp:~ > ruby a.rb

harp:~ > new RMagick-1.15.3.tar.gz
-rw-rw-r-- 1 ahoward ahoward 1048764 Mar 3 08:00 RMagick-1.15.3.tar.gz


-a

 

[Timothy Hunter]

harp:~ > cat a.rb
require 'open-uri'
uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'

open(uri) do |fin|
open(File.basename(uri), 'w') do |fout|
while(buf = fin.read(8192))
fout.write buf
end
end
end

harp:~ > ruby a.rb

harp:~ > new RMagick-1.15.3.tar.gz
-rw-rw-r-- 1 ahoward ahoward 1048764 Mar 3 08:00
RMagick-1.15.3.tar.gz


-a

Thanks again, Ara!

 

[Robert Klemme]

Okay, it's easy to download a file from an FTP server with Net::FTP.
Given a URL like this:
"http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz" is
there a way using pure Ruby to download this file?

You might be surprised to learn that there is also Net::HTTP...

http://www.ruby-doc.org/stdlib/libdoc/net/http/rdoc/index.html

And then there's open-uri

http://www.ruby-doc.org/stdlib/libdoc/open-uri/rdoc/index.html

robert

 

[ara.t.howard]

Thanks again, Ara!

no way - thank you! i'm preparing a class right now, here's one of many
examples i've got put together:


#
# gem install RMagick @ http://rmagick.rubyforge.org/
#
require "rubygems"
require "RMagick"
#
# converts all images named on the command-line into thumbnails
#
ARGV.each do |path|
path = File.expand_path path
dirname, basename = File.split path
base, ext = basename.split %r/[.]/, 2

img = Magick::ImageList.new path

thumb = img.thumbnail 96, 96
thumb_path = File.join dirname, "#{ base }.thumb.#{ ext }"
thumb.write thumb_path

puts thumb_path
end



thanks for rmagick.

-a

 

[Timothy Hunter]

Thanks again, Ara!

no way - thank you! i'm preparing a class right now, here's one of many
examples i've got put together:


#
# gem install RMagick @ http://rmagick.rubyforge.org/
#
require "rubygems"
require "RMagick"
#
# converts all images named on the command-line into thumbnails
#
ARGV.each do |path|
path = File.expand_path path
dirname, basename = File.split path
base, ext = basename.split %r/[.]/, 2

img = Magick::ImageList.new path

thumb = img.thumbnail 96, 96
thumb_path = File.join dirname, "#{ base }.thumb.#{ ext }"
thumb.write thumb_path

puts thumb_path
end



thanks for rmagick.

-a

You're welcome!

The reason I'm asking these questions is because I'm working on a Ruby
program to download and install ImageMagick, all its delegate libraries,
and RMagick, on OS X, starting from scratch and not using MacPorts.

BTW, I saw your comment about the rmagick gem. You're right, it's
screwy. I'm hoping to mothball it in RMagick 2.0.0, if I can figure out
a way to replace my autoconf-generated configure script with pure Ruby.

 

[James Edward Gray II]

The reason I'm asking these questions is because I'm working on a
Ruby program to download and install ImageMagick, all its delegate
libraries, and RMagick, on OS X, starting from scratch and not
using MacPorts.

I just spent about a day fixing my RMagick install (on Mac OS X),
that had become horribly broken, so releasing something like this
would make you a hero in my eyes.

James Edward Gray II

 

[Timothy Hunter]

I just spent about a day fixing my RMagick install (on Mac OS X), that
had become horribly broken, so releasing something like this would
make you a hero in my eyes.

James Edward Gray II

Ouch! Sorry you had trouble. Email me when you have a problem, James.
I'll be glad to do what I can to help.

BTW, just last week I updated my "Installing RMagick on OS X HOWTO"
here: http://rmagick.rubyforge.org/install-osx.html. I guess it's too
late to help you but maybe it'll help somebody else.

 

[James Edward Gray II]

Ouch! Sorry you had trouble. Email me when you have a problem,
James. I'll be glad to do what I can to help.

No worries. It wasn't your fault. I broke it.

James Edward Gray II

 

[Ezra Zygmuntowicz]

Ouch! Sorry you had trouble. Email me when you have a problem,
James. I'll be glad to do what I can to help.

BTW, just last week I updated my "Installing RMagick on OS X HOWTO"
here: http://rmagick.rubyforge.org/install-osx.html. I guess it's
too late to help you but maybe it'll help somebody else.

Here is a shell script I wrote a while ago to build ruby/imagemagick/
rmagick on OSX without using macports. Feel free to use it if you want.

#!/bin/sh

# Install Ruby & IMageMagick & RMagick on Macintosh OS X 10.4.7 (Intel)
# Stock Apple Developer Tools (2.3)

# Readline
READLINE_VERSION="5.1"
wget ftp://ftp.gnu.org/gnu/readline/readline-${READLINE_VERSION}.tar.gz
tar xzvf readline-${READLINE_VERSION}.tar.gz
pushd readline-${READLINE_VERSION}
/configure --prefix=/usr/local
make
sudo make install
popd


# Ruby

RUBY_VERSION="1.8.5"

wget ftp://ftp.ruby-lang.org/pub/ruby/ruby-${RUBY_VERSION}.tar.gz
tar xfz ruby-${RUBY_VERSION}.tar.gz

pushd ruby-${RUBY_VERSION}
/configure
make
sudo make install
popd

# Ruby Gems

GEM_VERSION="0.9.2"

wget http://rubyforge.org/frs/download.php/11289/rubygems-$
{GEM_VERSION}.tgz
tar xfz rubygems-${GEM_VERSION}.tgz

pushd rubygems-${GEM_VERSION}
sudo ruby setup.rb
popd

# ImageMagick

LPNG_VERSION="1.2.12"

wget ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng-$
{LPNG_VERSION}.tar.gz
tar xfz libpng-${LPNG_VERSION}.tar.gz

pushd libpng-${LPNG_VERSION}
/configure
make
sudo make install
popd

JPEG_VERSION="6b"

wget http://www.ijg.org/files/jpegsrc.v${JPEG_VERSION}.tar.gz
tar xfz jpegsrc.v${JPEG_VERSION}.tar.gz

pushd jpeg-${JPEG_VERSION}
/configure
make
sudo make install
sudo make install-lib
popd

TIFF_VERSION="3.8.2"

wget ftp://ftp.remotesensing.org/pub/libtiff/tiff-${TIFF_VERSION}.tar.gz
tar xfz tiff-${TIFF_VERSION}.tar.gz

pushd tiff-${TIFF_VERSION}
/configure
make
sudo make install
popd

IM_VERSION_MAJOR="6.2.9"
IM_VERSION_MINOR="4"

IM_VERSION="${IM_VERSION_MAJOR}-${IM_VERSION_MINOR}"

wget ftp://ftp.imagemagick.net/pub/ImageMagick/ImageMagick-$
{IM_VERSION}.tar.gz
tar xfz ImageMagick-${IM_VERSION}.tar.gz

pushd ImageMagick-${IM_VERSION_MAJOR}
/configure
make
sudo make install
popd


sudo gem install rmagick


Cheers-

-- Ezra Zygmuntowicz
-- Lead Rails Evangelist
-- (e-mail address removed)
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)

 

[Timothy Hunter]

Here is a shell script I wrote a while ago to build
ruby/imagemagick/rmagick on OSX without using macports. Feel free to
use it if you want.

Thanks, Ezra! That's a very nice gift. I'll incorporate all the best
parts into my script.

There are a couple of these kinds of bash scripts for installing RMagick
floating around. None of them implement my exact set of superstitions
about installing ImageMagick[1], so I thought it would be useful to do
my own. I'm using Ruby instead of bash, though, since I get very little
opportunity to write Ruby code.

[1] "Speed up your RMagick apps in 1 easy step"
http://rubyforge.org/forum/forum.php?thread_id=10975&forum_id=1618

 

[Bil Kleb]

require 'open-uri'
uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'

open(uri) do |fin|
open(File.basename(uri), 'w') do |fout|
while(buf = fin.read(8192))
fout.write buf
end
end
end

He asks naively...

Why is this so complicated, e.g., the 8192 buffered write, etc.?

Later,

 

[James Edward Gray II]

He asks naively...

Why is this so complicated, e.g., the 8192 buffered write, etc.?

So you don't try to slurp a very large file into Ruby's memory I'm
guessing.

James Edward Gray II

 

[ara.t.howard]

He asks naively...

Why is this so complicated, e.g., the 8192 buffered write, etc.?

well, this will do it too

def wget src, dst
open(dst,'w'){|b| open(src){|a| b.write a.read}}
end

but if you

wget 'http://host/huge_giant_output.cgi', 'big.out'

and then start 1000 of them in threads you might eat a lot of memory

the 8192 thing is just being robust in the face of potential giant docs. not
an issue for html probably but the OP was talking about downloading tar.gz
files, so those could be big. we've got plenty of them that are > 10GB for
download on our servers....

cheers.


-a

 

[Joel VanderWerf]

He asks naively...

Why is this so complicated, e.g., the 8192 buffered write, etc.?

It does seem too low-level, doesn't it? Especially in the same snippet
with open(uri), which abstracts so much.

Didn't the rio library[1] have a way to abstract this operation?

This is from the docs:

Copy a web-page to a file
rio('http://rubydoc.org/') > rio('afile')

I suppose it's using the same buffering technique underneath that
abstraction.

[1] http://rio.rubyforge.org/ (never used it myself)

 

[ara.t.howard]

He asks naively...

Why is this so complicated, e.g., the 8192 buffered write, etc.?

It does seem too low-level, doesn't it? Especially in the same snippet with
open(uri), which abstracts so much.

Didn't the rio library[1] have a way to abstract this operation?

This is from the docs:

Copy a web-page to a file rio('http://rubydoc.org/') > rio('afile')

I suppose it's using the same buffering technique underneath that
abstraction.

[1] http://rio.rubyforge.org/ (never used it myself)

reminds me of an RCR i've had in mind:

io.relay another_io, bufsize

seems like a nice one huh?

-a

 

[Robert Klemme]

(e-mail address removed) wrote:

require 'open-uri'
uri =
'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'

open(uri) do |fin|
open(File.basename(uri), 'w') do |fout|
while(buf = fin.read(8192))
fout.write buf
end
end
end

He asks naively...

Why is this so complicated, e.g., the 8192 buffered write, etc.?

It does seem too low-level, doesn't it? Especially in the same snippet
with open(uri), which abstracts so much.

Didn't the rio library[1] have a way to abstract this operation?

This is from the docs:

Copy a web-page to a file rio('http://rubydoc.org/') > rio('afile')

I suppose it's using the same buffering technique underneath that
abstraction.

[1] http://rio.rubyforge.org/ (never used it myself)

reminds me of an RCR i've had in mind:

io.relay another_io, bufsize

seems like a nice one huh?

Yes, although I'm not sure whether "relay" is immediately clear to
everyone - especially non native speakers. I briefly thought of
"copy_to" but that is not correct in every case, i.e. not always it is
actually a copy. What about "forward_to"? Hm...

Kind regards

robert