Problem: open-uri blocking redirection from http to https

[Xavier Del Castillo]

Hello,

I was working on a small script to verify the presence of an element
through a list of URL, some of these URLs have a redirections from http to
https, when the script crawled into them I got the following error:

/usr/lib/ruby/1.9.1/open-uri.rb:216:in `open_loop': redirection forbidden:
http://beta.carsdirect.com/auto-loans/finance-app ->
https://beta.carsdirect.com/auto-loans/finance-app (RuntimeError)

I understand that this is intentional, as per the comments on open-uri.rb

# This test is intended to forbid a redirection from http://... to
# file:///etc/passwd.
# https to http redirect is also forbidden intentionally.
# It avoids sending secure cookie or referer by non-secure HTTP
protocol.
# (RFC 2109 4.3.1, RFC 2965 3.3, RFC 2616 15.1.3)
# However this is ad hoc. It should be extensible/configurable.

This mentions that "https to http" redirects are forbidden intentionally,
but redirections from "http to https" are also blocked.

Is there a way to override this security check? currently I had to change the
following line in the library to allow "http to https" re-directions:

(/\A(?:http|ftp)\z/i =~ uri1.scheme && /\A(?:http|ftp)\z/i =~ uri2.scheme)

to

(/\A(?:http|ftp|https)\z/i =~ uri1.scheme && /\A(?:http|ftp|https)\z/i =~
uri2.scheme)

Thanks,
Xavi

 

[Tanaka Akira]

# However this is ad hoc. It should be extensible/configurable.

This mentions that "https to http" redirects are forbidden intentionally,
but redirections from "http to https" are also blocked.

Is there a way to override this security check? currently I had to change
the
following line in the library to allow "http to https" re-directions:

(/\A(?:http|ftp)\z/i =~ uri1.scheme && /\A(?:http|ftp)\z/i =~ uri2.scheme)

Currently it is not configurable (as the comment says) except monkey patching.

Maybe open-uri should have some hooks.