Download csv file from secure web site (https

[Alice Gheorghiu]

I am a newbie to Ruby and I created a script to process a file which I
am getting from my local directory. However, in order to obtain this
file, I do a manual login to a secure http site (https) which presents
me with a logon screen where I enter the username and password. After
successful authentication, I am presented with a download page which has
as extension a php page which has a download button to get the file.
After I download on my local Windows Vista machine, I use FileZilla to
place it on the linux server.
How can I automate this in Ruby with open-uri or other extension for
that matter that would enable me to authenticate and get the file
directly on my linux server?
Any insight/code-example will be deeply appreciated.
Thanks so much, -- Alice

 

[Reid Thompson]

I am a newbie to Ruby and I created a script to process a file which I
am getting from my local directory. However, in order to obtain this
file, I do a manual login to a secure http site (https) which presents
me with a logon screen where I enter the username and password. After
successful authentication, I am presented with a download page which has
as extension a php page which has a download button to get the file.
After I download on my local Windows Vista machine, I use FileZilla to
place it on the linux server.
How can I automate this in Ruby with open-uri or other extension for
that matter that would enable me to authenticate and get the file
directly on my linux server?
Any insight/code-example will be deeply appreciated.
Thanks so much, -- Alice

can you ssh/scp directly to the secure http site (https) from a command line on
the linux host?

If so, you should be able to setup a passwordless login for scp and get the file
via a cron'd job using scp (google ssh passwordless login)

$ scp (e-mail address removed):/path/to/file /path/to/download/to/filename

 

[Alice Gheorghiu]

can you ssh/scp directly to the secure http site (https) from a command
line on
the linux host?

$ scp (e-mail address removed):/path/to/file
/path/to/download/to/filename
Tried ssh and I am getting error as follows in the ssh command:
$ ssh username@https://www.desired_website.com/pageWLinkToReport.php
$ ssh: https://www.desired_website.com/pageWLinkToReport.php: Name or
service > not known

If so, you should be able to setup a passwordless login for scp and get
the file
via a cron'd job using scp (google ssh passwordless login) - not possible yet

but maybe - I can ask the data provider to set our account up for
automatic
download?

 

[Alice Gheorghiu]

$ scp (e-mail address removed):/path/to/file
/path/to/download/to/filename

Tried ssh and I am getting error as follows in the ssh command:
$ ssh username@https://www.desired_website.com/pageWLinkToReport.php
$ ssh: https://www.desired_website.com/pageWLinkToReport.php: Name or
service not known- not possible yet but maybe - I can ask the data provider to set our
account up for automaticdownload?
I spoke with both data providers and they cannot help much. Any
suggestions of Ruby articles or sample code that may treat a similar
problem? Thanks again so very much.

 

[Joel VanderWerf]

Tried ssh and I am getting error as follows in the ssh command:
$ ssh username@https://www.desired_website.com/pageWLinkToReport.php
$ ssh: https://www.desired_website.com/pageWLinkToReport.php: Name or
service not known

If it works at all, it would be something like:

ssh username@desired_website.com

but the fact that there is a https interface does not imply that there
is an ssh interface. They are handled by different server software, and
unless the provider explicitly set up an ssh server, it won't be there.
And for that matter, having an account on the https server doesn't imply
having an account that you can access via ssh, even if the ssh server is
running.

It sounds like a task for something like mechanize[1], but I don't know
how it is with https. Some others on this list will know, I hope... or
try googling for "ruby mechanize https".

[1] http://mechanize.rubyforge.org/mechanize/

 

[Alice Gheorghiu]

Alice Gheorghiu wrote:

Tried ssh and I am getting error as follows in the ssh command:
$ ssh username@https://www.desired_website.com/pageWLinkToReport.php
$ ssh: https://www.desired_website.com/pageWLinkToReport.php: Name or
service not known

If it works at all, it would be something like:
ssh username@desired_website.com

but the fact that there is a https interface does not imply that there
is an ssh interface. They are handled by different server software, and
unless the provider explicitly set up an ssh server, it won't be there.
And for that matter, having an account on the https server doesn't imply
having an account that you can access via ssh, even if the ssh server is
running.

It sounds like a task for something like mechanize[1], but I don't know
how it is with https. Some others on this list will know, I hope... or
try googling for "ruby mechanize https".
[1] http://mechanize.rubyforge.org/mechanize/

You are right, whereby their reluctance to assist. The site is a good
starting point. Will browse and will be back with the code if I can make
it work. Thanks a million.

 

[Alice Gheorghiu]

Alice Gheorghiu wrote:

Tried ssh and I am getting error as follows in the ssh command:
$ ssh username@https://www.desired_website.com/pageWLinkToReport.php
$ ssh: https://www.desired_website.com/pageWLinkToReport.php: Name or
service not known

If it works at all, it would be something like:
ssh username@desired_website.com

but the fact that there is a https interface does not imply that there
is an ssh interface. They are handled by different server software, and
unless the provider explicitly set up an ssh server, it won't be there.
And for that matter, having an account on the https server doesn't imply
having an account that you can access via ssh, even if the ssh server is
running.

It sounds like a task for something like mechanize[1], but I don't know
how it is with https. Some others on this list will know, I hope... or
try googling for "ruby mechanize https".
[1] http://mechanize.rubyforge.org/mechanize/

You are right, whereby their reluctance to assist. The site is a good
starting point. Will browse and will be back with the code if I can make
it work. Thanks a million.

I was able to work through the code: here it is:
<code>
!#/usr/bin/rb
####################################################
#This program handles files from Vitelity web-site #
#using nokogiri mechanize package #
#Created on 06/09/09 #
#By Alice G. #
####################################################

require 'rubygems'
require 'mechanize'
require 'date'

unless ARGV.size == 3
puts "Usage: Enter username, password and e-mail address"
exit
end

#Define the cdr start and end date
d = DateTime.now

#Create a new object from mechanize class to parse the web pages
a = WWW::Mechanize.new
a.get('http://portal.vitelity.net/login.php') do |page|
# Click the login link
#login_page = a.click(page.links.text(/Log In/))
#login_page = a.click(page.links.find { |link| /Log In/.match
link.text })

# Submit the login form
my_page = page.form_withaction => 'signin.php') do |f|
f.username = ARGV[0]
f.pass = ARGV[1]
end.click_button

# Click the Call Detail Report link on the Vitelity index page
#cdr_page = a.click(my_page.links.find { |link| /Call
Records/.match link.text})
cdr_page = a.get('http://portal.vitelity.net/index.php?p=cdr')
cdr_report=cdr_page.form_withaction => 'index.php') do |f|
f.toemail = ARGV[2]
f.frommonth=d.month
f.fromday = d.day - 1
f.fromyear = d.year
f.tomonth=d.month
f.today=d.day
f.toyear=d.year
end.click_button
end
</code>
The script accepts 3 parameters, username, password to the site and the
e-mail where the file should be sent. To get info on form field names
you need to source the web page and refer to them in your source code.