download a text file from a URL

[vivekmahanta]

Hey guys,

I want to download a text file from a URL and store its contents in my
webhosting server. Eg, if the URL is something like http://www.yahoo.com/file.txt
then I want to download file.txt and then store its contents. How can
I do that in Perl ?? Any help will be really appreciated. Please reply
soon.

Thank you

Vivek

 

[John Bokma]

Hey guys,

I want to download a text file from a URL and store its contents in my
webhosting server. Eg, if the URL is something like
http://www.yahoo.com/file.txt then I want to download file.txt and
then store its contents. How can I do that in Perl ?? Any help will be
really appreciated.

check out LWP::UserAgent and friends

 

[tfe]

You can use LWP::Simple with method getstore:

getstore($url, $file);

 

[gf]

Or...

use LWP::UserAgent;

my $ua = LWP::UserAgent->new;

my $resp = $ua->get('http://url', './file_to_save_to');




There are so many ways to get there that it must be Perl.

 

[vivekmahanta]

Or...

use LWP::UserAgent;

my $ua = LWP::UserAgent->new;

my $resp = $ua->get('http://url', './file_to_save_to');

There are so many ways to get there that it must be Perl.

thank you so much for all the tips. Now, I want to create Cron Jobs
that will keep downloading such files from the URLs every 5 mins or
so. How do I create the scripts for doing that which can be later fed
to a Cron Job setup menu ?? I think after I have written the scripts
in Perl I can definitely convert them into executables which can be
set up as Cron Jobs. Can you tell me how that can be done ?? Any help
is really appreciated.

thank you,

Vivek

 

[John Bokma]

thank you so much for all the tips. Now, I want to create Cron Jobs
that will keep downloading such files from the URLs every 5 mins or
so. How do I create the scripts for doing that which can be later fed
to a Cron Job setup menu ?? I think after I have written the scripts
in Perl I can definitely convert them into executables which can be
set up as Cron Jobs. Can you tell me how that can be done ?? Any help
is really appreciated.

You might want to check out wget or curl.

 

[gf]

On Feb 6, 1:44 pm, "(e-mail address removed)" <[email protected]>
wrote:
[...]

How do I create the scripts for doing that which can be later fed
to a Cron Job setup menu ?? I think after I have written the scripts
in Perl I can definitely convert them into executables which can be
set up as Cron Jobs. Can you tell me how that can be done ?? Any help
is really appreciated.

Convert to executables? Not with Perl. It's a scripting language and
has no compiler output like C/C++ or assembler. You can set a script
to be executable, but it remains a script. (Yes there are converted
core-dumps but that's just a veneer.)

Once you have a Perl script for downloading one file you can easily
have cron launch it at intervals. Read the documentation for cron and
you'll be done since cron does pretty much anything that you do from
the command line.

Whether to download every time is an additional question you need to
consider. From Perl you can check and make a decision to skip the
get() if the file hasn't changed - see the head() command in LWP - but
you might want to consider using wget or one of the other apps that
have been invented to solve this sort of problem. If you're mirroring
a file you could easily add the --mirror parameter and be done with
it.

Sometimes it's a lot easier to use wget or curl rather than code from
scratch, but we get wrapped up in doing it ourselves.

 

[vivekmahanta]

On Feb 6, 1:44 pm, "(e-mail address removed)" <[email protected]>
wrote:
[...]

How do I create the scripts for doing that which can be later fed
to a Cron Job setup menu ?? I think after I have written the scripts
in Perl I can definitely convert them into executables which can be
set up as Cron Jobs. Can you tell me how that can be done ?? Any help
is really appreciated.

Convert to executables? Not with Perl. It's a scripting language and
has no compiler output like C/C++ or assembler. You can set a script
to be executable, but it remains a script. (Yes there are converted
core-dumps but that's just a veneer.)

Once you have a Perl script for downloading one file you can easily
have cron launch it at intervals. Read the documentation for cron and
you'll be done since cron does pretty much anything that you do from
the command line.

Whether to download every time is an additional question you need to
consider. From Perl you can check and make a decision to skip the
get() if the file hasn't changed - see the head() command in LWP - but
you might want to consider using wget or one of the other apps that
have been invented to solve this sort of problem. If you're mirroring
a file you could easily add the --mirror parameter and be done with
it.

Sometimes it's a lot easier to use wget or curl rather than code from
scratch, but we get wrapped up in doing it ourselves.

Actually, I have to download a lot of files every 5 mins or so. I have
written such programs in java and C++ already, but I am not sure how I
can use Cron Jobs to execute them every 5 mins. I can write a similar
program in Perl or any other language too, but I want it to be fast
and reliable. Can you suggest what language I should use ??

Vivek

 

[J. Gleixner]

Actually, I have to download a lot of files every 5 mins or so. I have
written such programs in java and C++ already, but I am not sure how I
can use Cron Jobs to execute them every 5 mins. I can write a similar
program in Perl or any other language too, but I want it to be fast
and reliable. Can you suggest what language I should use ??

The language doesn't matter! Use the language or tools that you
know and move on.

Talk to your administrator about using cron, or look at
the documentation: man crontab Discussing how to use cron
isn't the purpose of this newsgroup, try one dealing with
your particular OS.

Also, if it's your machine and you have to "download a lot
of files", use something like rsync (no.. I won't discuss
how to use it here.. search the Internet..).