Automating FTP file transfers

[Limey Drink]

Hi all,

Firstly , is there any where I can search through archived newsgroup
posts so I am not cluttering up the newsgroup with repeated queries ?

And secondly , I know this has probably been discussed before but.

I am wanting to do some scripting to automate a few administration tasks,
one of the first tasks is automating FTP file transfers.

I would like to know if the following could be made more robust using python
and its FTP libraries rather than executing native OS commands in a shell
script.

Basically I need to...
1. Check on the local system for new files in a specific directory
If new files exist then...
2. Connect to a remote FTP server.
3. Transfer local files to a specific directory on the remote FTP server.
4. Then remove/archive local files and end session.

This is critical operation and though while in the past I have written DOS
scripts etc. to do simple tasks I have not needed to check for errors as
they weren't absolutely critical tasks. I am going to need to run this
overnight using AT or CRON then I need to know about file transfer errors
etc. and other problems that f I were actually performing the FTP transfer
manually I would get feedback of the problem and could re-try the operation.

Any help would be very appreciated.

 

[Rene Pijlman]

Limey Drink:

is there any where I can search through archived newsgroup posts
http://groups.google.com

I would like to know if the following could be made more robust using python
and its FTP libraries rather than executing native OS commands in a shell
script.

Basically I need to... [upload a file]

Sure. Use ftplib. See
http://www.python.org/doc/2.3.2/lib/module-ftplib.html

Here's a code snippet from one of my scripts.

import os, ftplib
ftp = ftplib.FTP(Hostname,Username,Password)
ftp.cwd(WorkingDirectory)
ftp.storbinary("STOR " + RemoteZipFile, file(LocalZipFile, "rb"))
ftp.quit()
os.remove(LocalZipFile)

 

[Matt Goodall]

Hi all,

Firstly , is there any where I can search through archived newsgroup
posts so I am not cluttering up the newsgroup with repeated queries ?

Yep, info on all the python lists, including archives, are available here:

http://python.org/community/lists.html

And secondly , I know this has probably been discussed before but.

I am wanting to do some scripting to automate a few administration tasks,
one of the first tasks is automating FTP file transfers.

I would like to know if the following could be made more robust using python
and its FTP libraries rather than executing native OS commands in a shell
script.

Basically I need to...
1. Check on the local system for new files in a specific directory
If new files exist then...
2. Connect to a remote FTP server.
3. Transfer local files to a specific directory on the remote FTP server.
4. Then remove/archive local files and end session.

Most of this should be fairly straightforward. (1) and (4) should be
covered using the os and os.path modules; for (2) and (3) you need the
ftplib module.

The only problem you are likely to encounter is getting a remote
directory listing. The FTP protocol neglected to formalise the response
to FTP's LIST command and different servers send different results back
. If this is a problem I would recommend using one of the ftpparse
Python APIs:

* http://effbot.org/downloads/#ftpparse
* http://c0re.23.nu/c0de/ftpparsemodule/ftpparse.html

This is critical operation and though while in the past I have written DOS
scripts etc. to do simple tasks I have not needed to check for errors as
they weren't absolutely critical tasks. I am going to need to run this
overnight using AT or CRON then I need to know about file transfer errors
etc. and other problems that f I were actually performing the FTP transfer
manually I would get feedback of the problem and could re-try the operation.

The ftplib module reports errors using the Python exception mechanism so
you can make it as robust as you need and take whatever action is
appropriate on errors.

Hope this helps.

Cheers, Matt

 

[Limey Drink]

Thanks very much for the help guys I now know how to connect using python
ftp library and I am happy with this but...

Just wondering about the bigger picture with regards to sys admin scripts,
in your opinion, is creating these type of system administration scripts
preferable/easier than using say bash scripts, or DOS .bat scripts ?

Because I don't have alot of experience of sys admin and shell scripting I'm
just wondering if I am applying Python for one of its intended uses doing
these type of scripts.

IMHO i would rather learn and stick with Python than to learn all the
different scripting languages as it is cleaner and more powerful and also if
the script is written correctly it seems could be cross platform.

Just wondering if you think I should be trying to execute the in built OS
programs such as FTP etc. calling them from python or should I be using the
libraries that come with Python such as FTPlib and say if I wanted to send
email should I be using the SMTP lib rather than say executing the mail
program in unix and passing arguments from a Python script.

It would hear interesting to hear your views on this as it would be nice if
I can gain from others experience and not make the same mistakes.

Thanks again

 

[Cameron Laird]

Thanks very much for the help guys I now know how to connect using python
ftp library and I am happy with this but...

Just wondering about the bigger picture with regards to sys admin scripts,
in your opinion, is creating these type of system administration scripts
preferable/easier than using say bash scripts, or DOS .bat scripts ?

Because I don't have alot of experience of sys admin and shell scripting I'm
just wondering if I am applying Python for one of its intended uses doing
these type of scripts.

IMHO i would rather learn and stick with Python than to learn all the
different scripting languages as it is cleaner and more powerful and also if
the script is written correctly it seems could be cross platform.

Just wondering if you think I should be trying to execute the in built OS
programs such as FTP etc. calling them from python or should I be using the
libraries that come with Python such as FTPlib and say if I wanted to send
email should I be using the SMTP lib rather than say executing the mail
program in unix and passing arguments from a Python script.

It would hear interesting to hear your views on this as it would be nice if
I can gain from others experience and not make the same mistakes.

.
.
.
Yes.

That is, I take very seriously this question of what tools
are best for sysads; I'll be running a series on the subject
most of next year in *System Administration* magazine. My
short answer (to which I might return later in the week) is
that, yes, Python makes a great vehicle, particularly be-
cause its networking is so much easier *and* more potent
than what bash, .bat, and so on offer. While there are
counter-arguments, I think you'll find Python quite satisfy-
ing, and you will *not* be tempted to return to bash and
friends.

 

[Rene Pijlman]

Limey Drink:

Just wondering about the bigger picture with regards to sys admin scripts,
in your opinion, is creating these type of system administration scripts
preferable/easier than using say bash scripts, or DOS .bat scripts ?

Yes! You get real data structures, classes, exception handling, powerful
libraries and such.

For example, you can write system administration scripts in Python that
use a relational database, parse XML config files, send e-mail,
up/download with FTP ...

Just wondering if you think I should be trying to execute the in built OS
programs such as FTP etc. calling them from python

No. Why would you want to do that? Only as a last resort.

or should I be using the libraries that come with Python such as FTPlib and
say if I wanted to send email should I be using the SMTP lib

Yes. Otherwise you'll loose the exception handling, classes, data
structures etc.

 

[Limey Drink]

OK then, case closed, Python it is

Its just nice to have your gut feeling backed up by other peoples
experience.

Thanks for all the help guys.

 

[Jamey Cribbs]

Just wondering about the bigger picture with regards to sys admin scripts,
in your opinion, is creating these type of system administration scripts
preferable/easier than using say bash scripts, or DOS .bat scripts ?

Just wondering if you think I should be trying to execute the in built OS
programs such as FTP etc. calling them from python or should I be using the
libraries that come with Python such as FTPlib and say if I wanted to send
email should I be using the SMTP lib rather than say executing the mail
program in unix and passing arguments from a Python script.

Just to give you my $.02 on this. I have literally dozens of Python ftp
scripts that run on a Linux server at my company daily via cron. All of
my scripts use ftplib and smtplib. My basic script structure is:

1). Check local directory to see if files are available for sending.
2). Encrypt local files via PGP.
3). FTP files to external vendor using ftplib.
4). Send "success" email to myself and other interested parties via
smtplib.
5). If the script error's out instead, send error email to me so I know
there is a problem.

IMO, Python is an excellent choice for this kind of stuff.

Hope this helps!

Jamey