FTP site upload

[Roedy Green]

The standard FTP software for uploading files leaves a few things to
be desired.


1. I can't upload a file if someone is busy reading it. This aborts
my run.

2. The update should be atomic. If the upload takes a hour, I don't
want the public during that hour seeing half old and half new files.
None of the new files should be visible until after the upload is
complete and verifiably complete.

3. if I disconnect or crash, the program should on it own reliably
notice it has crashed and pick up where it left off.

4. The program should not be confused by manual uploads or deletes.


Is there anything to do this?
--
Bush crime family lost/embezzled $3 trillion from Pentagon.
Complicit Bush-friendly media keeps mum. Rumsfeld confesses on video.
http://www.infowars.com/articles/us/mckinney_grills_rumsfeld.htm

Canadian Mind Products, Roedy Green.
See http://mindprod.com/iraq.html photos of Bush's war crimes

 

[Lucy]

The standard FTP software for uploading files leaves a few things to
be desired.


1. I can't upload a file if someone is busy reading it. This aborts
my run.

2. The update should be atomic. If the upload takes a hour, I don't
want the public during that hour seeing half old and half new files.
None of the new files should be visible until after the upload is
complete and verifiably complete.

3. if I disconnect or crash, the program should on it own reliably
notice it has crashed and pick up where it left off.

4. The program should not be confused by manual uploads or deletes.


Is there anything to do this?

Yes, but it involves tweasers and a blow torch. ;-)

 

[John Currier]

Bullet 2 is somewhat achievable by uploading to a temporary location
and moving the file to it's final location after it's all on the
server. This assumes, of course, that the temporary and final
locations are on the same filesystem. If it's not the same filesystem
(and depending on the OS) it could take more than a fraction of a
second to do the move and not be somewhat atomic.

Depending on the OS Bullet 1 might require retries of the move
operation. On AIX, for example, I can move a file (on the same
filesystem) while it is being written to and the program will continue
to write to it in its new location without missing a beat.

John
http://schemaspy.sourceforge.net

 

[Nigel Wade]

The standard FTP software for uploading files leaves a few things to
be desired.


1. I can't upload a file if someone is busy reading it. This aborts
my run.

2. The update should be atomic. If the upload takes a hour, I don't
want the public during that hour seeing half old and half new files.
None of the new files should be visible until after the upload is
complete and verifiably complete.

3. if I disconnect or crash, the program should on it own reliably
notice it has crashed and pick up where it left off.

4. The program should not be confused by manual uploads or deletes.


Is there anything to do this?

I'm not sure if there is anything which can achieve those stated goals, but
you might be able to put something together which approaches it.

The normal tool used for mirroring FTP/Web sites etc. is rsync. It's not
something I do, so I've never used it.

I don't know what you mean by 4.

 

[Roedy Green]

I don't know what you mean by 4.

I use Netload which keeps a local copy of what it THINKS is on the
server. If I go behind its back and upload a file with another tool
or delete something with another tool, it gets horribly confused.

For example, it tries to delete directories that are already deleted
and goes into a funk. It should just resynch itself without further
fuss.



--
Bush crime family lost/embezzled $3 trillion from Pentagon.
Complicit Bush-friendly media keeps mum. Rumsfeld confesses on video.
http://www.infowars.com/articles/us/mckinney_grills_rumsfeld.htm

Canadian Mind Products, Roedy Green.
See http://mindprod.com/iraq.html photos of Bush's war crimes

 

[Roedy Green]

Bullet 2

Have you an URL for that. They chose a name unfriendly to google.

--
Bush crime family lost/embezzled $3 trillion from Pentagon.
Complicit Bush-friendly media keeps mum. Rumsfeld confesses on video.
http://www.infowars.com/articles/us/mckinney_grills_rumsfeld.htm

Canadian Mind Products, Roedy Green.
See http://mindprod.com/iraq.html photos of Bush's war crimes

 

[John Currier]

What I meant by "Bullet 2" was your second "bullet point" about atomic
updates. I'm sorry if I wasn't clear. It's something that you have to
manage yourself. There are quite a few companies that will do all of
this for you (I'm helping to review several secure FTP RFPs now), but
they cost /big/ bucks.

John

 

[Owen Jacobson]

I use Netload which keeps a local copy of what it THINKS is on the server.
If I go behind its back and upload a file with another tool or delete
something with another tool, it gets horribly confused.

For example, it tries to delete directories that are already deleted and
goes into a funk. It should just resynch itself without further fuss.

Subversion.

No, really. The features you've described mirror almost exactly the
features of any decent version control system. The approach I'd take is
to have a repository with
$ROOT/publish
as the version visible on the web and
$ROOT/trunk, $ROOT/braches, etc as working paths.

To publish something, it's just a case of
svn rm $ROOT/publish; svn cp $ROOT/trunk $ROOT/publish

(This might be achievable though merge instead of rm;cp. Read the docs.)

and then cause the copy in the website to be updated (either by scheduling
regular updates from the repository, or using svn's trigger scripts to
automatically update the site when publish is checked in and not
being removed.

To get the kind of non-confusability you want, you'd have to have the
disclipline to always add things to trunk and then publish them,
regardless of where you're adding them from.

Owen

 

[Roedy Green]

Subversion.

No, really. The features you've described mirror almost exactly the
features of any decent version control system.

I could see that if my clients were viewing with Subversion. But what
has to happen is an HTTP request comes in and something has to convert
it to look as if it were a subversion request to view the latest
version. I gather I would have to write that piece, or do some batch
process after an update was complete to refresh the ordinary HTML
copies.

--
Bush crime family lost/embezzled $3 trillion from Pentagon.
Complicit Bush-friendly media keeps mum. Rumsfeld confesses on video.
http://www.infowars.com/articles/us/mckinney_grills_rumsfeld.htm

Canadian Mind Products, Roedy Green.
See http://mindprod.com/iraq.html photos of Bush's war crimes

 

[Owen Jacobson]

I could see that if my clients were viewing with Subversion. But what has
to happen is an HTTP request comes in and something has to convert it to
look as if it were a subversion request to view the latest version. I
gather I would have to write that piece,

Possible, but frightening and likely slow as all get-out.

or do some batch process after an update was complete to refresh the
ordinary HTML copies.

That's pretty much implicit in the "transactional" part. It's hard to do
really transactional things with a filesystem without doing some batch
moving from a working version to the "committed" version at some point.
(Under the hood, even subversion does just that when you update the
working copy.) FTP certainly doesn't have any capabilities in that
direction, as it has no association between different files uploaded in
the same session to build a transaction with.

Of course, a solution based on version control hinges on being able to run
programs on the host, which some providers don't give you.

It's just a suggestion,
Owen