queues? inotify? anything else?

[Matt Harrison]

I don't even really know how to describe this problem. I know what I
want the end result to be, but I've run into a dead end trying to figure
out possible solutions.

I've got a script which looks for files of a given type in a directory,
processes them, then moves the result elsewhere and removes the sources.

Currently, if I add another file to the directory while the script is
running, it will not be noticed (I'm using a simple directory read).

I would like to be able to add file while the script is busy and have it
append the new filename to its list. I've considered inotify type
monitoring but I don't know much about it and if it could be made to fit
this single-threaded script.

I've also thought about re-reading the directory after each file
operation is complete, and adding the previously unknown filenames to
the list.

I'm just not sure which way I should begin looking, if there's a good
solution for this already out there, or...what.

Thanks for some input

 

[Eric Wong]

I would like to be able to add file while the script is busy and
have it append the new filename to its list. I've considered inotify
type monitoring but I don't know much about it and if it could be
made to fit this single-threaded script.

Inotify descriptors have an internal queue[1] so events can be
backlogged while your script is doing something else.

The sleepy_penguin[2] RubyGem provides inotify bindings that are
subclasses of Ruby IO objects, so you can IO.select on them in
an event loop (sleepy_penguin also provides an epoll wrapper).

sleepy_penguin is Linux-only and a fairly "raw" interface to the
underlying system calls. Cool.io (libev wrapper)[3] and EventMachine[4]
also support inotify and they may be more portable to systems without
inotify (using kqueue or stat() loops).

I've also thought about re-reading the directory after each file
operation is complete, and adding the previously unknown filenames
to the list.

I had to do this once in a daemon since it ran on an NFS directory
since inotify doesn't work on NFS :<


[1] The inotify(7) manpage is a great resource for learning about
how it works.
[2] http://bogomips.org/sleepy_penguin (disclaimer: I'm the author)
[3] http://coolio.github.com/
[4] http://rubyeventmachine.com/

 

[Matt Harrison]

I would like to be able to add file while the script is busy and
have it append the new filename to its list. I've considered inotify
type monitoring but I don't know much about it and if it could be
made to fit this single-threaded script.

Inotify descriptors have an internal queue[1] so events can be
backlogged while your script is doing something else.

The sleepy_penguin[2] RubyGem provides inotify bindings that are
subclasses of Ruby IO objects, so you can IO.select on them in
an event loop (sleepy_penguin also provides an epoll wrapper).

sleepy_penguin is Linux-only and a fairly "raw" interface to the
underlying system calls. Cool.io (libev wrapper)[3] and EventMachine[4]
also support inotify and they may be more portable to systems without
inotify (using kqueue or stat() loops).

I've also thought about re-reading the directory after each file
operation is complete, and adding the previously unknown filenames
to the list.

I had to do this once in a daemon since it ran on an NFS directory
since inotify doesn't work on NFS :<


[1] The inotify(7) manpage is a great resource for learning about
how it works.
[2] http://bogomips.org/sleepy_penguin (disclaimer: I'm the author)
[3] http://coolio.github.com/
[4] http://rubyeventmachine.com/

Thanks Eric,

I'll read up on inotify when I get some time. I'm not quite
understanding how it would work with a single-threaded process. I mean,
if it's busy processing data, I don't see how it could receive events.

I'm sure it'll become more clear as I read up.

Thanks

 

[Thiago Massa]

[Note: parts of this message were removed to make it a legal post.]

Hi!!!

I know what is the problem, PROBABLY.

I've used inotify with node to do the same, the reason why it isn't working
it's because the "inotify script" is probably waiting to a certain event
happen(like a file moved to the directory) and that isn't happening.

When a file is moved to that directory, a chain of events is sent by inotify
like "directory X is open, X file have been moved here, directory X is
closed".

I don't remember correctly, but there's events that:
opens a file to read
opens a file to write
and so on...

So that script is probably waiting for a event that NEVER happens.

Instead of "moving" the file to the directory, try to create a new one, or
PASTE the file and see if it works...

Inotify is simple, if you have the script waste a little of your time
checking what is going on and change the script into your needs.

I hope my answer were helpful to you!

 

[Robert Klemme]

I don't even really know how to describe this problem. I know what I want
the end result to be, but I've run into a dead end trying to figure out
possible solutions.

I've got a script which looks for files of a given type in a directory,
processes them, then moves the result elsewhere and removes the sources.

Currently, if I add another file to the directory while the script is
running, it will not be noticed (I'm using a simple directory read).

I would like to be able to add file while the script is busy and have it
append the new filename to its list. I've considered inotify type monitoring
but I don't know much about it and if it could be made to fit this
single-threaded script.

I've also thought about re-reading the directory after each file operation
is complete, and adding the previously unknown filenames to the list.

In this case you would need to keep track of all files seen already so
you do not add files multiple times.

I'm just not sure which way I should begin looking, if there's a good
solution for this already out there, or...what.

Thanks for some input

Since your script is single threaded anyway and if you do not want to
change too much you could simply wrap your whole script in a loop
which is terminated if the reading of the directory does not find any
matching files any more, e.g.

files = nil

until (files = find_files).empty?
files.each do |f|
process f
move v
end
end

Kind regards

robert

 

[Matt Harrison]

In this case you would need to keep track of all files seen already so
you do not add files multiple times.


Since your script is single threaded anyway and if you do not want to
change too much you could simply wrap your whole script in a loop
which is terminated if the reading of the directory does not find any
matching files any more, e.g.

files = nil

until (files = find_files).empty?
files.each do |f|
process f
move v
end
end

Kind regards

robert

Thanks Robert,

It's such a simple solution but it's always the easy ones that I tend to
overlook.