interactively processing a line at a time

[Forrest Chang]

HI All:

For the following code segement running on win32 ruby (1 click installer)

sout = IO.popen( "output");
while( line = sout.gets)
do_something_with( line)
end


If for example, "output" printed out something 5 times every second,
then on the command line, I would see something printed out every
second, 5 times. Running through the above code, I would not see
anything until after 5 secs, which I assume that after the program
exits, the output is flushed, and then it would be all 5 entries. I
ran the same sort of code successfully solaris, so I expected it to
work on windows.

I did an experiment, and wrote a ruby program that printed a
line then flushed STDOUT, paused, repeated, etc., and it works as
suspected. Since I do not have control over the actual "output"
program, how can I get it so that each line is flushed? sout.sync =
true didn't work (I imagine it only works if I write to sout)

Any ideas?

Thanks

FOrrest

 

[Ara.T.Howard]

HI All:

For the following code segement running on win32 ruby (1 click installer)

sout = IO.popen( "output");
while( line = sout.gets)
do_something_with( line)
end


If for example, "output" printed out something 5 times every second,
then on the command line, I would see something printed out every
second, 5 times. Running through the above code, I would not see
anything until after 5 secs, which I assume that after the program
exits, the output is flushed, and then it would be all 5 entries. I
ran the same sort of code successfully solaris, so I expected it to
work on windows.

I did an experiment, and wrote a ruby program that printed a
line then flushed STDOUT, paused, repeated, etc., and it works as
suspected. Since I do not have control over the actual "output"
program, how can I get it so that each line is flushed? sout.sync =
true didn't work (I imagine it only works if I write to sout)

Any ideas?

try this - it should/might work on windoze - i assume it inherits open file
handles from the parent. i ran it on xp and it seemed fine:

harp:~ > ruby a.rb
[Fri Aug 05 20:49:51 MDT 2005, "42\n"]
[Fri Aug 05 20:49:56 MDT 2005, "42\n"]
[Fri Aug 05 20:50:01 MDT 2005, "42\n"]
[Fri Aug 05 20:50:06 MDT 2005, "42\n"]
[Fri Aug 05 20:50:11 MDT 2005, "42\n"]
...
...
...


harp:~ > cat a.rb
require 'tempfile'

def popen_sync cmd
tf = Tempfile::new $PROGRAM_NAME
tf.puts <<-ruby_code
STDOUT.sync = true
exec STDIN.read
ruby_code
tf.close

sync_cmd = "ruby #{ tf.path }"
pipe = IO:open sync_cmd, 'r+'
pipe.puts cmd
pipe.close_write
pipe
end

output = "ruby -e 'loop{ p 42; sleep 5}'"
sout = popen_sync output

while(line = sout.gets)
p [Time::now, line]
STDOUT.flush
end


again - this will only work if exce inherits open files and there properties...

hth.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| My religion is very simple. My religion is kindness.
| --Tenzin Gyatso
===============================================================================

 

[Forrest Chang]

HI Ara:

That did the trick. Thanks.

BTW, nice article on Grids w/RUby and SQLite

Forrest

HI All:

For the following code segement running on win32 ruby (1 click installer)

sout = IO.popen( "output");
while( line = sout.gets)
do_something_with( line)
end


If for example, "output" printed out something 5 times every second,
then on the command line, I would see something printed out every
second, 5 times. Running through the above code, I would not see
anything until after 5 secs, which I assume that after the program
exits, the output is flushed, and then it would be all 5 entries. I
ran the same sort of code successfully solaris, so I expected it to
work on windows.

I did an experiment, and wrote a ruby program that printed a
line then flushed STDOUT, paused, repeated, etc., and it works as
suspected. Since I do not have control over the actual "output"
program, how can I get it so that each line is flushed? sout.sync =
true didn't work (I imagine it only works if I write to sout)

Any ideas?

try this - it should/might work on windoze - i assume it inherits open file
handles from the parent. i ran it on xp and it seemed fine:

harp:~ > ruby a.rb
[Fri Aug 05 20:49:51 MDT 2005, "42\n"]
[Fri Aug 05 20:49:56 MDT 2005, "42\n"]
[Fri Aug 05 20:50:01 MDT 2005, "42\n"]
[Fri Aug 05 20:50:06 MDT 2005, "42\n"]
[Fri Aug 05 20:50:11 MDT 2005, "42\n"]
...
...
...


harp:~ > cat a.rb
require 'tempfile'

def popen_sync cmd
tf = Tempfile::new $PROGRAM_NAME
tf.puts <<-ruby_code
STDOUT.sync = true
exec STDIN.read
ruby_code
tf.close

sync_cmd = "ruby #{ tf.path }"
pipe = IO:open sync_cmd, 'r+'
pipe.puts cmd
pipe.close_write
pipe
end

output = "ruby -e 'loop{ p 42; sleep 5}'"
sout = popen_sync output

while(line = sout.gets)
p [Time::now, line]
STDOUT.flush
end


again - this will only work if exce inherits open files and there properties...

hth.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| My religion is very simple. My religion is kindness.
| --Tenzin Gyatso
===============================================================================

 

[Forrest Chang]

Hi Ara:

I tried tweaking your code to use IO.win32_popen3 coz I wanted to
process stderr as well and we're back to the "doesn't look like
stdout/stderr are being flushed". Any additional ideas?

Thanks again

Forrest

 

[Ara.T.Howard]

HI Ara:

That did the trick. Thanks.

great. i wouldn't launch rockets with it ;-)

BTW, nice article on Grids w/RUby and SQLite

thanks! it's been really good for us - millions of jobs run w/o a single
incident or bug for nearly 9 months of 24x7 operation - it's amazing what a
garbage collector can do ;-)

cheers.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| Your life dwells amoung the causes of death
| Like a lamp standing in a strong breeze. --Nagarjuna
===============================================================================

 

[Ara.T.Howard]

Hi Ara:

I tried tweaking your code to use IO.win32_popen3 coz I wanted to
process stderr as well and we're back to the "doesn't look like
stdout/stderr are being flushed". Any additional ideas?

Thanks again

Forrest

here's something using open4:


require 'tempfile'
require 'open4'

def maim pid, opts = {}
#--{{{
sigs = opts['signals'] || opts[:signals] || %w(SIGTERM SIGQUIT SIGKILL)
suspend = opts['suspend'] || opts[:suspend] || 4
pid = Integer "#{ pid }"
existed = false
sigs.each do |sig|
begin
Process::kill sig, pid
existed = true
rescue Errno::ESRCH
unless existed
return nil
else
return true
end
end
return true unless alive?(pid)
sleep suspend
return true unless alive?(pid)
end
return(not alive?(pid))
#--}}}
end
def alive? pid
#--{{{
pid = Integer "#{ pid }"
begin
Process::kill 0, pid
true
rescue Errno::ESRCH
false
end
#--}}}
end
def popen_sync cmd
#--{{{
tf = Tempfile::new $PROGRAM_NAME
tf.puts <<-ruby_code
STDOUT.sync = STDERR.sync = true
exec STDIN.read
ruby_code
tf.close

sync_cmd = "ruby #{ tf.path }"

pid, stdin, stdout, stderr = Open4:open4 sync_cmd

stdin.puts cmd
stdin.close

if block_given?
begin
yield [pid, stdout, stderr]
ensure
maim pid
end
else
at_exit{ maim pid rescue nil }
[pid, stdout, stderr]
end
#--}}}
end

output_program = "ruby -e 'loop{ p 42; sleep 5}'"

popen_sync(output_program) do |pid, stdout, stderr|
while(line = stdout.gets)
puts "stdout => <#{ line.strip }>"
STDOUT.flush
end
while(line = stderr.gets)
puts "stderr => <#{ line.strip }>"
STDOUT.flush
end
end


handling stdout and stderr simoultaneously on windows will be tricky though...
see many threads on this list for that.

cheers.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| Your life dwells amoung the causes of death
| Like a lamp standing in a strong breeze. --Nagarjuna
===============================================================================