"Dummy" IO object to push and pull data?

[Shay Hawkins]

Perhaps it would be helpful to distinguish symptoms from your assumed
causes.

Firstly, are you saying that the stdin.puts line never completes? You
need to prove this. I suggest:

STDERR.puts "About to send..."
stdin.puts "Received 0"
STDERR.puts "Finished sending."

Thirdly, because of buffering, the stuff you wrote to stdin might not
actually be sent until you do

stdin.flush

I've actually tried these myself; it doesn't get past the stdin.puts
line, but rather only outputs the first line (where I'm using puts(),
not STDERR.puts()). As for flush, I've tried calling it in a few
different places - before the stdin.puts, after the stdin.puts,
inbetween loops, but it doesn't seem to do anything.

I'll look into the other things you mentioned and post my results.

Of course, if it weren't for the "5 second delay" requirement, then it
could be written quite happily without a thread:

This is true; it works fine without a delay. The delay, unfortunately,
is the crucial aspect that I've been trying to get to work from the
beginning.

 

[Shay Hawkins]

Test program:

# Required support, go to process dir
require("open3.so")
Dir.chdir("..")

# Initiate the process and its streams
command = "********"
inline, outline, errline = Open3.popen3(command)

# Quick function for short timestamps
require("time")
def tstamp(line)
puts("#{Time.now().to_s().split(" ")[3]} #{line}")
end

# Read from stderr, thread stdin input
loop do
tstamp("> Awaiting data...")
line = errline.gets().strip()
tstamp("Output: #{line}")

if(line == "0")
tstamp("> Received the required line.")

Thread.new() do
tstamp("> The thread has been created.")
sleep(5)

tstamp("> Time's up! Sending input...")
inline.puts("EXIT")
tstamp("> The input has been sent.")
end
end
end

Output:

11:35:03 > Awaiting data...
11:35:03 Output: 2
11:35:03 > Awaiting data...
11:35:13 Output: 3
11:35:13 > Awaiting data...
11:35:23 Output: 0
11:35:23 > Received the required line.
11:35:23 > The thread has been created.
11:35:23 > Awaiting data...
11:35:33 Output: 4
11:35:33 > Awaiting data...
11:35:43 Output: 1
11:35:43 > Awaiting data...

 

[Caleb Clausen]

It creates the thread properly, but when the sleep expires and it is
ready to call stdin.puts(), it cannot do anything, because the loop
continued around and it is reading from stderr. However, when it
receives a new set of input from stderr, since the stream is momentarily
being unused, it can then grab the stdin stream and properly input the
old line.

Ah, that's right. On windows, there's a special method you have to use
instead of select to check if input is available on a pipe. Let's see
..... I believe the method you're looking for is kbhit (or _kbhit
maybe? ms likes their underscores...) Check this page on MSDN:
http://msdn.microsoft.com/en-us/library/58w7c94c(VS.80).aspx

There should be some library somewhere that makes this available to
you in rubyland.... Unfortunately, it does mean you have to poll the
pipe for available input instead of letting it notify you.

 

[Brian Candler]

Test program: ...
tstamp("> Time's up! Sending input...")

But that message never appears in the output you showed - and this line
occurs *before* the puts. So at the moment looks either (a) sleep(5) is
sleeping indefinitely, or (b) the whole thread has terminated with an
exception.

A better test would be a standalone pair of ruby programs, because then
you have something which anyone can reproduce the problem with (*).

Here's your test program, very slightly modified, with a partner
program.

==> prg1.rb <==
Thread.abort_on_exception = true
require "open3"

command = "ruby prg2.rb"
inline, outline, errline = Open3.popen3(command)

def tstamp(line)
puts("#{Time.now().to_s().split(" ")[3]} #{line}")
end

# Read from stderr, thread stdin input
loop do
tstamp("> Awaiting data...")
line = errline.gets().strip()
tstamp("Output: #{line}")

if(line == "0")
tstamp("> Received the required line.")

Thread.new() do
tstamp("> The thread has been created.")
sleep(5)

tstamp("> Time's up! Sending input...")
inline.puts("EXIT")
tstamp("> The input has been sent.")
end
end
end

==> prg2.rb <==
i = -2
loop do
$stderr.puts i
i += 1
if select([$stdin],nil,nil,2)
line = $stdin.gets.chomp
$stderr.puts ">>> Received #{line} <<<"
end
end

And if I run it under Linux, here's what I get:

$ ruby prg1.rb
19:58:05 > Awaiting data...
19:58:05 Output: -2
19:58:05 > Awaiting data...
19:58:07 Output: -1
19:58:07 > Awaiting data...
19:58:09 Output: 0
19:58:09 > Received the required line.
19:58:09 > The thread has been created.
19:58:09 > Awaiting data...
19:58:11 Output: 1
19:58:11 > Awaiting data...
19:58:13 Output: 2
19:58:13 > Awaiting data...
19:58:14 > Time's up! Sending input...
19:58:14 > The input has been sent.
19:58:14 Output: >>> Received EXIT <<<
19:58:14 > Awaiting data...
19:58:14 Output: 3
19:58:14 > Awaiting data...
19:58:16 Output: 4
19:58:16 > Awaiting data...
19:58:18 Output: 5
19:58:18 > Awaiting data...
^Cprg1.rb:13:in `gets': Interrupt
from prg1.rb:13
from prg1.rb:11:in `loop'
from prg1.rb:11

What happens if you run this under Windows?

Now, this slightly muddies the water because prg2.rb also depends on
being able to select(), as I wanted it to be able to sleep *and* receive
data on stdin, the same as your main program does. So it would be also
interesting to know what happens if you do "ruby prg2.rb" by itself. It
*should* print a new number every 2 seconds, and if you type something
onto stdin, it should echo it back as >>> Received ... <<<

What I'm trying to say is, if you can make prg2.rb work properly (or
make another of version of prg2 work using threads), then you ought to
be able to make prg1.rb work properly too.

(*) Incidentally: for someone to reproduce this problem, they'll need to
know exactly what platform you have. You said "Windows Vista" but you
didn't say which version of Ruby, nor which package (Cygwin? One-Click
Installer? Other?)

Regards,

Brian.

 

[Caleb Clausen]

Ah, that's right. On windows, there's a special method you have to use
instead of select to check if input is available on a pipe. Let's see
..... I believe the method you're looking for is kbhit (or _kbhit
maybe? ms likes their underscores...) Check this page on MSDN:
http://msdn.microsoft.com/en-us/library/58w7c94c(VS.80).aspx

There should be some library somewhere that makes this available to
you in rubyland.... Unfortunately, it does mean you have to poll the
pipe for available input instead of letting it notify you.

Except when I actually read the msdn link above, I see it only works
with stdin... that may not help you. I think there might be some other
windows-specific call you can make that checks any fd for input
availability... you might have to hunt around to find it, or ask on a
windows programming list.

 

[Shay Hawkins]

What happens if you run this under Windows?

C:\Users\Shay\Desktop>ruby prg1.rb
15:51:20 > Awaiting data...
15:51:21 Output: -2
15:51:21 > Awaiting data...

Then nothing else happens. I'm 90% sure this is because Windows, being
the operating system that it is, holds up everything when .gets() is
called on stdin until something is provided.

Now, this slightly muddies the water because prg2.rb also depends on
being able to select(), as I wanted it to be able to sleep *and* receive
data on stdin, the same as your main program does. So it would be also
interesting to know what happens if you do "ruby prg2.rb" by itself. It
*should* print a new number every 2 seconds, and if you type something
onto stdin, it should echo it back as >>> Received ... <<<

C:\Users\Shay\Desktop>ruby prg2.rb
-2
C:\Users\Shay\Desktop>Hello World!

-1
C:\Users\Shay\Desktop>Blah~

0

And so forth and so on.

(*) Incidentally: for someone to reproduce this problem, they'll need to
know exactly what platform you have. You said "Windows Vista" but you
didn't say which version of Ruby, nor which package (Cygwin? One-Click
Installer? Other?)

C:\Users\Shay\Desktop>ruby -v
ruby 1.8.6 (2008-08-11 patchlevel 287) [i386-mswin32]

The program I used to install it (or at least what I have in my
downloads folder) is ruby186-27_rc2 from rubyinstaller.rubyforge.org.

 

[Brian Candler]

C:\Users\Shay\Desktop>ruby prg2.rb
-2
C:\Users\Shay\Desktop>Hello World!
0

And so forth and so on.

It returns you to a command prompt? How odd.

What about a threaded version of prg2.rb, does it behave any
differently?

Thread.new do
while line = $stdin.gets
$stderr.puts ">>> Received #{line.chomp} <<<"
end
end
i = -2
loop do
$stderr.puts i
i += 1
sleep 2
end

 

[Shay Hawkins]

It returns you to a command prompt? How odd.

What about a threaded version of prg2.rb, does it behave any
differently?

Thread.new do
while line = $stdin.gets
$stderr.puts ">>> Received #{line.chomp} <<<"
end
end
i = -2
loop do
$stderr.puts i
i += 1
sleep 2
end

Oh, my apologies, I had put the user info there to show that I was
giving input (as if @echo hadn't been turned off). As for that program,
it behaves almost the same way - locking up until user gives input, then
displaying i if the 2 seconds have expired.

 

[Brian Candler]

Oh, my apologies, I had put the user info there to show that I was

giving input (as if @echo hadn't been turned off). As for that program,
it behaves almost the same way - locking up until user gives input, then
displaying i if the 2 seconds have expired.

OK. I've now got a Windows XP box beside me. I tried the threaded
version of prg2.rb with:

(1) rubyinstaller-1.8.6-p383-rc1.exe from
rubyforge.org/projects/rubyinstaller
(this is about a year newer than the 186-27_rc2 you have)

(2) cygwin-1.7.1 which has ruby-1.8.7-p72-2 package under 'interpreters'

(3) ruby 1.8.7-p72 binary from www.ruby-lang.org/en/downloads/

The first fails in the way you described: that is, the sleeping thread
doesn't "wake up" until some input has been given to stdin.

However the second and third work just fine, in the same way as it does
under Linux. That is, I see a new number every 2 seconds, and I can type
in input whenever I like and get it echoed back immediately.

So clearly the problem is with how ruby is built in the one-click
installer. You may wish to raise a ticket against it, using this as a
demonstration of the problem. But in any case there are two other
working versions of ruby you can use instead.

HTH,

Brian.

 

[Robert Klemme]

My apologies: I'll see if I can elaborate a bit...

Well, I originally was trying to find a workaround to this, but it seems
the root of the problem is in the process' streams. I have a seperate
program that I did not create myself (and therefore cannot change the
code of) - this program puts output into stderr, and accepts input
through stdin.

I call Open3.popen3 to create the three streams. In a loop, I want to
constantly read input from stderr. (Assuming I assigned the pipes to
stdin, stdout, and stderr variable names.)

loop do
line = stderr.gets()
puts("Output: #{line}")
end

Then, if I receive an output of say, 0, I want to input something to the
program through stdin five seconds later.

loop do
line = stderr.gets().strip()
puts("Output: #{line}")

if(line == "0")
Thread.new() do
sleep(5)
stdin.puts("Received 0 five seconds ago.")
end
end
end

It creates the thread properly, but when the sleep expires and it is
ready to call stdin.puts(), it cannot do anything, because the loop
continued around and it is reading from stderr.

That can't normally be, because the thread runs independently of the
loop. I am rather suspecting that you are having an issue with not
waiting for all threads to proper finish or do not make sure pipes are
read properly. You could also have created a dead lock that way, i.e.
by not reading all pipes properly because a process that tries to write
to a filled pipe is blocked.

I have created a pair of test scripts which I will place below.
Strangely enough, there seems to be an issue with detecting closing of
the stderr pipe which I need to research separately. If you just invoke
the client script with "./clnt.rb 15 x" you will see the interaction
properly working. If you, for example, increase the number and omit the
reading of sin in the client the server will eventually block in one of
the puts or printf statements. It may be that this is what you are
observing.

Kind regards

robert


Place both files in the same directory.

file clnt.rb:
#!/usr/local/bin/ruby19

# "client" which starts the server and every time a
# zero is read from the server's stderr pipe we send
# a notification to the server's stdin.
# server's stdout is used for logging state and is mirrored
# to this process's stdout with prefix "From server".

require 'open3'

# for debugging
Thread.abort_on_exception = true

Open3.popen3 File.join(File.dirname($0), "serv.rb"), *ARGV do |sin,
sout, serr|
sin.sync = true

# we must make sure all pipes are either read or closed!
r = Thread.new do
sout.each do |line|
puts "From server: #{line}"
end
end

threads = []

serr.each do |line|
printf "%-20s read %p\n", Time.now, line

line.chomp!

if line == "0"
threads << Thread.new do
sleep 5
sin.puts "Received 0 five seconds ago."
end
elsif /finish/i =~ line
# interestingly enough EOF detection does not
# work with serv.rb
break
end
end

puts "finished reading"

threads.each {|th| th.join}

puts "finished notifying"

# init shutdown sequence
sin.close
r.join
end


file serv.rb:
#!/usr/local/bin/ruby19

# "server" process which writes a fixed amount of numbers
# to stderr and then closes stderr to indicate it's finished.
# first arg is no of repetitions
# second arg if present says indicate end with a particular
# message to stderr.

$stdout.sync = $stderr.sync = true

puts "started"

th = Thread.new do
$stdin.each do |line|
printf "%-30s read %p\n", Time.now, line
end
end

rep = (ARGV.shift || 10).to_i
ind = ARGV.shift

rep.times do
$stderr.puts rand(3)
sleep 1
end

puts "finished writing"

# indicate we are done
$stderr.puts "Finish" if ind
$stderr.close
puts "stderr.closed? #{$stderr.closed?}"

puts "finishing"

th.join

puts "finished"

 

[Brian Candler]

That can't normally be, because the thread runs independently of the
loop.

He's right, and I have replicated the problem. It occurs with with 1.8.6
one-click installer under Windows. With this interpreter, threads are
borked: when one thread is waiting on I/O, it prevents all other threads
from running.

The "official" 1.8.7 Windows build is OK, and so is a cygwin 1.8.7.

You can replicate the problem with this:

---------------------------------------
Thread.new do
while line = $stdin.gets
$stderr.puts ">>> Received #{line.chomp} <<<"
end
end
i = -2
loop do
$stderr.puts i
i += 1
sleep 2
end

 

[Phillip Gawlowski]

He's right, and I have replicated the problem. It occurs with with 1.8.6
one-click installer under Windows. With this interpreter, threads are
borked: when one thread is waiting on I/O, it prevents all other threads
from running.

The "official" 1.8.7 Windows build is OK, and so is a cygwin 1.8.7.

You can replicate the problem with this:

---------------------------------------
Thread.new do
while line = $stdin.gets
$stderr.puts ">>> Received #{line.chomp}<<<"
end
end
i = -2
loop do
$stderr.puts i
i += 1
sleep 2
end

For the heck of it, I've tested 1.8.6 and 1.9.1 compiled with MinGW:
PS C:\Scripts> ruby -v
ruby 1.9.1p243 (2009-07-16 revision 24175) [i386-mingw32]
PS C:\Scripts> c:\ruby\bin\ruby -v
ruby 1.8.6 (2009-08-04 patchlevel 383) [i386-mingw32]

Ruby 1.9.1 performs as expected: The threads start, and i gets
incremented every 2 seconds:
PS C:\Scripts> ruby .\thread.rb
-2
-1
0
1
2
3
14
1

5
6
:q

7
8


Same script with 1.8.6:
PS C:\Scripts> c:\ruby\bin\ruby .\thread.rb
-2

-1
2 0
:q

1
2

(Yeah, vim user here...)

It looks like 1.8.7 introduced (or rather got backported from 1.9) a
change that makes threads work independent from the underlying OS.

 

[Robert Klemme]

C:\Users\Shay\Desktop>ruby prg1.rb
15:51:20 > Awaiting data...
15:51:21 Output: -2
15:51:21 > Awaiting data...

Then nothing else happens. I'm 90% sure this is because Windows, being
the operating system that it is, holds up everything when .gets() is
called on stdin until something is provided.

Actually part of that functionality is in Ruby itself because IO uses
buffering by default. For tests like these it helps to place a line
like this at the beginning of the script:

$stdout.sync = $stderr.sync = true

As far as I can see this wasn't done in Brian's test scripts so
repeating with that line introduced will likely yield more accurate results.

Kind regards

robert

 

[Brian Candler]

For tests like these it helps to place a line
like this at the beginning of the script:

$stdout.sync = $stderr.sync = true

I agree that's a good idea, but I don't think it actually makes a
difference here.

For one thing, the first output line ("-2") was shown on the terminal
immediately, so there's no reason why the second line ("-1") should not
be.

For another thing, the same program worked fine on the two 1.8.7
platforms but not the 1.8.6, on the same machine.

And finally, $stderr is usually unbuffered anyway (on Unix; I know I
can't make such assumptions about Windows).

 

[Shay Hawkins]

(3) ruby 1.8.7-p72 binary from www.ruby-lang.org/en/downloads/

Ah, that'd be it. Everything works fine now - Thank you all for your
large amount of help.