working with a slow pipe (IO.popen)

[bwv549]

I'm doing some work with pymol and having trouble getting all the
pymol output out:

# I open it in quiet/commandline mode with a pipe:
IO.popen("pymol -cq -p", 'w+') do |pipe|
pipe.puts "load file.pdb, mymodel\n"
pipe.puts "run my_script.py"
# this command will generate a whole bunch of output to stdout
pipe.puts "my_script mymodel"
sleep(5) # <--- this is what I have to do in order to get the
output
pipe.close_write
output = pipe.read
end

What is annoying is that unless I sleep for 2-5 seconds I don't get
output or (even worse) my output is cut off. There has to be a way
besides arbitrarily sleeping to ensure that the command is finished.
I've tried things like piping in "quit", but that doesn't seem to
work. Anybody know how to solve this? I'm happy to use a different
approach if it means I can be sure I get all of my output out. --
Thanks!

 

[Eric Wong]

I'm doing some work with pymol and having trouble getting all the
pymol output out:

# I open it in quiet/commandline mode with a pipe:
IO.popen("pymol -cq -p", 'w+') do |pipe|
pipe.puts "load file.pdb, mymodel\n"
pipe.puts "run my_script.py"
# this command will generate a whole bunch of output to stdout
pipe.puts "my_script mymodel"
sleep(5) # <--- this is what I have to do in order to get the
output
pipe.close_write
output = pipe.read
end

What is annoying is that unless I sleep for 2-5 seconds I don't get
output or (even worse) my output is cut off. There has to be a way
besides arbitrarily sleeping to ensure that the command is finished.
I've tried things like piping in "quit", but that doesn't seem to
work. Anybody know how to solve this? I'm happy to use a different
approach if it means I can be sure I get all of my output out. --
Thanks!

Hi,

I don't know about pymol, but some apps do not respond well to having
its stdin closed on it with close_write. pipe.read should just block
until you get output, otherwise put an IO.select([pipe) in front of it.

Does this work?

IO.popen("pymol -cq -p", 'w+') do |pipe|
pipe.puts "load file.pdb, mymodel\n"
pipe.puts "run my_script.py"
pipe.puts "my_script mymodel"

# IO.select([pipe]) # you shouldn't need to uncomment this...

output = pipe.read
end

 

[bwv549]

I'm doing some work with pymol and having trouble getting all the
pymol output out:

# I open it in quiet/commandline mode with a pipe:
IO.popen("pymol -cq -p", 'w+') do |pipe|
  pipe.puts "load file.pdb, mymodel\n"
  pipe.puts "run my_script.py"
  # this command will generate a whole bunch of output to stdout
  pipe.puts "my_script mymodel"
  sleep(5)  # <--- this is what I have to do in order to get the
output
  pipe.close_write
  output = pipe.read
end

What is annoying is that unless I sleep for 2-5 seconds I don't get
output or (even worse) my output is cut off.  There has to be a way
besides arbitrarily sleeping to ensure that the command is finished.
I've tried things like piping in "quit", but that doesn't seem to
work.  Anybody know how to solve this?  I'm happy to use a different
approach if it means I can be sure I get all of my output out.   --
Thanks!

Hi,

I don't know about pymol, but some apps do not respond well to having
its stdin closed on it with close_write.  pipe.read should just block
until you get output, otherwise put an IO.select([pipe) in front of it.

Does this work?

  IO.popen("pymol -cq -p", 'w+') do |pipe|
    pipe.puts "load file.pdb, mymodel\n"
    pipe.puts "run my_script.py"
    pipe.puts "my_script mymodel"

    # IO.select([pipe]) # you shouldn't need to uncomment this...

    output = pipe.read
  end

both of those hang indefinitely. It doesn't hang if I add the line:

pipe.puts "quit"

but that doesn't give me all the output either. So maybe pymol
behaves differently than many commandline programs?

Here is what I've worked out for the time being. It opens a thread
for reading the output and asks if there is any output every 1/2
second. Once we go 1/2 second without output we kill the thread.

my_string = ""
Open3.popen3("pymol -cq -p") do |si, so, se|
si.puts "load file.pdb, mymodel\n"
si.puts "run my_script.py\n"
si.puts "myscript mymodel\n"

forstdout = Thread.new do
Thread.current['lines'] = []
while line = so.gets
Thread.current['lines'] << line
end
end

past_size = -1
loop do
sleep(0.5)
current_size = forstdout['lines'].size
break if current_size == past_size
past_size = current_size
end
my_string = forstdout['lines']
forstdout.kill
end

This is not the fastest method, but it does seem to work OK. I would
still love to hear other ideas.

 

[Eric Wong]

I'm doing some work with pymol and having trouble getting all the
pymol output out:

# I open it in quiet/commandline mode with a pipe:
IO.popen("pymol -cq -p", 'w+') do |pipe|
  pipe.puts "load file.pdb, mymodel\n"
  pipe.puts "run my_script.py"
  # this command will generate a whole bunch of output to stdout
  pipe.puts "my_script mymodel"
  sleep(5)  # <--- this is what I have to do in order to get the
output
  pipe.close_write
  output = pipe.read
end

What is annoying is that unless I sleep for 2-5 seconds I don't get
output or (even worse) my output is cut off.  There has to be a way
besides arbitrarily sleeping to ensure that the command is finished.
I've tried things like piping in "quit", but that doesn't seem to
work.  Anybody know how to solve this?  I'm happy to use a different
approach if it means I can be sure I get all of my output out.   --
Thanks!

Hi,

I don't know about pymol, but some apps do not respond well to having
its stdin closed on it with close_write.  pipe.read should just block
until you get output, otherwise put an IO.select([pipe) in front of it.

Does this work?

  IO.popen("pymol -cq -p", 'w+') do |pipe|
    pipe.puts "load file.pdb, mymodel\n"
    pipe.puts "run my_script.py"
    pipe.puts "my_script mymodel"

    # IO.select([pipe]) # you shouldn't need to uncomment this...

    output = pipe.read
  end

both of those hang indefinitely. It doesn't hang if I add the line:

pipe.puts "quit"

but that doesn't give me all the output either. So maybe pymol
behaves differently than many commandline programs?

Even IO.select([pipe]) hangs indefinitely? Oh, I guess pymol
expects an explicit quit of some sort.... What if you did:

IO.select([pipe])
pipe.readpartial(whatever_size_you_are_comfortable_with)

Here is what I've worked out for the time being. It opens a thread
for reading the output and asks if there is any output every 1/2
second. Once we go 1/2 second without output we kill the thread.

Here's a version without threads, should work the same:

my_string = ""
Open3.popen3("pymol -cq -p") do |si, so, se|
si.puts "load file.pdb, mymodel\n"
si.puts "run my_script.py\n"
si.puts "myscript mymodel\n"

# await input for 0.5 seconds, will return nil and
# break the loop if there is nothing to read from so after 0.5s
while ready = IO.select([so], nil, nil, 0.5)
# ready.first == so # in this case

# read until the current pipe buffer is empty
begin
my_string << so.read_nonblock(4096)
rescue Errno::EAGAIN
break
end while true
end
end

 

[Robert Klemme]

I'm doing some work with pymol and having trouble getting all the
pymol output out:

# I open it in quiet/commandline mode with a pipe:
IO.popen("pymol -cq -p", 'w+') do |pipe|
=A0 pipe.puts "load file.pdb, mymodel\n"
=A0 pipe.puts "run my_script.py"
=A0 # this command will generate a whole bunch of output to stdout
=A0 pipe.puts "my_script mymodel"
=A0 sleep(5) =A0# <--- this is what I have to do in order to get the
output
=A0 pipe.close_write
=A0 output =3D pipe.read
end

What is annoying is that unless I sleep for 2-5 seconds I don't get
output or (even worse) my output is cut off. =A0There has to be a way
besides arbitrarily sleeping to ensure that the command is finished.
I've tried things like piping in "quit", but that doesn't seem to
work. =A0Anybody know how to solve this? =A0I'm happy to use a differe= nt
approach if it means I can be sure I get all of my output out. =A0 --
Thanks!

Hi,

I don't know about pymol, but some apps do not respond well to having
its stdin closed on it with close_write. =A0pipe.read should just block
until you get output, otherwise put an IO.select([pipe) in front of it.

Does this work?

=A0 IO.popen("pymol -cq -p", 'w+') do |pipe|
=A0 =A0 pipe.puts "load file.pdb, mymodel\n"
=A0 =A0 pipe.puts "run my_script.py"
=A0 =A0 pipe.puts "my_script mymodel"

=A0 =A0 # IO.select([pipe]) # you shouldn't need to uncomment this...

=A0 =A0 output =3D pipe.read
=A0 end

both of those hang indefinitely. =A0It doesn't hang if I add the line:

=A0 pipe.puts "quit"

but that doesn't give me all the output either. =A0So maybe pymol
behaves differently than many commandline programs?

What does the documentation of "pymol" say?

Here is what I've worked out for the time being. =A0It opens a thread
for reading the output and asks if there is any output every 1/2
second. =A0Once we go 1/2 second without output we kill the thread.

my_string =3D ""
Open3.popen3("pymol -cq -p") do |si, so, se|
=A0si.puts "load file.pdb, mymodel\n"
=A0si.puts "run my_script.py\n"
=A0si.puts "myscript mymodel\n"

=A0forstdout =3D Thread.new do
=A0 =A0Thread.current['lines'] =3D []
=A0 =A0while line =3D so.gets
=A0 =A0 =A0Thread.current['lines'] << line
=A0 =A0end
=A0end

You don't need the thread local for this to work. Note there is also
Thread#value!

How does this work:

output =3D IO.popen("pymol -cq -p", 'w+') do |pipe|
reader =3D Thread.new { pipe.to_a }

pipe.puts "load file.pdb, mymodel"
pipe.puts "run my_script.py"
# this command will generate a whole bunch of output to stdout
pipe.puts "my_script mymodel"

pipe.close_write
reader.value
end

Kind regards

robert

--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

 

[bwv549]

@Robert: I've looked through a lot of pymol docs and haven't found
anything on it yet, will post if I do.

The solution you post looks very intriguing. It runs fine, but closes
before I get back the output I need. It seems like there must be some
kind of a pause to let pymol start spitting out output. I may play
around with it some to see if I can get it working.

@Eric: this solution works nicely and returns all the expected
output. I will be using it unless something better comes along.

Thanks to both of you for your excellent input on this. --John

2009/12/16 bwv549 <[email protected]>:

I'm doing some work with pymol and having trouble getting all the
pymol output out:
# I open it in quiet/commandline mode with a pipe:
IO.popen("pymol -cq -p", 'w+') do |pipe|
  pipe.puts "load file.pdb, mymodel\n"
  pipe.puts "run my_script.py"
  # this command will generate a whole bunch of output to stdout
  pipe.puts "my_script mymodel"
  sleep(5)  # <--- this is what I have to do in order to get the
output
  pipe.close_write
  output = pipe.read
end
What is annoying is that unless I sleep for 2-5 seconds I don't get
output or (even worse) my output is cut off.  There has to be a way
besides arbitrarily sleeping to ensure that the command is finished.
I've tried things like piping in "quit", but that doesn't seem to
work.  Anybody know how to solve this?  I'm happy to use a different
approach if it means I can be sure I get all of my output out.   --
Thanks!
Hi,
I don't know about pymol, but some apps do not respond well to having
its stdin closed on it with close_write.  pipe.read should just block
until you get output, otherwise put an IO.select([pipe) in front of it..
Does this work?
  IO.popen("pymol -cq -p", 'w+') do |pipe|
    pipe.puts "load file.pdb, mymodel\n"
    pipe.puts "run my_script.py"
    pipe.puts "my_script mymodel"
    # IO.select([pipe]) # you shouldn't need to uncomment this...
    output = pipe.read
  end

both of those hang indefinitely.  It doesn't hang if I add the line:

  pipe.puts "quit"

but that doesn't give me all the output either.  So maybe pymol
behaves differently than many commandline programs?

What does the documentation of "pymol" say?

Here is what I've worked out for the time being.  It opens a thread
for reading the output and asks if there is any output every 1/2
second.  Once we go 1/2 second without output we kill the thread.

my_string = ""
Open3.popen3("pymol -cq -p") do |si, so, se|
 si.puts "load file.pdb, mymodel\n"
 si.puts "run my_script.py\n"
 si.puts "myscript mymodel\n"

 forstdout = Thread.new do
   Thread.current['lines'] = []
   while line = so.gets
     Thread.current['lines'] << line
   end
 end

You don't need the thread local for this to work.  Note there is also
Thread#value!

How does this work:

output = IO.popen("pymol -cq -p", 'w+') do |pipe|
  reader = Thread.new { pipe.to_a }

  pipe.puts "load file.pdb, mymodel"
  pipe.puts "run my_script.py"
  # this command will generate a whole bunch of output to stdout
  pipe.puts "my_script mymodel"

  pipe.close_write
  reader.value
end

Kind regards

robert

 

[Robert Klemme]

@Robert: I've looked through a lot of pymol docs and haven't found
anything on it yet, will post if I do.

The solution you post looks very intriguing. It runs fine, but closes
before I get back the output I need.

Frankly, I don't see how this can be: the reader thread should read
*everthing* that comes out of the pipe and only stops if there is EOF.
The only explanation I have so far is that maybe some CTRL-D or other
EOF character is sent through the pipe making the reader think it has
reached EOF.

It seems like there must be some
kind of a pause to let pymol start spitting out output. I may play
around with it some to see if I can get it working.

Even if there is a pause the reader must wait. There must be something
else happening. Maybe try to do it on the shell and use 'od' to see
what is sent across the pipe.

@Eric: this solution works nicely and returns all the expected
output. I will be using it unless something better comes along.

Good that you have a working solution. However, what you state worries
me a bit. I believe we might not have identified the root cause of the
issue.

Thanks to both of you for your excellent input on this. --John

You're welcome!

Kind regards

robert