Simulate `bash` behaviour using Python and named pipes.

[Luca Cerone]

Hi everybody,
I am trying to understand how to use named pipes in python to launch external processes (in a Linux environment).

As an example I am trying to "imitate" the behaviour of the following sets of commands is bash:

mkfifo named_pipe
ls -lah > named_pipe &
cat < named_pipe

In Python I have tried the following commands:

import os
import subprocess as sp

os.mkfifo("named_pipe",0777) #equivalent to mkfifo in bash..
fw = open("named_pipe",'w')
#at this point the system hangs...

My idea it was to use subprocess.Popen and redirect stdout to fw...
next open named_pipe for reading and giving it as input to cat (still using Popen).

I know it is a simple (and rather stupid) example, but I can't manage to make it work..


How would you implement such simple scenario?

Thanks a lot in advance for the help!!!

Luca

 

[Paul Wiseman]

Hi everybody,
I am trying to understand how to use named pipes in python to launch
external processes (in a Linux environment).

As an example I am trying to "imitate" the behaviour of the following sets
of commands is bash:


In Python I have tried the following commands:

import os
import subprocess as sp

os.mkfifo("named_pipe",0777) #equivalent to mkfifo in bash..
fw = open("named_pipe",'w')
#at this point the system hangs...

My idea it was to use subprocess.Popen and redirect stdout to fw...
next open named_pipe for reading and giving it as input to cat (still
using Popen).

I know it is a simple (and rather stupid) example, but I can't manage to
make it work..


How would you implement such simple scenario?

Thanks a lot in advance for the help!!!

You can pipe using subprocess

p1 = subprocess.Popen(["ls", "-lah"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["cat"], stdin=p1.stdout)
p1.wait()
p2.wait()

You can also pass a file object to p1's stdout and p2's stdin if you want
to pipe via a file.

with open("named_pipe", "rw") as named_pipe:
p1 = subprocess.Popen(["ls", "-lah"], stdout=named_pipe)
p2 = subprocess.Popen(["cat"], stdin=named_pipe)
p1.wait()
p2.wait()

 

[Luca Cerone]

Hi Paul, first of all thanks for the help.

I am aware of the first solutions, just now I would like to experiment a bit with using named pipes (I also know that the example is trivial, but it just to grasp the main concepts)

You can also pass a file object to p1's stdout and p2's stdin if you wantto >pipe via a file.

with open("named_pipe", "rw") as named_pipe:
    p1 = subprocess.Popen(["ls", "-lah"], stdout=named_pipe)
    p2 = subprocess.Popen(["cat"], stdin=named_pipe)

    p1.wait()
    p2.wait()
 

Your second example doesn't work for me.. if named_file is not a file in the folder I get an error saying that there is not such a file.

If I create named_pipe as a named pipe using os.mkfifo("named_file",0777) than the code hangs.. I think it is because there is not process that reads the
content of the pipe, so the system waits for the pipe to be emptied.

Thanks a lot in advance for the help in any case.
Luca

 

[MRAB]

Hi everybody,
I am trying to understand how to use named pipes in python to launch external processes (in a Linux environment).

As an example I am trying to "imitate" the behaviour of the following sets of commands is bash:


In Python I have tried the following commands:

import os
import subprocess as sp

os.mkfifo("named_pipe",0777) #equivalent to mkfifo in bash..
fw = open("named_pipe",'w')
#at this point the system hangs...

My idea it was to use subprocess.Popen and redirect stdout to fw...
next open named_pipe for reading and giving it as input to cat (still using Popen).

I know it is a simple (and rather stupid) example, but I can't manage to make it work..


How would you implement such simple scenario?

Thanks a lot in advance for the help!!!

Opening the pipe for reading will block until it's also opened for
writing, and vice versa.

In your bash code, 'ls' blocked until you ran 'cat', but because you
ran 'ls' in the background you didn't notice it!

In your Python code, the Python thread blocked on opening the pipe for
writing. It was waiting for another thread or process to open the pipe
for reading.

 

[Luca Cerone]

Hi MRAB, thanks for the reply!

Opening the pipe for reading will block until it's also opened for

writing, and vice versa.

OK.



In your bash code, 'ls' blocked until you ran 'cat', but because you

ran 'ls' in the background you didn't notice it!

Right.

In your Python code, the Python thread blocked on opening the pipe for

writing. It was waiting for another thread or process to open the pipe

for reading.

OK. What you have written makes sense to me. So how do I overcome the block?
As you said in bash I run the ls process in background. How do I do that in Python?

Thanks again for the help,
Luca

 

[MRAB]

Hi MRAB, thanks for the reply!

OK. What you have written makes sense to me. So how do I overcome the block?
As you said in bash I run the ls process in background. How do I do that in Python?

You need to ensure that the pipe is already open at the other end.

Why are you using a named pipe anyway?

If you're talking to another program, then that needs to be running
already, waiting for the connection, at the point that you open the
named pipe from this end.

If you're using a pipe _within_ a program (a queue would be better),
then you should opening for writing in one thread and for reading in
another.

 

[Luca Cerone]

Hi Alister,

Are you sure you are using the correct tool for the task?

Yes. For two reasons: 1. I want to learn how to do this in Python 2. foran application I have in mind I will need to run external tools (not developed by me) and process the output using some tools that I have written in Python.

For technical reasons I can't use the subprocess.communicate() (the output to process is very large) method, and due to a bug in the interactive shellI am using (https://github.com/ipython/ipython/issues/3884) I cannot pipe processes just using the standard subprocess.Popen() approach.

I tend to find that in most cases if you are trying to execute bash

commands from Python you are doing it wrong.

As I said, the example in my question is just for learning purposes, I don't want to reproduce ls and cat in Python...

I just would like to learn how to handle named pipes in Python, which I find it easier to do by using a simple example that I am comfortable to use

Thanks in any case for your answer,
Luca

 

[Luca Cerone]

Thanks MRAB,

You need to ensure that the pipe is already open at the other end.

So I need to open the process that reads the pipe before writing in it?

Why are you using a named pipe anyway?

For some bug in ipython (see my previous email) I can't use subprocess.Popen and pipe in the standard way.
One of Ipython developers has suggested me to use named pipes as a temporary workaround. So I am taking the occasion to learn

If you're talking to another program, then that needs to be running

already, waiting for the connection, at the point that you open the

named pipe from this end.

I am not entirely sure I got this: ideally I would like to have a function that runs an external tool (the equivalent of ls in my example) redirecting its output in a named pipe.

A second function (the cat command in my example) would read the named_pipe, parse it and extract some features from the output.

I also would like that the named_pipe is deleted when the whole communication is ended.

If you're using a pipe _within_ a program (a queue would be better),

then you should opening for writing in one thread and for reading in

another.

Let's stick with the pipe I will ask about the queue when I manage to use pipes

I should have explained better that I have no idea how to run threads in Python : how do I open a thread that executes "ls -lah" in background and writes into a named pipe? And how do I open a thread that reads from the named pipe?

Can you please post a small example, so that I have something to work on?

Thanks a lot in advance for your help!

Luca

 

[Neil Cerutti]

I just would like to learn how to handle named pipes in Python,
which I find it easier to do by using a simple example that I
am comfortable to use

Names pipes are a unix concept that saves you the hassle and
limitations of writing to and reading from a temp file.

You'll have to create the temp file and manage attaching
processes to it yourself.

 

[MRAB]

Thanks MRAB,

So I need to open the process that reads the pipe before writing in
it?


For some bug in ipython (see my previous email) I can't use
subprocess.Popen and pipe in the standard way. One of Ipython
developers has suggested me to use named pipes as a temporary
workaround. So I am taking the occasion to learn

An alternative workaround is to use CPython.

I am not entirely sure I got this: ideally I would like to have a
function that runs an external tool (the equivalent of ls in my
example) redirecting its output in a named pipe.

A second function (the cat command in my example) would read the
named_pipe, parse it and extract some features from the output.

I also would like that the named_pipe is deleted when the whole
communication is ended.


Let's stick with the pipe I will ask about the queue when I
manage to use pipes

I should have explained better that I have no idea how to run
threads in Python : how do I open a thread that executes "ls -lah"
in background and writes into a named pipe? And how do I open a
thread that reads from the named pipe?

Can you please post a small example, so that I have something to
work on?

You could try something like this:

os.mkfifo("named_pipe", 0777)
ls_process = subprocess.Popen("ls -lah > named_pipe")
pipe = open("named_pipe", "r")
# Read the output of the subprocess from the pipe.

When the subprocess terminates (look at the docs for Popen objects),
close and delete the fifo.

 

[Luca Cerone]

Thanks this works (if you add shell=True in Popen).
If I don't want to use shell = True, how can I redirect the stdout to named_pipe? Popen accepts an open file handle for stdout, which I can't open for writing because that blocks the process...

 

[MRAB]

Thanks this works (if you add shell=True in Popen).
If I don't want to use shell = True, how can I redirect the stdout to named_pipe? Popen accepts an open file handle for stdout, which I can't open for writing because that blocks the process...

You're back to using separate threads for the reader and the writer.
The one that opens the pipe first will block until the other one opens
the other end.

 

[Luca Cerone]

You're back to using separate threads for the reader and the writer.And how do I create separate threads in Python? I was trying to use the threading library without not too success..

 

[MRAB]

And how do I create separate threads in Python? I was trying to use the threading library without not too success..

To run a function in a separate thread:

import threading

def my_func(arg_1, arg_2):
...

my_thread = threading.Thread(target=my_func, args=(arg_1, arg_2))
my_thread.start()


Is the thread still running?

if my_thread.is_alive():
...


Wait for the thread to terminate:

my_thread.join()

 

[Luca Cerone]

my_thread.join()

Thanks! I managed to make it work using the threading library

 

[Gregory Ewing]

Thanks! I managed to make it work using the threading library

If at least one of the external programs can accept the source
or destination as a filename argument instead of redirecting its
stdin or stdout, you can also do something like this:

import subprocess

p2 = subprocess.Popen(["cat", "named_pipe"])
pipe = open("named_pipe", "w")
p1 = subprocess.Popen(["ls", "-lah"], stdout = pipe)
pipe.close()
p1.wait()
p2.wait()

That works because opening the reading end of the pipe is
done by the subprocess executing cat, leaving the main
process free to open the other end.