improving performance of writing into a pipe

[mikprog]

Hi guys,

on an embedded linux system (BeagleBoard) I am writing data coming from bluetooth dongle into a pipe.
The function is the following one:


def write_to_pipe(line):

# next line ensures that bytes like '0x09' are not translated into '\t' for
#example, and they are sent as such
hexbytes = "\\x" + "\\x".join([hex(ord(c))[2:].zfill(2) for c in line])
wrap = ["echo -en '", "' > /tmp/mypipe"]
msg = hexbytes.join(wrap)
print "DBG: sending: ", msg

try:
os.popen( msg )
except:
print "Error: write_to_pipe has failed!"


Now I typically receive 4 bytes from the bluetooth dongle and that is fine.
However when I receive many more than that it seems that the writing into the pipe is too slow.

Is there any clever/obvious way to improve the code above?
(I am quite sure there is to be honest).

Thanks for any suggestion!
Mik

 

[mikprog]

On Monday, February 18, 2013 3:21:53 PM UTC, Oscar Benjamin wrote:
[..]

Can you not open the pipe file directly in Python code? e.g.



fout = open('/tmp/mypipe', 'w')

fout.write(data)



I guess that this would be more efficient than using os.popen to run echo.

that's an idea,
thanks Oscar.
However I get an exception while trying to open the queue:
fout = open('/tmp/mypipe', 'w')

I have tried it in a command line and the call doesn't return until in another terminal I open the same queue for reading (???)

I have created the queue with:
mkfifo /tmp/mypipe

any clue?
mik

 

[mikprog]

On Monday, February 18, 2013 3:21:53 PM UTC, Oscar Benjamin wrote:
[..]

Can you not open the pipe file directly in Python code? e.g.



fout = open('/tmp/mypipe', 'w')

fout.write(data)



I guess that this would be more efficient than using os.popen to run echo.

that's an idea,
thanks Oscar.
However I get an exception while trying to open the queue:
fout = open('/tmp/mypipe', 'w')

I have tried it in a command line and the call doesn't return until in another terminal I open the same queue for reading (???)

I have created the queue with:
mkfifo /tmp/mypipe

any clue?
mik

 

[Thomas Rachel]

Am 18.02.2013 17:31 schrieb (e-mail address removed):

However I get an exception while trying to open the queue:
fout = open('/tmp/mypipe', 'w')

I don't see an exception in your answer. Where did you put it for us?

I have tried it in a command line and the call doesn't return until in another terminal I open the same queue for reading (???)

That's normal. An open call to a pipe blocks until someone reads from it.

But it's the same with your popen() version.


Thomas

 

[mikprog]

[..]

I don't see an exception in your answer. Where did you put it for us?

well I just did print a message:

PIPEPATH = ["/tmp/mypipe"]

[..]
try:
self.process = os.popen( self.PIPEPATH, 'w')
except:
print "Error while trying opening the pipe!"
print "check: ", self.PIPEPATH
exit()

I see the error messages.

It's quite frustrating as I think I am doing something really stupid in here.

Mik

 

[Michael Torrie]

[..]

I don't see an exception in your answer. Where did you put it for us?

well I just did print a message:

PIPEPATH = ["/tmp/mypipe"]

[..]
try:
self.process = os.popen( self.PIPEPATH, 'w')
except:
print "Error while trying opening the pipe!"
print "check: ", self.PIPEPATH
exit()

I see the error messages.

Unfortunately your attempt to catch this exception is hiding the true
cause. You need to give us the actual exception. Otherwise it could be
anything from self.PIPEPATH not existing to who knows what.

Almost never do you want to catch all exceptions like you're doing. You
should only catch the specific exceptions you know how to deal with in
your code.

For testing purposes, if your code really is as you put it, then
catching exceptions is kind of silly since you're just re-raising the
exception (sort of) but without any contextual information that would
make the error meaningful.

 

[Serhiy Storchaka]

on an embedded linux system (BeagleBoard) I am writing data coming from bluetooth dongle into a pipe.
The function is the following one:


def write_to_pipe(line):

# next line ensures that bytes like '0x09' are not translated into '\t' for
#example, and they are sent as such
hexbytes = "\\x" + "\\x".join([hex(ord(c))[2:].zfill(2) for c in line])
wrap = ["echo -en '", "' > /tmp/mypipe"]
msg = hexbytes.join(wrap)
print "DBG: sending: ", msg

try:
os.popen( msg )
except:
print "Error: write_to_pipe has failed!"


Now I typically receive 4 bytes from the bluetooth dongle and that is fine.
However when I receive many more than that it seems that the writing into the pipe is too slow.

Is there any clever/obvious way to improve the code above?
(I am quite sure there is to be honest).

def write_to_pipe(line):
hexbytes = ''.join('\\x%02x' % ord(c) for c in line)
with open('/tmp/mypipe', 'w') as f:
f.write(hexbytes)

 

[mikprog]

[..]

I don't see an exception in your answer. Where did you put it for us?

well I just did print a message:
PIPEPATH = ["/tmp/mypipe"]

[..]

try:

self.process = os.popen( self.PIPEPATH, 'w')

except:

print "Error while trying opening the pipe!"

print "check: ", self.PIPEPATH


I see the error messages.

Unfortunately your attempt to catch this exception is hiding the true

cause. You need to give us the actual exception. Otherwise it could be

anything from self.PIPEPATH not existing to who knows what.



Almost never do you want to catch all exceptions like you're doing. You

should only catch the specific exceptions you know how to deal with in

your code.



For testing purposes, if your code really is as you put it, then
catching exceptions is kind of silly since you're just re-raising the
exception (sort of) but without any contextual information that would
make the error meaningful.

Ok, I get your point.
But on the other hand how do I know what to catch if I have no clue what is causing the error?
There must be a way to catch all the possible errors and then investigate what is the problem, right? (which is not what I have done so far).

Or rather: what would you try to catch in this particular case?

Thanks

 

[mikprog]

[..]

I don't see an exception in your answer. Where did you put it for us?

well I just did print a message:
PIPEPATH = ["/tmp/mypipe"]

[..]

try:

self.process = os.popen( self.PIPEPATH, 'w')

except:

print "Error while trying opening the pipe!"

print "check: ", self.PIPEPATH


I see the error messages.

Unfortunately your attempt to catch this exception is hiding the true

cause. You need to give us the actual exception. Otherwise it could be

anything from self.PIPEPATH not existing to who knows what.



Almost never do you want to catch all exceptions like you're doing. You

should only catch the specific exceptions you know how to deal with in

your code.



For testing purposes, if your code really is as you put it, then
catching exceptions is kind of silly since you're just re-raising the
exception (sort of) but without any contextual information that would
make the error meaningful.

Ok, I get your point.
But on the other hand how do I know what to catch if I have no clue what is causing the error?
There must be a way to catch all the possible errors and then investigate what is the problem, right? (which is not what I have done so far).

Or rather: what would you try to catch in this particular case?

Thanks

 

[Peter Otten]

[..]
I don't see an exception in your answer. Where did you put it for us?
well I just did print a message:
PIPEPATH = ["/tmp/mypipe"]

[..]

try:

self.process = os.popen( self.PIPEPATH, 'w')

except:

print "Error while trying opening the pipe!"

print "check: ", self.PIPEPATH


I see the error messages.

Unfortunately your attempt to catch this exception is hiding the true

cause. You need to give us the actual exception. Otherwise it could be

anything from self.PIPEPATH not existing to who knows what.



Almost never do you want to catch all exceptions like you're doing. You

should only catch the specific exceptions you know how to deal with in

your code.



For testing purposes, if your code really is as you put it, then
catching exceptions is kind of silly since you're just re-raising the
exception (sort of) but without any contextual information that would
make the error meaningful.

Ok, I get your point.
But on the other hand how do I know what to catch if I have no clue what
is causing the error? There must be a way to catch all the possible errors
and then investigate what is the problem, right? (which is not what I have
done so far).

Or rather: what would you try to catch in this particular case?

Nothing.

Once you get your script working you can try to provoke errors, and for
those errors you can recover from you can write error handlers. For IOError
and Python < 3.3 that may involve inspecting the errno attribute and
conditionally reraising.

By the way, I don't think

PIPEPATH = ["/tmp/mypipe"]
self.process = os.popen( self.PIPEPATH, 'w')

can work. As a few people already told you the built-in open()

with open(PIPEPATH, "w") as f:
f.write(...)

is the way to go.

 

[mikprog]

Once you get your script working you can try to provoke errors, and for

those errors you can recover from you can write error handlers. For IOError

and Python < 3.3 that may involve inspecting the errno attribute and

conditionally reraising.

Ok.



By the way, I don't think

PIPEPATH = ["/tmp/mypipe"]
self.process = os.popen( self.PIPEPATH, 'w')

can work. As a few people already told you the built-in open()

Few people?
I thought Oscar was a singular person, not a group of people
Seriously, I am convinced by that approach (thanks) and I wish to go that way, but the problem I am getting now is that the open fails and then I can't go on.

Also, I am now looking at the subprocess as os.popen seems deprecated.
Any opinion on that?
Thanks for your suggestion.

Mik

 

[mikprog]

Once you get your script working you can try to provoke errors, and for

those errors you can recover from you can write error handlers. For IOError

and Python < 3.3 that may involve inspecting the errno attribute and

conditionally reraising.

Ok.



By the way, I don't think

PIPEPATH = ["/tmp/mypipe"]
self.process = os.popen( self.PIPEPATH, 'w')

can work. As a few people already told you the built-in open()

Few people?
I thought Oscar was a singular person, not a group of people
Seriously, I am convinced by that approach (thanks) and I wish to go that way, but the problem I am getting now is that the open fails and then I can't go on.

Also, I am now looking at the subprocess as os.popen seems deprecated.
Any opinion on that?
Thanks for your suggestion.

Mik

 

[Oscar Benjamin]

Few people?
I thought Oscar was a singular person, not a group of people

Serhiy also suggested it.

Seriously, I am convinced by that approach (thanks) and I wish to go that way, but the problem I am getting now is that the open fails and then I can't go on.

Perhaps no-one has been explicit enough about what you should do here:

1) Remove all try/except from your code.
2) Run the code
3) Look at the *unadulterated* error message that Python prints out
4) Either fix the error if you know how or
5) Reply here posting the exact error message.

Also, in future:
6) Don't use bare try/except and don't catch errors while you're
debugging. Allow the errors to be printed as they are so that you can
read the message and see the line that triggers the error.
7) Don't post to a mailing list saying "I get an error", "I can see
errors" or "it doesn't work". If you have errors paste the exact error
message (all of it!). If you don't get errors but it doesn't do what
you want explain exactly what happened and also what you wanted to
happen. If you had followed this procedure at the start of this
thread, then you would already have a solution to (or at least an
explanation of) your problem by now.

Also, I am now looking at the subprocess as os.popen seems deprecated.
Any opinion on that?

I'm not convinced that either is appropriate for your problem.


Oscar

 

[mikprog]

on an embedded linux system (BeagleBoard) I am writing data coming from bluetooth dongle into a pipe.

The function is the following one:


def write_to_pipe(line):

# next line ensures that bytes like '0x09' are not translated into '\t' for

#example, and they are sent as such

hexbytes = "\\x" + "\\x".join([hex(ord(c))[2:].zfill(2) for c in line])

wrap = ["echo -en '", "' > /tmp/mypipe"]

msg = hexbytes.join(wrap)

print "DBG: sending: ", msg


os.popen( msg )

print "Error: write_to_pipe has failed!"


Now I typically receive 4 bytes from the bluetooth dongle and that is fine.

However when I receive many more than that it seems that the writing into the pipe is too slow.

Is there any clever/obvious way to improve the code above?

(I am quite sure there is to be honest).

def write_to_pipe(line):

hexbytes = ''.join('\\x%02x' % ord(c) for c in line)

with open('/tmp/mypipe', 'w') as f:

f.write(hexbytes)

I'll take your hexbytes = '' line (which is surely more efficient than mine).
However whit this approach open + write it seems the pipe doesn't get the data...
I am not sure what is going on.
At this point I suspect it could be a problem on the pipe itself (which I inherited).

It is just weird that the pipe accept this correctly:
wrap = ["echo -en '", "' > /tmp/midi"]
msg = hexbytes.join(wrap)
os.popen( msg )

but seems to be careless of approach open + write.

I need to investigate there.

Thanks a lot, to you and to everyone else.
Mik

 

[mikprog]

on an embedded linux system (BeagleBoard) I am writing data coming from bluetooth dongle into a pipe.

The function is the following one:


def write_to_pipe(line):

# next line ensures that bytes like '0x09' are not translated into '\t' for

#example, and they are sent as such

hexbytes = "\\x" + "\\x".join([hex(ord(c))[2:].zfill(2) for c in line])

wrap = ["echo -en '", "' > /tmp/mypipe"]

msg = hexbytes.join(wrap)

print "DBG: sending: ", msg


os.popen( msg )

print "Error: write_to_pipe has failed!"


Now I typically receive 4 bytes from the bluetooth dongle and that is fine.

However when I receive many more than that it seems that the writing into the pipe is too slow.

Is there any clever/obvious way to improve the code above?

(I am quite sure there is to be honest).

def write_to_pipe(line):

hexbytes = ''.join('\\x%02x' % ord(c) for c in line)

with open('/tmp/mypipe', 'w') as f:

f.write(hexbytes)

I'll take your hexbytes = '' line (which is surely more efficient than mine).
However whit this approach open + write it seems the pipe doesn't get the data...
I am not sure what is going on.
At this point I suspect it could be a problem on the pipe itself (which I inherited).

It is just weird that the pipe accept this correctly:
wrap = ["echo -en '", "' > /tmp/midi"]
msg = hexbytes.join(wrap)
os.popen( msg )

but seems to be careless of approach open + write.

I need to investigate there.

Thanks a lot, to you and to everyone else.
Mik

 

[mikprog]

def write_to_pipe(line):

hexbytes = ''.join('\\x%02x' % ord(c) for c in line)

with open('/tmp/mypipe', 'w') as f:

f.write(hexbytes)

Update:
with a fix in the pipe THIS was the right way to do it, and it now works.
Thanks a lot Serhiy to you and to everyone else.

Mik

 

[mikprog]

def write_to_pipe(line):

hexbytes = ''.join('\\x%02x' % ord(c) for c in line)

with open('/tmp/mypipe', 'w') as f:

f.write(hexbytes)

Update:
with a fix in the pipe THIS was the right way to do it, and it now works.
Thanks a lot Serhiy to you and to everyone else.

Mik

 

[Peter Otten]

I thought this was only needed to have 'echo' except your data.

Update:
with a fix in the pipe THIS was the right way to do it, and it now works.
Thanks a lot Serhiy to you and to everyone else.

Do you mind telling us what fix you applied?

 

[mikprog]

Thanks a lot Serhiy to you and to everyone else.

Do you mind telling us what fix you applied?

Oh, apologies Peter, I thought it was clear as I posted it after the lines written by Serhiy.
So it was what Serhiy suggest in addition to some (?minor?) modification to the pipe itself, which I cannot comment about as I don't have access to it.
But apparently the problem of not being able to open it was due to it.
(It does not help much I am afraid... but that's all I know).

Mik

 

[mikprog]

Thanks a lot Serhiy to you and to everyone else.

Do you mind telling us what fix you applied?

Oh, apologies Peter, I thought it was clear as I posted it after the lines written by Serhiy.
So it was what Serhiy suggest in addition to some (?minor?) modification to the pipe itself, which I cannot comment about as I don't have access to it.
But apparently the problem of not being able to open it was due to it.
(It does not help much I am afraid... but that's all I know).

Mik