How do I speedup this loop?

[Steve]

Hi,

I'm getting some output by running a command using os.popen. I need to
parse the output and transform it in some sense so that it's 'DB
compatible', (i.e I need to store the output in a database (postgres)
after escaping some characters). Since I'm new to python, I wasn't sure
if there was a better way of doing this so this is what I did:


# Parse the output returned by popen and return the script
out = os.popen('some command')
all_lines = out.readlines()

script = []
for i in xrange(len(all_lines)):
line = all_lines.replace("'", "\\'")[0:len(line)-1]
# replace ' with \'
line_without_carriage = line[0:len(line)-1] # remove
carriage
line_without_carriage =
line_without_carriage.replace("\\n", "$___n") # replace end of line with
$___n
line_without_carriage += "@___n" # add a 'end of line'
character to the end
script.append(line_without_carriage)
# end for

script = ''.join(script)

Please help because I'm pretty sure I'm wasting a lot of cpu time in
this loop. Thanks

Steve

 

[Marco Aschwanden]

I'm getting some output by running a command using os.popen. I need to
parse the output and transform it in some sense so that it's 'DB
compatible', (i.e I need to store the output in a database (postgres)
after escaping some characters).

If you are using Python's DB API 2.0 than this escaping would be done by
the API:

So, no need to parse (and afterwards unparse) the ouput - I don't think
that anyone can beat this speed up!

Regards,
Marco

 

[David Fraser]

If you are using Python's DB API 2.0 than this escaping would be done by
the API:



So, no need to parse (and afterwards unparse) the ouput - I don't think
that anyone can beat this speed up!

Except if you're aiming for database independence, as different database
drivers support different means of escaping parameters...

David

 

[Riccardo Attilio Galli]

Hi,

I'm getting some output by running a command using os.popen. I need to
parse the output and transform it in some sense so that it's 'DB
compatible', (i.e I need to store the output in a database (postgres)
after escaping some characters). Since I'm new to python, I wasn't sure
if there was a better way of doing this so this is what I did:

if you were replacing a character with another, the best and quick way was
to use a translation table, but you're not lucky

line = all_lines.replace("'", "\\'")[0:len(line)-1]
# replace ' with \'

? this can't work. You never defined "line" and you're using "len" on it.
I think you want to delete the last character of the string. if so, you
can use negative indexes

line = all_lines.replace("'", "\\'")[:-1]

the 0 disappeared is the default value

line_without_carriage = line[0:len(line)-1] # remove carriage

similar here
line_without_carriage = line[:-1]

but you're just deleting the current last character of the string.
so you could delete this line and change the indexing in the first one
so the first one would become
line = all_lines.replace("'", "\\'")[:-2]

ah, I don't think you're removing a carriage return ('\r') here.
If your line end with '\r\n' you're killing '\n' , a line feed.
This is important 'cause in the next line....

line_without_carriage = line_without_carriage.replace("\\n", "$___n")

# replace end of line with $___n
.... you try to replace '\\n' ,
are you intending to delete the line feed, the end of line ?
if this is the case you should write '\n' (one character) not '\\n' (a
string of len 2)

line_without_carriage += "@___n"
script.append(line_without_carriage)
# end for
script = ''.join(script)

the best here is to do
script.append(line_without_carriage)
script.append('@___n')
# end for
script = ''.join(script)

Appending '@___n' you don't need to loose memory for destroying and
creating a new string each time

Please help because I'm pretty sure I'm wasting a lot of cpu time in
this loop. Thanks

Steve

Ciao,
Riccardo

--
-=Riccardo Galli=-

_,e.
s~ ``
~@. ideralis Programs
.. ol
`**~ http://www.sideralis.net

 

[Istvan Albert]

Except if you're aiming for database independence, as different database
drivers support different means of escaping parameters...

IMHO database independence is both overrated not to mention impossible.
You can always try the 'greatest common factor' approach but that
causes more trouble (and work) than it saves.

I agree with the previous poster stating that escaping should be done
in the DB API, but it is better to use the 'typed' escaping:

sql = 'SELECT FROM users WHERE user_id=%d AND user_passwd=%s'
par = [1, 'something']
cursor.execute(sql, par)

Istvan.

 

[george young]

I'm getting some output by running a command using os.popen. I need to
parse the output and transform it in some sense so that it's 'DB
compatible', (i.e I need to store the output in a database (postgres)
after escaping some characters). Since I'm new to python, I wasn't sure
if there was a better way of doing this so this is what I did:

# Parse the output returned by popen and return the script
out = os.popen('some command')
all_lines = out.readlines()

script = []
for i in xrange(len(all_lines)):
line = all_lines.replace("'", "\\'")[0:len(line)-1]
# replace ' with \'
line_without_carriage = line[0:len(line)-1] # remove
carriage
line_without_carriage =
line_without_carriage.replace("\\n", "$___n") # replace end of line with
$___n
line_without_carriage += "@___n" # add a 'end of line'
character to the end
script.append(line_without_carriage)
# end for

script = ''.join(script)

How about:

lines = []
out = os.popen('some command')
for l in out:
lines.append(l.strip())
script = ''.join(lines)
out.close()

The "strip" actually removes white space from front and back of the string;
you could say l.strip('\n') if you only want the newlines removed (or '\r'
if they're really carriage return characters.)

Or if you want a clever (and most CPU efficient!) one-liner:

script = [l.strip() for l in os.popen('some command')]

I'm not advocating such a terse one-liner unless you are very comfortable
with it's meaning; will you easily know what it does when you see it
six months from now in the heat of battle?

Also, the one-liner does not allow you to explicitly close the file
descriptor from popen. This could be a serious problem if it gets run
hundreds of times in a loop.

Have fun,
-- George Young

 

[Jean Brouwers]

What about handling all output as one string?

script = os.popen('some command')
script = script.replace("'", "\\'") # replace ' with \'
script = script.replace("\r", ") # remove cr
script = script.replace("\\n", "$___n") # replace \n
script = script.replace("\n", "@___n'") # replace nl


/Jean Brouwers

 

[David Fraser]

IMHO database independence is both overrated not to mention impossible.
You can always try the 'greatest common factor' approach but that
causes more trouble (and work) than it saves.

Not overrated or impossible. It's part of our business model. It works.

I agree with the previous poster stating that escaping should be done
in the DB API, but it is better to use the 'typed' escaping:

sql = 'SELECT FROM users WHERE user_id=%d AND user_passwd=%s'
par = [1, 'something']
cursor.execute(sql, par)

Better if the database driver you are using supports it... otherwise
userless
I think there is a need to drive towards some sort of standard approach
to this in DB-API (maybe version 3?) as it otherwise nullifies
parameters for anyone using multiple database drivers.

David

 

[Lonnie Princehouse]

Welcome to Python =)
Somebody else already mentioned checking out the DBI API's way of
escaping data; this is a good idea. Besides that, here are some
general tips-

1. Consider using out.xreadlines() if you only need one line at a
time:

for line in out.xreadlines():
...

If you need all of the data at once, try out.read()

2. You can use negative numbers to index relative to the end of a
sequence:

line[0:-1] is equivalent to line[0:len(line)-1]
(i.e. cut off the last character of a string)

You can also use line.strip() to remove trailing whitespace,
including newlines.

3. If you omit the index on either side of a slice, Python will
default to the beginning and end of a sequence:

line[:] is equivalent to line[0:len(line)]

4. Check out the regular expression module. Here's how to read all of
your output and make multiple escape substitutions. Smashing this
into one regular expression means you only need one pass over the
data. It also avoids string concatenation.

import re, os

out = os.popen('some command')
data = out.read()

substitution_map = {
"'" : r"\'",
"\n": "$___n",
}

def sub_func(match_object, smap=substitution_map):
return smap[match_object.group(0)]

escape_expr = re.compile('|'.join(substitution_map.keys())))

escaped_data = escape_expr.sub(sub_func, data)

# et voila... now you've got a big escaped string without even
# writing a single for loop. Tastes great, less filling.

(caveat: I didn't run this code. It might have typos.)

Hi,

I'm getting some output by running a command using os.popen. I need to
parse the output and transform it in some sense so that it's 'DB
compatible', (i.e I need to store the output in a database (postgres)
after escaping some characters). Since I'm new to python, I wasn't sure
if there was a better way of doing this so this is what I did:


# Parse the output returned by popen and return the script
out = os.popen('some command')
all_lines = out.readlines()

script = []
for i in xrange(len(all_lines)):
line = all_lines.replace("'", "\\'")[0:len(line)-1]
# replace ' with \'
line_without_carriage = line[0:len(line)-1] # remove
carriage
line_without_carriage =
line_without_carriage.replace("\\n", "$___n") # replace end of line with
$___n

line_without_carriage += "@___n" # add a 'end of line'

 

[Bart Nessux]

How about:

lines = []
out = os.popen('some command')
for l in out:
lines.append(l.strip())
script = ''.join(lines)
out.close()

The "strip" actually removes white space from front and back of the string;
you could say l.strip('\n') if you only want the newlines removed (or '\r'
if they're really carriage return characters.)

The above is a great solution... should make for a good speed up. It's
how I might pproach it.

Or if you want a clever (and most CPU efficient!) one-liner:

script = [l.strip() for l in os.popen('some command')]

Clever progammers should be shot! I've had to work behind them... they
are too smart for their own good. They think everyone else in the world
is as clever as they are... this is where they are wrong

 

[george young]

How about:

lines = []
out = os.popen('some command')
for l in out:
lines.append(l.strip())
script = ''.join(lines)
out.close()

The "strip" actually removes white space from front and back of the string;
you could say l.strip('\n') if you only want the newlines removed (or '\r'
if they're really carriage return characters.)

The above is a great solution... should make for a good speed up. It's
how I might pproach it.

Or if you want a clever (and most CPU efficient!) one-liner:

script = ''.join([l.strip() for l in os.popen('some command')])

[fixed up a bit...I forgot about the join]

Clever progammers should be shot! I've had to work behind them... they
are too smart for their own good. They think everyone else in the world
is as clever as they are... this is where they are wrong

Oh, all right. How about:

out = os.popen('some command')
temp_script = [l.strip() for l in out]
script = ''.join(temp_script)

That's clear enough, and still takes advantage of the efficiency of
the list comprehension! (I admit, that for reading the whole file,
the other postings of regexp substitution on the total string are
certainly faster, given enough RAM, but still not as clear and concise
and elegant ... blah blah blah as mine...

-- George Young

 

[Steve]

How about:

lines = []
out = os.popen('some command')
for l in out:
lines.append(l.strip())
script = ''.join(lines)
out.close()

The "strip" actually removes white space from front and back of the string;
you could say l.strip('\n') if you only want the newlines removed (or '\r'
if they're really carriage return characters.)

The above is a great solution... should make for a good speed up. It's
how I might pproach it.

Or if you want a clever (and most CPU efficient!) one-liner:

script = ''.join([l.strip() for l in os.popen('some command')])

[fixed up a bit...I forgot about the join]

Clever progammers should be shot! I've had to work behind them... they
are too smart for their own good. They think everyone else in the world
is as clever as they are... this is where they are wrong

Oh, all right. How about:

out = os.popen('some command')
temp_script = [l.strip() for l in out]
script = ''.join(temp_script)

That looks good but the problem is that I don't want to 'strip' off the
'end of line' characters etc., because I need to reproduce/print the
output exactly as it was at a later stage. What's more... I need to
print it out on a HTML page, and so if I know the different between \n
(in code) and the implicit end of line character, I can interpret that
in HTML accordingly. For example, the output can contain something like:

printf("Hey there\n");

and so, there's a \n embedded inside the text as well as the end of line
character which isn't visible. Although escaping characters using a
DB-API function might do the trick, this still won't help me much in
the end, where I need to print a "<br>" for each 'end of line character'.

 

[Steve]

What about handling all output as one string?

script = os.popen('some command')
script = script.replace("'", "\\'") # replace ' with \'
script = script.replace("\r", ") # remove cr
script = script.replace("\\n", "$___n") # replace \n
script = script.replace("\n", "@___n'") # replace nl

This won't do any better than what I was already doing. I need the code
to be very fast and this will only end up creating a lot of copies
everytime the string is going to be modified (strings are immutable). I
really like the idea of using regex for this (proposed by lonnie), but I
still need to a hang of it.

Cheers,
Steve

------------ And now a word from our sponsor ------------------
Want to have instant messaging, and chat rooms, and discussion
groups for your local users or business, you need dbabble!
-- See http://netwinsite.com/sponsor/sponsor_dbabble.htm ----