Need help running external program

[Rigga]

Hi,

I am running the line of code below from a shell script and it works fine,
however I am at a total loss on how i can run it from within a Python
script as every option I have tried fails and it appears to be down to the
escaping of certain characters.

wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
's/.*url="\([^"]*\)".*/\1/p'

I want to use the above code in my program by using popen2 so i can query
the results i.e.

output, input = popen2("the code here")

print output

Any help would be appreciated.

Many thanks

RiGGa

 

[Pink]

Hi,

I am running the line of code below from a shell script and it works fine,
however I am at a total loss on how i can run it from within a Python
script as every option I have tried fails and it appears to be down to the
escaping of certain characters.

wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
's/.*url="\([^"]*\)".*/\1/p'

If your problem is getting a python string without worrying about how to
escape the escape sequences, try:

r"""wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
's/.*url="\([^"]*\)".*/\1/p'"""

You should be able to pass this directly to a popen() function.

 

[Rigga]

Hi,

I am running the line of code below from a shell script and it works
fine, however I am at a total loss on how i can run it from within a
Python script as every option I have tried fails and it appears to be
down to the escaping of certain characters.

wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
's/.*url="\([^"]*\)".*/\1/p'

If your problem is getting a python string without worrying about how to
escape the escape sequences, try:

r"""wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
's/.*url="\([^"]*\)".*/\1/p'"""

You should be able to pass this directly to a popen() function.

Hi,

Thanks for replying however I have just tried that and it does not seem to
work, it doesnt return any results (i take it the r was a typo)

Thanks

RiGGa

 

[Leif B. Kristensen]

I'm using wget from Python to get extactly one line from a reports page.
I made this function that works for me:

def wgetline(exp): # see Python Cookbook p. 228
print "Getting result from server ..."
command = 'wget -q -O - \
http://www.foobar.com/report.pl\?UserID=xxx\&UserPW=xxx \
| grep ' + exp
child = os.popen(command)
data = child.read()
return data

I had to escape the ? and & in the url, or the CGI script at the other
end would refuse to cooperate with "Invalid UserID or UserPW".

 

[Tim Jarman]

Hi,

I am running the line of code below from a shell script and it works
fine, however I am at a total loss on how i can run it from within a
Python script as every option I have tried fails and it appears to be
down to the escaping of certain characters.

wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
's/.*url="\([^"]*\)".*/\1/p'

If your problem is getting a python string without worrying about how to
escape the escape sequences, try:

r"""wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
's/.*url="\([^"]*\)".*/\1/p'"""

You should be able to pass this directly to a popen() function.

Hi,

Thanks for replying however I have just tried that and it does not seem to
work, it doesnt return any results (i take it the r was a typo)

Thanks

RiGGa

No, the r was the point - it's there to tell Python not to do any escaping
on the string. Try it again with the r and see what happens.

 

[Rigga]

Rigga wrote:

Hi,

I am running the line of code below from a shell script and it works
fine, however I am at a total loss on how i can run it from within a
Python script as every option I have tried fails and it appears to be
down to the escaping of certain characters.

wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
's/.*url="\([^"]*\)".*/\1/p'
If your problem is getting a python string without worrying about how to
escape the escape sequences, try:

r"""wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
's/.*url="\([^"]*\)".*/\1/p'"""

You should be able to pass this directly to a popen() function.

Hi,

Thanks for replying however I have just tried that and it does not seem
to work, it doesnt return any results (i take it the r was a typo)

Thanks

RiGGa

No, the r was the point - it's there to tell Python not to do any escaping
on the string. Try it again with the r and see what happens.

Brilliant!!! that works a treat thankyou!!, where on earth did you find out
about the 'r' any pointers to documentation appreciated.

Thanks

RiGGa

 

[Brian van den Broek]

Rigga said unto the world upon 2005-02-27 15:04:

Tim Jarman wrote:


Brilliant!!! that works a treat thankyou!!, where on earth did you find out
about the 'r' any pointers to documentation appreciated.

Thanks

RiGGa

<http://www.python.org/doc/current/ref/strings.html>
<http://www.python.org/doc/current/lib/module-re.html>

Best,

Brian vdB

 

[Pink]

Brilliant!!! that works a treat thankyou!!, where on earth did you find
out
about the 'r' any pointers to documentation appreciated.

This is a pretty common problem when working with regular expression (which
usually contain many backslashes) - that's where I saw this syntax for the
first time (e.g. http://docs.python.org/lib/match-objects.html).
The official reference for string literals is here:
http://docs.python.org/ref/strings.html

c ya

 

[Rigga]

Rigga said unto the world upon 2005-02-27 15:04:

<http://www.python.org/doc/current/ref/strings.html>
<http://www.python.org/doc/current/lib/module-re.html>

Best,

Brian vdB

Thanks for all your help with this it is appreciated, one further question
though, how do I pass a variable to the external program while using the
r"""

Thanks

RiGGa

 

[Tim Jarman]

(snip stuff about raw strings)

Thanks for all your help with this it is appreciated, one further question
though, how do I pass a variable to the external program while using the
r"""

Thanks

RiGGa

I'm not sure I understand the question. Say you have:

parameter = r"my \funky \text"

then surely you just pass it to your external program using whichever method
you like, e.g.

import os
os.execl("your_external_prog", parameter) # replaces the current process

or some variant of:

return_code = os.spawnl(os.P_WAIT, "your_external_prog", parameter)

or you can build a command line:

command = "your_external_prog %s" % parameter
return_code = os.system(command)

(see docs on the os module for more variations on this theme than you can
shack a stick at)

It's just a string, after all.

 

[Rigga]

(snip stuff about raw strings)


I'm not sure I understand the question. Say you have:

parameter = r"my \funky \text"

then surely you just pass it to your external program using whichever
method you like, e.g.

import os
os.execl("your_external_prog", parameter) # replaces the current process

or some variant of:

return_code = os.spawnl(os.P_WAIT, "your_external_prog", parameter)

or you can build a command line:

command = "your_external_prog %s" % parameter
return_code = os.system(command)

(see docs on the os module for more variations on this theme than you can
shack a stick at)

It's just a string, after all.

This is the command I am trying to run:

feed is a list of web addresses

output, input = popen2("wget -q %s -O - | tr '\r' '\n' | tr \' \" | sed -n
's/.*url="\([^"]*\)".*/\1/p'" % feed[counter])

But it does not work, if I escape the string using r""" and hard code in the
web address rather than use %s and feed[counter] it works, my question is
how do I escape the string to get it to work with the %s and feed[counter]

Im new to python as you can tell

 

[Tim Jarman]

Rigga wrote:

(snip)

This is the command I am trying to run:

feed is a list of web addresses

output, input = popen2("wget -q %s -O - | tr '\r' '\n' | tr \' \" | sed -n
's/.*url="\([^"]*\)".*/\1/p'" % feed[counter])

But it does not work, if I escape the string using r""" and hard code in
the web address rather than use %s and feed[counter] it works, my question
is how do I escape the string to get it to work with the %s and
feed[counter]

Im new to python as you can tell

Disclaimer: I know nothing about wget beyond what just having typed 'man
wget' told me!

1. What *exactly* does "it does not work" mean? Do you get a traceback? If
so, post it - that will help others to help you. What results are you
expecting?

2. Are you sure feed contains what you think it contains at this point in
your program? What do you see if you do:

for thing in feed: print thing

?

A good strategy in these cases is to go in small steps. before you try
getting fancy with popen2, have your program just print the command-line
correctly. Then maybe try it with something like echo just to see that
you're passing what you think you're passing. And so on.

We were all new once - no blame!

 

[Steve Holden]

Tim Jarman wrote:

Rigga wrote:



(snip stuff about raw strings)



I'm not sure I understand the question. Say you have:

parameter = r"my \funky \text"

then surely you just pass it to your external program using whichever
method you like, e.g.

import os
os.execl("your_external_prog", parameter) # replaces the current process

or some variant of:

return_code = os.spawnl(os.P_WAIT, "your_external_prog", parameter)

or you can build a command line:

command = "your_external_prog %s" % parameter
return_code = os.system(command)

(see docs on the os module for more variations on this theme than you can
shack a stick at)

It's just a string, after all.

This is the command I am trying to run:

feed is a list of web addresses

output, input = popen2("wget -q %s -O - | tr '\r' '\n' | tr \' \" | sed -n
's/.*url="\([^"]*\)".*/\1/p'" % feed[counter])

But it does not work, if I escape the string using r""" and hard code in the
web address rather than use %s and feed[counter] it works, my question is
how do I escape the string to get it to work with the %s and feed[counter]

Im new to python as you can tell

Right, using raw strings (r" ... ") makes sure that backslashes in the
literal are retained rather than used as escapes. Socould you show us an
example where the expression (using ... % feed[counter]) gives you a
different value from "hard coding the web address"?

I suspect if you use a raw string for the format then that will be
enough - in other words, does

output, input = popen2(r"""wget -q %s -O - | tr '\r' '\n' | tr \' \" |
sed -n 's/.*url="\([^"]*\)".*/\1/p'""" % feed[counter])

work?

regards
Steve

 

[Rigga]

Thanks to all for your help it is now working, I rant he code through a
debugger and found that the input file I was using to create my list of
addresses to wget had newlines in them and were therefore breaking my
command line.

All your advice has been appreciated.

RiGGa