subprocess and & (ampersand)

[Steven Bethard]

I'm having trouble using the subprocess module on Windows when my
command line includes special characters like "&" (ampersand)::
.... stdout=subprocess.PIPE,
.... stderr=subprocess.PIPE)"'y' is not recognized as an internal or external command,\r\noperable
program or batch file.\r\n"

As you can see, Windows is interpreting that "&" as separating two
commands, instead of being part of the single argument as I intend it to
be above. Is there any workaround for this? How do I get "&" treated
like a regular character using the subprocess module?

Thanks,

STeVe

 

[Steven D'Aprano]

I'm having trouble using the subprocess module on Windows when my
command line includes special characters like "&" (ampersand)::

... stdout=subprocess.PIPE, ...
stderr=subprocess.PIPE)
"'y' is not recognized as an internal or external command,\r\noperable
program or batch file.\r\n"

As you can see, Windows is interpreting that "&" as separating two
commands, instead of being part of the single argument as I intend it to
be above. Is there any workaround for this? How do I get "&" treated
like a regular character using the subprocess module?

That's nothing to do with the subprocess module. As you say, it is
Windows interpreting the ampersand as a special character, so you need to
escape the character to the Windows shell.

Under Windows, the escape character is ^, or you can put the string in
double quotes:

# untested
command = 'lynx.bat -dump http://www.example.com/?x=1^&y=2'
command = 'lynx.bat -dump "http://www.example.com/?x=1&y=2"'

In Linux land, you would use a backslash or quotes.

To find the answer to this question, I googled for "windows how to escape
special characters shell" and found these two pages:


http://www.microsoft.com/technet/archive/winntas/deploy/prodspecs/shellscr.mspx

http://technet2.microsoft.com/WindowsServer/en/library/44500063-fdaf-4e4f-8dac-476c497a166f1033.mspx


Hope this helps,

 

[Steven Bethard]

That's nothing to do with the subprocess module. As you say, it is
Windows interpreting the ampersand as a special character, so you need to
escape the character to the Windows shell.

Under Windows, the escape character is ^, or you can put the string in
double quotes:

# untested
command = 'lynx.bat -dump http://www.example.com/?x=1^&y=2'
command = 'lynx.bat -dump "http://www.example.com/?x=1&y=2"'

Sorry, I should have mentioned that I already tried that. You get the
same result::
... stdin=subprocess.PIPE,
... stdout=subprocess.PIPE,
... stderr=subprocess.PIPE) "'y' is not recognized as an internal or external command,\r\noperable
program or batch file.\r\n"

In fact, the "^" doesn't seem to work at the command line either::

>lynx.bat -dump http://www.example.com/?x=1^&y=2

Can't Access `file://localhost/C:/PROGRA~1/lynx/1'
Alert!: Unable to access document.

lynx: Can't access startfile
'y' is not recognized as an internal or external command,
operable program or batch file.

Using quotes does work at the command line::

C:\PROGRA~1\lynx>lynx.bat -dump "http://www.example.com/?x=1&y=2"
You have reached this web page by typing "example.com",
"example.net", or "example.org" into your web browser.

These domain names are reserved for use in documentation and are
not available for registration. See [1]RFC 2606, Section 3.

References

1. http://www.rfc-editor.org/rfc/rfc2606.txt

But I get no output at all when using quotes with subprocess::
... stdin=subprocess.PIPE,
... stdout=subprocess.PIPE,
... stderr=subprocess.PIPE) ''

Any other ideas?

STeVe

 

[Tim Golden]

I'm having trouble using the subprocess module on Windows when my
command line includes special characters like "&" (ampersand)::

... stdout=subprocess.PIPE,
... stderr=subprocess.PIPE)
"'y' is not recognized as an internal or external command,\r\noperable
program or batch file.\r\n"

As you can see, Windows is interpreting that "&" as separating two
commands, instead of being part of the single argument as I intend it to
be above. Is there any workaround for this? How do I get "&" treated
like a regular character using the subprocess module?

A little experimentation suggests that the problem's somehow
tied up with the .bat file. ie this works for me (doubly
complicated because of the long firefox path:

<code>
import subprocess

cmd = [
r"c:\Program Files\Mozilla Firefox\firefox.exe",
"http://local.goodtoread.org/search?word=tim&cached=0"
]
subprocess.Popen (cmd)

</code>

but this doesn't:

<c:/temp/firefox.bat>
"c:\Program Files\Mozilla Firefox\firefox.exe" "%*"
</c:/temp/firefox.bat>

<code>
import subprocess

cmd = [
r"c:\temp\firefox.bat",
"http://local.goodtoread.org/search?word=tim&cached=0"
]
subprocess.Popen (cmd)

</code>

although, interestingly, it seems to cut off at the
"=" before the "&". Not sure how significant that is.

So, even assuming we're looking at the same situation,
I suppose one solution for you is to break out the
..bat file. But that may not be a possibility.

TJG

 

[Tim Golden]

I'm having trouble using the subprocess module on Windows when my
command line includes special characters like "&" (ampersand)::

... stdout=subprocess.PIPE,
... stderr=subprocess.PIPE)
"'y' is not recognized as an internal or external command,\r\noperable
program or batch file.\r\n"

As you can see, Windows is interpreting that "&" as separating two
commands, instead of being part of the single argument as I intend it to
be above. Is there any workaround for this? How do I get "&" treated
like a regular character using the subprocess module?

Although I'm sure you'll have looked into this already, the
subprocess module on Windows is using CreateProcess pretty
straightforwardly:

http://msdn2.microsoft.com/en-us/library/ms682425(VS.85).aspx

The docs there say that, to call a batch file, you need to
specify the command interpreter with /c and pass the batch
file, but as far as I can see it makes no difference! (Probably
means there's a special-caser behind the scenes of CreateProcess).

TJG

 

[Ross Ridge]

but this doesn't:

<c:/temp/firefox.bat>
"c:\Program Files\Mozilla Firefox\firefox.exe" "%*"
</c:/temp/firefox.bat>

<code>
import subprocess

cmd = [
r"c:\temp\firefox.bat",
"http://local.goodtoread.org/search?word=tim&cached=0"
]
subprocess.Popen (cmd)

</code>

You need to use double quotes both in the .BAT file and in the string
you pass to subprocess.Popen().

Ross Ridge

 

[Tim Golden]

but this doesn't:

<c:/temp/firefox.bat>
"c:\Program Files\Mozilla Firefox\firefox.exe" "%*"
</c:/temp/firefox.bat>

<code>
import subprocess

cmd = [
r"c:\temp\firefox.bat",
"http://local.goodtoread.org/search?word=tim&cached=0"
]
subprocess.Popen (cmd)

</code>

You need to use double quotes both in the .BAT file and in the string
you pass to subprocess.Popen().

Ross Ridge

In the context of my example above, could you just
say which bit you thing should be quoted and isn't?
(That sounds sarcastic, but isn't; I just want to
understand if I've missed something). If you simply
requote the second element in the cmd list
('"http:/....."') then the internal quotes are escaped
by some part of the mechanism and it still doesn't work.

TJG

 

[Ross Ridge]

but this doesn't:

<c:/temp/firefox.bat>
"c:\Program Files\Mozilla Firefox\firefox.exe" "%*"
</c:/temp/firefox.bat>

<code>
import subprocess

cmd = [
r"c:\temp\firefox.bat",
"http://local.goodtoread.org/search?word=tim&cached=0"
]
subprocess.Popen (cmd)

</code>

You need to use double quotes both in the .BAT file and in the string
you pass to subprocess.Popen().

... If you simply requote the second element in the cmd list
('"http:/....."') then the internal quotes are escaped by some part of
the mechanism and it still doesn't work.

Hmm... I guess things are much more messy than that. CMD doesn't do
standard quote processing of it's arguments or .BAT file arguments, and
so is incompatible with how subprocess quotes args lists. It looks like
you need use a string instead of list with subprocess.Popen and not use
quotes in the batch file. So something like:

firefox.bat:
"c:\Program Files\Mozilla Firefox\firefox.exe" %1

code:
subprocess.Popen(r'c:\temp\firefox.bat "http://local.goodtoread.org/search?word=tim&cached=0"')

Ross Ridge

 

[Steven Bethard]

but this doesn't:

<c:/temp/firefox.bat>
"c:\Program Files\Mozilla Firefox\firefox.exe" "%*"
</c:/temp/firefox.bat>

<code>
import subprocess

cmd = [
r"c:\temp\firefox.bat",
"http://local.goodtoread.org/search?word=tim&cached=0"
]
subprocess.Popen (cmd)

</code>

You need to use double quotes both in the .BAT file and in the string
you pass to subprocess.Popen().

... If you simply requote the second element in the cmd list
('"http:/....."') then the internal quotes are escaped by some part of
the mechanism and it still doesn't work.

Hmm... I guess things are much more messy than that. CMD doesn't do
standard quote processing of it's arguments or .BAT file arguments, and
so is incompatible with how subprocess quotes args lists. It looks like
you need use a string instead of list with subprocess.Popen and not use
quotes in the batch file. So something like:

firefox.bat:
"c:\Program Files\Mozilla Firefox\firefox.exe" %1

code:
subprocess.Popen(r'c:\temp\firefox.bat "http://local.goodtoread.org/search?word=tim&cached=0"')

This works. The other thing that works is to put extra quotes around
the URL when you pass in the list::

... stdin=subprocess.PIPE,
... stdout=subprocess.PIPE,
... stderr=subprocess.PIPE) ' You have reached this web page by typing "example.com",
"example.net",\n or "example.org" into your web browser.\n\n These
domain names are reserved for use in documentation and are not\n
available for registration. See [1]RFC 2606, Section
3.\n\nReferences\n\n 1. http://www.rfc-editor.org/rfc/rfc2606.txt\n'


I tried this before, and posted it, but I had mistakenly read from
stderr instead of stdout. Thanks for making me double-check that code!

STeVe

 

[Albert van der Horst]

So, even assuming we're looking at the same situation,
I suppose one solution for you is to break out the
.bat file. But that may not be a possibility.

Unless you need compatibility with Windows 3.1., I suggest you
stay away from .bat files. Better call them .cmd, such that they
are executed by cmd.exe (32 bits).
It could be part of your problem.

TJG

Groetjes Albert