using python on the command line with 'here-documents' and pipes

[calmar]

Hi all,

I would like to use python for a replacement for some binutils. I would
like to be able to pipe things into python. Actually I would not like
writing a 'script' to handle the input, but write the python commands
right onto the command line.

It would be possible to use here-documents (for having the correct
identation during e.g while loops) for writing python code on
the fly, but then how and where to get the pipe output:

echo -e 'line1\nline2' | python <<EOF ....

Does not really work...

Would anybody have an idea how to use python like e.g. awk etc. As a
tool within, where I can handle the stdin data, and write the code right
there (e.g. with here documents)?


thanks a lot
calmar

 

[Pekka Niiranen]

Hi,

This is from C:\Python23\Lib\site-packages\win32\Demos\pipes\cat.py:

'''cat.py
a version of unix cat, tweaked to show off runproc.py
'''

import sys
data = sys.stdin.read(1)
sys.stdout.write(data)
sys.stdout.flush()
while data:
data = sys.stdin.read(1)
sys.stdout.write(data)
sys.stdout.flush()
# Just here to have something to read from stderr.
sys.stderr.write("Blah...")

# end of cat.py

-pekka-

 

[Roy Smith]

I would like to be able to pipe things into python.

Python is perfectly happy reading stdin and writing to stdout, so it's
also perfectly happy acting as a component in a pipe.

Actually I would not like
writing a 'script' to handle the input, but write the python commands
right onto the command line.

You can do this with the -c command line option:

$ python -c 'print "hello world"'
hello world

It would be possible to use here-documents (for having the correct
identation during e.g while loops) for writing python code on
the fly, but then how and where to get the pipe output:

I think you're confusing two different things: where python gets the
code it's going to execute, and where it gets the input it's going to
feed to that executing code. You can use here-is documents for the
input, and -c for the code, and end up with something like this:

$ python -c 'import sys

for line in sys.stdin:
print line[:10]

' << EOF
line one of many
line two of many
line three of many
EOF

line one o
line two o
line three

Whether or not the single-quote mechanism reads past newlines and
properly preserves indentation is more a function of what shell you're
using than the python interpreter itself (the above was done with bash).

Would anybody have an idea how to use python like e.g. awk etc. As a
tool within, where I can handle the stdin data, and write the code right
there (e.g. with here documents)?

Well, the above will work, but I don't think it's a particularly good
idea. I think you would do better putting the text of your python
script in a .py file, make the first line #!/usr/bin/python, and make it
executable. Then you've got a stand-alone program that you can edit and
maintain apart from the shell script that surrounds it. In the long
run, this is likely to be simpler to test and to maintain.

Of course, if you want to be a bit perverse, you can always do:

$ python /dev/stdin << EOF

print "hello, bizarre world"
EOF

hello, bizarre world

but I think that's more of a curiosity than a useful tool. Along the
same lines, I once knew somebody who linked /dev/tty.c to /dev/tty. He
could then do "cc /dev/tty.c" and type in c code directly to the
compiler". He was a sick puppy.

 

[calmar]

hi,

$ python -c 'import sys

for line in sys.stdin:
print line[:10]

' << EOF
line one of many
line two of many
line three of many
EOF

line one o
line two o
line three

Yeah, that works nicely.

Actually, I didn't find the solution to just press enter (after a -c
'..) and write the idented code I want.

executable. Then you've got a stand-alone program that you can edit and
maintain apart from the shell script that surrounds it. In the long
run, this is likely to be simpler to test and to maintain.

Yeah, true.


Thanks a lot to all!

 

[calmar]

hi,

Can you post a working example for awk? I'll admit I'm not a shell
expert,
but it seems like you are trying to stuff two files onto stdin, the
"echo"
result and the python program. I don't understand how that could ever
work, and now you've piqued my curiosity.

Well, it would be something like (easy example):

echo -e 'line1\nline2\nline3' | python -c '
import sys
for line in sys.stdin:
print line'

That works now (with bash). I get the text into python and can do there
whatever I want. Awk/Sed/grep or whatever kind of tasks.

My problem, I didn't find the solution to just write the code after the
-c '...' including newlines. Sooo easy, but I didn't find that solution
(not even after hours doing curious stuff

Cheers
calmar

PS: would there be a smarter way to get the
stdin lines into a loop?

 

[Jeremy Bowers]

It would be possible to use here-documents (for having the correct
identation during e.g while loops) for writing python code on the fly, but
then how and where to get the pipe output:

echo -e 'line1\nline2' | python <<EOF ....

Does not really work...

Would anybody have an idea how to use python like e.g. awk etc. As a tool
within, where I can handle the stdin data, and write the code right there
(e.g. with here documents)?

Can you post a working example for awk? I'll admit I'm not a shell expert,
but it seems like you are trying to stuff two files onto stdin, the "echo"
result and the python program. I don't understand how that could ever
work, and now you've piqued my curiosity.

 

[Nick Craig-Wood]

echo -e 'line1\nline2\nline3' | python -c '
import sys
for line in sys.stdin:
print line'

The fileinput module is useful, it reads from stdin or from the files
provided on the command line (sys.argv), eg

$ python -c '
import fileinput, sys
for line in fileinput.input():
# Your code to manipulate line here
sys.stdout.write(line)
'
This is a line
This is another line
This is a line
This is another line

(The above is (almost) equivalent to perl -pe '# Your code here' BTW
which was in itself derived from the awk usage I think. The almost
comes in when stdin is from the terminal - you need to press CTRL-D
twice for the python version and once for the perl version)