determine if os.system() is done

[Xah Lee]

suppose i'm calling two system processes, one to unzip, and one to
“tail†to get the last line. How can i determine when the first
process is done?

Example:

subprocess.Popen([r"/sw/bin/gzip","-d","access_log.4.gz"]);

last_line=subprocess.Popen([r"/usr/bin/tail","-n 1","access_log.4"],
stdout=subprocess.PIPE).communicate()[0]

of course, i can try workarounds something like os.system("gzip -d
thiss.gz && tail thiss"), but i wish to know if there's non-hack way to
determine when a system process is done.

Xah
(e-mail address removed)
∑ http://xahlee.org/

 

[Alessandro Bottoni]

of course, i can try workarounds something like os.system("gzip -d
thiss.gz && tail thiss"), but i wish to know if there's non-hack way to
determine when a system process is done.

Well, if you use a function of the "popen" family, you get some kind of
return value from the subprocess on your "output" pipe. You should be able
to determine if your subprocess has terminated by examining (parsing) this
output pipe.

If you use os.system(), you should get a single return value (usually "None"
or a error code) that you can use for this task.

In both cases, you may have to use a loop (sleep(x)) to wait for the return
value (and check it) from within your code.

Have a look at the docu of the os.process module for details. (Maybe the
newer "subprocess" module is a better choice...)

HTH

 

[Thomas Bellman]

suppose i'm calling two system processes, one to unzip, and one to
tail to get the last line. How can i determine when the first
process is done?

Example:

subprocess.Popen([r"/sw/bin/gzip","-d","access_log.4.gz"]);

last_line=subprocess.Popen([r"/usr/bin/tail","-n 1","access_log.4"],
stdout=subprocess.PIPE).communicate()[0]

of course, i can try workarounds something like os.system("gzip -d
thiss.gz && tail thiss"), but i wish to know if there's non-hack way to
determine when a system process is done.

Have you tried reading the manual for the subprocess module? You
just *might* find the answer to your question if you look at what
you can do with Popen objects. Actually, just learning about the
exact semantics of the communicate() method might be enought to
solve your problem.

 

[Fredrik Lundh]

Have you tried reading the manual for the subprocess module?

han har försökt, men hans tourette tog överhanden:

http://mail.python.org/pipermail/python-list/2005-September/297642.html

</F>

 

[Jeremy Jones]

suppose i'm calling two system processes, one to unzip, and one to
tail to get the last line. How can i determine when the first
process is done?

Example:

subprocess.Popen([r"/sw/bin/gzip","-d","access_log.4.gz"]);

last_line=subprocess.Popen([r"/usr/bin/tail","-n 1","access_log.4"],
stdout=subprocess.PIPE).communicate()[0]




of course, i can try workarounds something like os.system("gzip -d
thiss.gz && tail thiss"), but i wish to know if there's non-hack way to
determine when a system process is done.

Have you tried reading the manual for the subprocess module? You
just *might* find the answer to your question if you look at what
you can do with Popen objects.

Oh, come on. Don't you know that all Python documentation is rubbish
and not worth reading, written by IT idiots who throw around useless
jargon and indulge in extreme forms of self-gratification? Someone of
the caliber of Xah Lee would *never* stoop so low as to actually read
the documentation. It is beneath him. Instead, he posts messages to a
group of IT idiots who throw around useless jargon and indulge in
extreme forms of self-gratification in posting answers to questions.

<snip>

JMJ

 

[Martin Franklin]

suppose i'm calling two system processes, one to unzip, and one to
“tail†to get the last line. How can i determine when the first
process is done?

Example:

subprocess.Popen([r"/sw/bin/gzip","-d","access_log.4.gz"]);

last_line=subprocess.Popen([r"/usr/bin/tail","-n 1","access_log.4"],
stdout=subprocess.PIPE).communicate()[0]

of course, i can try workarounds something like os.system("gzip -d
thiss.gz && tail thiss"), but i wish to know if there's non-hack way to
determine when a system process is done.

Xah
(e-mail address removed)
∑ http://xahlee.org/

I think the idea is you wait for the first call to subprocess.call to
finish before executing the second...



http://docs.python.org/lib/node231.html


call( *args, **kwargs)
Run command with arguments. *Wait for command to complete*, then
return the returncode attribute.

The arguments are the same as for the Popen constructor. Example:

retcode = call(["ls", "-l"])

 

[Nainto]

Yeah, I agree. The Python documentation just merey describes what
arguements a function can take not as much how to use the actual
function.

 

[Fredrik Lundh]

Yeah, I agree. The Python documentation just merey describes what
arguements a function can take not as much how to use the actual
function.

yeah, that's a really relevant criticism when we're talking about a
module that contains one function and one class, and for which the
documentation contains *sixteen* examples.

</F>

 

[Lars =?iso-8859-1?q?Gust=E4bel?=]

[Fredrik Lundh]

han har försökt, men hans tourette tog överhanden:

IMHO it's more likely an Asperger's syndrome.

http://en.wikipedia.org/wiki/Asperger_Syndrome

--
Lars Gustäbel
(e-mail address removed)

Any sufficiently advanced technology is indistinguishable
from magic.
(Arthur C. Clarke)

 

[Steve Horsley]

suppose i'm calling two system processes, one to unzip, and one to
“tail†to get the last line. How can i determine when the first
process is done?

Example:

subprocess.Popen([r"/sw/bin/gzip","-d","access_log.4.gz"]);

last_line=subprocess.Popen([r"/usr/bin/tail","-n 1","access_log.4"],
stdout=subprocess.PIPE).communicate()[0]

of course, i can try workarounds something like os.system("gzip -d
thiss.gz && tail thiss"), but i wish to know if there's non-hack way to
determine when a system process is done.

Xah
(e-mail address removed)
∑ http://xahlee.org/

As far as I can tell from the docs (worth reading),
system(command) doesn't return until the process has completed.
It would have to do some fancy footwork to return the exit code
BEFORE the process had completed!

Steve

 

[Martin P. Hellwig]

[Fredrik Lundh]

han har försökt, men hans tourette tog överhanden:

IMHO it's more likely an Asperger's syndrome.

http://en.wikipedia.org/wiki/Asperger_Syndrome

I disagree, in his writings I found no evidence of autisme.
Actually most of it can be classified as being stubborn against better
knowledge, what actually a common thing is.

But he does ask the question and by this does admits he is not
knowledged in that topic.

The only thing I am disappointed at his writing style, most likely he
has a disrupted view on social acceptable behavior and communication.
These skills might be still in development, so perhaps it is reasonable
to give him a chance and wait until he is out of his puberty.

 

[Peter Hansen]

The only thing I am disappointed at his writing style, most likely he
has a disrupted view on social acceptable behavior and communication.
These skills might be still in development, so perhaps it is reasonable
to give him a chance and wait until he is out of his puberty.

He's 37 years old! How long should one be given to mature?

-Peter

 

[Xah Lee]

Thanks all.

I found the answer, rather easily.
To make a system call and wait for it, do:

subprocess.Popen([r"/sw/bin/gzip","-d","access_log.4.gz"]).wait();

------
this post is archived at:
http://xahlee.org/perl-python/system_calls.html

Xah
(e-mail address removed)
∑ http://xahlee.org/

 

[Martin P. Hellwig]

He's 37 years old! How long should one be given to mature?

Yeah well, eeh I guess that rules out the development part.
Although puberty can take a long time (according to my SO)

 

[Martin Franklin]

suppose i'm calling two system processes, one to unzip, and one to
“tail†to get the las

t line. How can i determine when the first

process is done?

Example:

subprocess.Popen([r"/sw/bin/gzip","-d","access_log.4.gz"]);

last_line=subprocess.Popen([r"/usr/bin/tail","-n 1","access_log.4"],
stdout=subprocess.PIPE).communicate()[0]

of course, i can try workarounds something like os.system("gzip -d
thiss.gz && tail thiss"), but i wish to know if there's non-hack way to
determine when a system process is done.

Xah
(e-mail address removed)
∑ http://xahlee.org/

I know you've found the answer to your question, however for the exact
example you gave a much better solution comes to mind...


import gzip

log_file = gzip.open("access_log.4.gz")
last_line = log_file.readlines()[-1]
log_file.close()


Regards
Martin

 

[Xah Lee]

import gzip
log_file = gzip.open("access_log.4.gz")
last_line = log_file.readlines()[-1]
log_file.close()

does the
log_file.readlines()[-1]
actually read all the lines first?

i switched to system call with tail because originally i was using a
pure Python solution

inF = gzip.GzipFile(ff, 'rb');
s=inF.readlines()
inF.close()
last_line=s[-1]

and since the log file is 100 megabytes it takes a long time and hogs
massive memory.

Xah
(e-mail address removed)
∑ http://xahlee.org/

 

[Martin Franklin]

import gzip
log_file = gzip.open("access_log.4.gz")
last_line = log_file.readlines()[-1]
log_file.close()

does the
log_file.readlines()[-1]
actually read all the lines first?

Yes I'm afraid it does.

i switched to system call with tail because originally i was using a
pure Python solution

inF = gzip.GzipFile(ff, 'rb');
s=inF.readlines()
inF.close()
last_line=s[-1]

and since the log file is 100 megabytes it takes a long time and hogs
massive memory.

Ok, in that case stick to your shell based solution, although 100
megabytes does not sound that large to me I guess it is relative
to the system you are running on (I have over a gig of memory here)

 

[Martin Franklin]

Martin Franklin wrote:

import gzip
log_file = gzip.open("access_log.4.gz")
last_line = log_file.readlines()[-1]
log_file.close()

does the
log_file.readlines()[-1]
actually read all the lines first?

Yes I'm afraid it does.

i switched to system call with tail because originally i was using a
pure Python solution

inF = gzip.GzipFile(ff, 'rb');
s=inF.readlines()
inF.close()
last_line=s[-1]

and since the log file is 100 megabytes it takes a long time and hogs
massive memory.

Ok, in that case stick to your shell based solution, although 100
megabytes does not sound that large to me I guess it is relative
to the system you are running on (I have over a gig of memory here)

And just a few minutes after I sent that... this...

import gzip

logfile = gzip.open("access_log.4.BIG.gz")

## seek relative to the end of the file
logfile.seek(-500)

last_line = logfile.readlines()[-1]

logfile.close()

print last_line


Works quite fast on my machine...

Regards
Martin

 

[Martin Franklin]

Xah Lee wrote:

Martin Franklin wrote:



import gzip
log_file = gzip.open("access_log.4.gz")
last_line = log_file.readlines()[-1]
log_file.close()


does the
log_file.readlines()[-1]
actually read all the lines first?

Yes I'm afraid it does.

i switched to system call with tail because originally i was using a
pure Python solution

inF = gzip.GzipFile(ff, 'rb');
s=inF.readlines()
inF.close()
last_line=s[-1]

and since the log file is 100 megabytes it takes a long time and hogs
massive memory.

Ok, in that case stick to your shell based solution, although 100
megabytes does not sound that large to me I guess it is relative
to the system you are running on (I have over a gig of memory here)

And just a few minutes after I sent that... this...

import gzip

logfile = gzip.open("access_log.4.BIG.gz")

## seek relative to the end of the file
logfile.seek(-500)

last_line = logfile.readlines()[-1]

logfile.close()

print last_line


Works quite fast on my machine...

whoops, no it doesn't looking at wrong window just ignore
me please

 

[Fredrik Lundh]

Ok, in that case stick to your shell based solution, although 100
megabytes does not sound that large to me I guess it is relative
to the system you are running on (I have over a gig of memory here)

since the file is gzipped, you need to read all of it to get the last line.
however, replacing

last_line = logfile.readlines()[-1]

with a plain

for last_line in logfile: pass

avoids storing the entire file in memory.

(note that seek() actually reads the file in 1024 blocks until it gets
to the right location; reading via a line iterator only seems to be
marginally slower. zcat|tail is a LOT faster. ymmv.)

</F>