atexit handlers - getting the return code

[Mike Hull]

Hi,
I work in neuroscience modelling and use python for lots of my work.
One problem I have is management of scripts and results. To that end,
I have
written code that make its easier to keep track of scripts, to see
which ones
have run, or need to be rerun, to see output errors and generally
organise running of scripts better.

The way i have implemented it is that I register a function to the
atexit module,
which takes care of recording information about the script run; such
as whether an
exception was raised and not caught, and how long it took to run and
the
stdout/stderr streams; which can then be uploaded into a database.

One thing I am struggling to get though is the 'return code' that the
is going to be returned after my
atexit handlers have finished. Could anyone tell me how it get at
this!?


Thanks


Mike

 

[Giampaolo Rodolà]

You mean the registered function return value or the process return code?
In the first case you can do something such as:

import atexit

@atexit.register
def cleanup():
def actual_function():
...
return something
ret = actual_function()
# do something with ret

In the latter case you can use sys.exit(code).

--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/

 

[Mike Hull]

Hi Giampaolo,
Sorry, I didn't explain very clearly.
I have a python file, 'simulation_logger.py', (given below).
Then, in my scripts, I add the following lines to the top,



#simulation1.py:
###################
from simulation_logger import SimulationDecorator
SimulationDecorator.Init()

# Rest of the simulation script....

#####################

and at the end of the simulation, it should write the stdout/stderr,
the return code and any top level exception details into a database.
This is working except I can't work out how to get the return code
the script is going to return.
Any help would be gratefully appreciated,

Thanks!

Mike










# simulation_logger.py
########################

import atexit
import sys
import os
import cStringIO
import time
import traceback
import datetime
import inspect

class SimulationRunInfo(object):
def __init__(self, script_name ):
self.return_code = None
self.time_taken = None
self.time_out = None
self.std_out = None
self.std_err = None
self.exception_details = None,None

self.script_name = script_name



class IOStreamDistributor(object):
def __init__(self, outputs):
self.outputs = outputs

def write(self, *args, **kwargs):
for op in self.outputs:
op.write(*args, **kwargs)




class SimulationDecorator(object):

start_time = None
time_out = None
is_initialised = False
exception_details = None,None

std_out = None
std_err = None
script_name = None





@classmethod
def exit_handler(cls, *args, **kwargs):
print 'Exit Handler'
print 'Args', args
print 'KwArgs', kwargs

info = SimulationRunInfo( cls.script_name)

# Read and restore the StdOut/Err
info.std_out = cls.std_out.getvalue()
sys.stdout = sys.__stdout__
info.std_err = cls.std_err.getvalue()
sys.stderr = sys.__stderr__

# Get the return value:
info.return_code = 0

# Get the timing:
info.time_taken = int( time.time() - cls.start_time )

# Has thier been an exception?
info.exception_details = cls.exception_details
if info.exception_details != (None,None):
print 'Exception details', info.exception_details
info.return_code = -1

# Write to SimulationDataBase

# Save the information to a database:
SimulationDBWriter.write_to_database(info)


@classmethod
def top_level_exception_handler(cls, exception_type, exception,
traceback, *args):
print 'TopLevel Exception Handler'
cls.exception_details = exception_type, exception, traceback

@classmethod
def Init(cls, time_out=None):
assert not cls.is_initialised

if 'MF_TIMEOUT' in os.environ:
timeout = int( os.environ['MF_TIMEOUT'] )


# Filename of the Simulation script
cwd = os.getcwd()
cls.script_name = os.path.join( cwd, traceback.extract_stack()
[0][0] )
#cls.script_name = traceback.extract_stack()[0][2].
#cls.script_name = inspect.stack()[-1][1]

#Intercept StdOut and StdErr:
cls.std_out = cStringIO.StringIO()
sys.stdout = IOStreamDistributor( [cls.std_out, sys.stdout] )
cls.std_err = cStringIO.StringIO()
sys.stderr = IOStreamDistributor( [cls.std_err, sys.stderr] )

# Set an exit handler and a top level exception-handler
# Set a top level exception handler:
atexit.register( cls.exit_handler )
sys.excepthook = cls.top_level_exception_handler


# Set a time-out alarm
cls.start_time = time.time()
cls.time_out = time_out

 

[Robert Kern]

Hi Giampaolo,
Sorry, I didn't explain very clearly.
I have a python file, 'simulation_logger.py', (given below).
Then, in my scripts, I add the following lines to the top,



#simulation1.py:
###################
from simulation_logger import SimulationDecorator
SimulationDecorator.Init()

# Rest of the simulation script....

#####################

and at the end of the simulation, it should write the stdout/stderr,
the return code and any top level exception details into a database.
This is working except I can't work out how to get the return code
the script is going to return.

I don't think that's available inside the process. You may want to use a
"runner" script that records this information. Additionally, the runner script
could record the stdout/stderr information more cleanly than stubbing out
sys.stdout/sys.stderr. For example, if you have C or Fortran modules that use
their own mechanisms to print things directly to the STDOUT/STDERR file
descriptors, your sys.stdout/sys.stderr stubs will never be informed about it.
However, if the runner script directs stdout/stderr to a pipe, it can read every
bit of text that gets printed. Timing is probably also best recorded by the
runner script to record the startup overhead of the Python interpreter. Continue
to use the SimulationDecorator to record the traceback information, though.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Mike Hull]

I don't think that's available inside the process. You may want to use a
"runner" script that records this information. Additionally, the runner script
could record the stdout/stderr information more cleanly than stubbing out
sys.stdout/sys.stderr. For example, if you have C or Fortran modules thatuse
their own mechanisms to print things directly to the STDOUT/STDERR file
descriptors, your sys.stdout/sys.stderr stubs will never be informed about it.
However, if the runner script directs stdout/stderr to a pipe, it can read every
bit of text that gets printed. Timing is probably also best recorded by the
runner script to record the startup overhead of the Python interpreter. Continue
to use the SimulationDecorator to record the traceback information, though.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

Hi Robert,
Thanks for this. The return code is not super important at this stage,
but I thought I would
try and get everything working neatly. Creating a separate runner
script is probably the way
to go. If i get some time I will implement it like that. My reason for
wanting to do it 'in
the same script' was that I also have another library that intercepts
calls to matplotlib's show(),
so I could ultimately create a pdf containing the script with figures
interjected at the right place.
Anyway, I will have to think about that some more.

Thanks again!

Mike