Is there a more elegant way to handle determing fail status?

[J]

Ok, so I have a diagnostic tool, written by someone else. That tool
runs a series of small tests defined by the user and can simplified
summary output that can be one of the following:

FAILED_CRITICAL
FAILED_HIGH
FAILED_MEDIUM
FAILED_LOW
PASSED

I also have a wrapper script I wrote to run these tests, summarize the
results of all tests aggregated and then fail based on a particular
fail level.

The idea is that if I run 3 tests with the diagnostic tool and it
tells me the following:

testA: PASSED
testB: FAILED_MEDIUM
testC: PASSED

AND I told the wrapper to only fail on HIGH or above, the wrapper will
tell me that I had a medium failure, but the wrapper will still exit
with a 0 (success)

if I get the same results as above, but tell the wrapper to fail on
LOW, then it will tell me I had that medium failure, but the wrapper
will exit with a 1 (failure).

The problem is that my exit determination looks like this:

if fail_priority == fail_levels['FAILED_CRITICAL']:
if critical_fails:
return 1
if fail_priority == fail_levels['FAILED_HIGH']:
if critical_fails or high_fails:
return 1
if fail_priority == fail_levels['FAILED_MEDIUM']:
if critical_fails or high_fails or medium_fails:
return 1
if fail_priority == fail_levels['FAILED_LOW']:
if critical_fails or high_fails or medium_fails or low_fails:
return 1

return 0

So, to explain the above... the fail level can be set by the user when
running the wrapper using -f (or it defaults to 'high')
the wrapper assigns a number to each level using this:

# Set correct fail level
args.fail_level = 'FAILED_%s' % args.fail_level.upper()

# Get our failure priority and create the priority values
fail_levels = {'FAILED_CRITICAL':4,
'FAILED_HIGH':3,
'FAILED_MEDIUM':2,
'FAILED_LOW':1}
fail_priority = fail_levels[args.fail_level]

the variables critical_fails, high_fails, medium_fails, low_fails are
all counters that are etiher None, or the number of tests that were
failed.

So using this output from the diagnostic tool:

testA: PASSED
testB: FAILED_HIGH
testC: PASSED
testD: FAILED_MEDIUM
testE: PASSED

critical_fails would be None
high_fails would be 1
medium_fails would be 1
low_fails would be None.

The exit code determination above works, but it just feels inelegant.
It feels like there's a better way of implementing that, but I can't
come up with one that still honors the fail level properly (e.g. other
solutions will fail on medium, but won't fail properly on medium OR
higher).

I can provide the full script if necessary, if the above isn't enough
to point me in a direction that has a better way of doing this...

Thanks for looking,

Jeff

 

[donarb]

Ok, so I have a diagnostic tool, written by someone else. That tool

runs a series of small tests defined by the user and can simplified

summary output that can be one of the following:



FAILED_CRITICAL

FAILED_HIGH

FAILED_MEDIUM

FAILED_LOW

PASSED



I also have a wrapper script I wrote to run these tests, summarize the

results of all tests aggregated and then fail based on a particular

fail level.



The idea is that if I run 3 tests with the diagnostic tool and it

tells me the following:



testA: PASSED

testB: FAILED_MEDIUM

testC: PASSED



AND I told the wrapper to only fail on HIGH or above, the wrapper will

tell me that I had a medium failure, but the wrapper will still exit

with a 0 (success)



if I get the same results as above, but tell the wrapper to fail on

LOW, then it will tell me I had that medium failure, but the wrapper

will exit with a 1 (failure).



The problem is that my exit determination looks like this:



if fail_priority == fail_levels['FAILED_CRITICAL']:

if critical_fails:

return 1

if fail_priority == fail_levels['FAILED_HIGH']:

if critical_fails or high_fails:

return 1

if fail_priority == fail_levels['FAILED_MEDIUM']:

if critical_fails or high_fails or medium_fails:

return 1

if fail_priority == fail_levels['FAILED_LOW']:

if critical_fails or high_fails or medium_fails or low_fails:

return 1



return 0



So, to explain the above... the fail level can be set by the user when

running the wrapper using -f (or it defaults to 'high')

the wrapper assigns a number to each level using this:



# Set correct fail level

args.fail_level = 'FAILED_%s' % args.fail_level.upper()



# Get our failure priority and create the priority values

fail_levels = {'FAILED_CRITICAL':4,

'FAILED_HIGH':3,

'FAILED_MEDIUM':2,

'FAILED_LOW':1}

fail_priority = fail_levels[args.fail_level]



the variables critical_fails, high_fails, medium_fails, low_fails are

all counters that are etiher None, or the number of tests that were

failed.



So using this output from the diagnostic tool:



testA: PASSED

testB: FAILED_HIGH

testC: PASSED

testD: FAILED_MEDIUM

testE: PASSED



critical_fails would be None

high_fails would be 1

medium_fails would be 1

low_fails would be None.



The exit code determination above works, but it just feels inelegant.

It feels like there's a better way of implementing that, but I can't

come up with one that still honors the fail level properly (e.g. other

solutions will fail on medium, but won't fail properly on medium OR

higher).



I can provide the full script if necessary, if the above isn't enough

to point me in a direction that has a better way of doing this...



Thanks for looking,



Jeff

My back of the envelope coding would do it this way. Use an array of fail_counters, with PASSED as the first element all the way up to FAILED_CRITICALas the last element. Then use a simple loop starting from index fail_priority to the end of the list looking for errors. Like this:

# Array of fail counters
fail_counters = [
0, # PASSED
0, # LOW
0, # MEDIUM
0, # HIGH
0 # CRITICAL
]

.... run tests, accumulate error counts in fail_counters

for i in range(fail_priority, len(fail_counters)):
if fail_counters:
return 1
return 0

 

[donarb]

Ok, so I have a diagnostic tool, written by someone else. That tool

runs a series of small tests defined by the user and can simplified

summary output that can be one of the following:



FAILED_CRITICAL

FAILED_HIGH

FAILED_MEDIUM

FAILED_LOW

PASSED



I also have a wrapper script I wrote to run these tests, summarize the

results of all tests aggregated and then fail based on a particular

fail level.



The idea is that if I run 3 tests with the diagnostic tool and it

tells me the following:



testA: PASSED

testB: FAILED_MEDIUM

testC: PASSED



AND I told the wrapper to only fail on HIGH or above, the wrapper will

tell me that I had a medium failure, but the wrapper will still exit

with a 0 (success)



if I get the same results as above, but tell the wrapper to fail on

LOW, then it will tell me I had that medium failure, but the wrapper

will exit with a 1 (failure).



The problem is that my exit determination looks like this:



if fail_priority == fail_levels['FAILED_CRITICAL']:

if critical_fails:

return 1

if fail_priority == fail_levels['FAILED_HIGH']:

if critical_fails or high_fails:

return 1

if fail_priority == fail_levels['FAILED_MEDIUM']:

if critical_fails or high_fails or medium_fails:

return 1

if fail_priority == fail_levels['FAILED_LOW']:

if critical_fails or high_fails or medium_fails or low_fails:

return 1



return 0



So, to explain the above... the fail level can be set by the user when

running the wrapper using -f (or it defaults to 'high')

the wrapper assigns a number to each level using this:



# Set correct fail level

args.fail_level = 'FAILED_%s' % args.fail_level.upper()



# Get our failure priority and create the priority values

fail_levels = {'FAILED_CRITICAL':4,

'FAILED_HIGH':3,

'FAILED_MEDIUM':2,

'FAILED_LOW':1}

fail_priority = fail_levels[args.fail_level]



the variables critical_fails, high_fails, medium_fails, low_fails are

all counters that are etiher None, or the number of tests that were

failed.



So using this output from the diagnostic tool:



testA: PASSED

testB: FAILED_HIGH

testC: PASSED

testD: FAILED_MEDIUM

testE: PASSED



critical_fails would be None

high_fails would be 1

medium_fails would be 1

low_fails would be None.



The exit code determination above works, but it just feels inelegant.

It feels like there's a better way of implementing that, but I can't

come up with one that still honors the fail level properly (e.g. other

solutions will fail on medium, but won't fail properly on medium OR

higher).



I can provide the full script if necessary, if the above isn't enough

to point me in a direction that has a better way of doing this...



Thanks for looking,



Jeff

My back of the envelope coding would do it this way. Use an array of fail_counters, with PASSED as the first element all the way up to FAILED_CRITICALas the last element. Then use a simple loop starting from index fail_priority to the end of the list looking for errors. Like this:

# Array of fail counters
fail_counters = [
0, # PASSED
0, # LOW
0, # MEDIUM
0, # HIGH
0 # CRITICAL
]

.... run tests, accumulate error counts in fail_counters

for i in range(fail_priority, len(fail_counters)):
if fail_counters:
return 1
return 0

 

[Steven D'Aprano]

The problem is that my exit determination looks like this:

if fail_priority == fail_levels['FAILED_CRITICAL']:
if critical_fails:
return 1
if fail_priority == fail_levels['FAILED_HIGH']:
if critical_fails or high_fails:
return 1
if fail_priority == fail_levels['FAILED_MEDIUM']:
if critical_fails or high_fails or medium_fails:
return 1
if fail_priority == fail_levels['FAILED_LOW']:
if critical_fails or high_fails or medium_fails or low_fails:
return 1
return 0

Here's a general solution that doesn't require the failure levels to be
numbers, or listed in order, or to support any operation except equality
testing.

Start with an ordered list of failure severity levels, and an equivalent
ordered list of failure counts. Find the offset of the requested failure
level in the FAILURES list. Then inspect the failure counts, up to and
including that offset, ignoring any failure outside of that range:


failures = [FAILED_CRITICAL, FAILED_HIGH, FAILED_MEDIUM, FAILED_LOW]
counts = [critical_fails, high_fails, medium_fails, low_fails]
i = failures.index(fail_priority)
if any(counts[:i+1]):
print "Failed"
else:
print "No failures that we care about"


The actual values for FAILED_CRITICAL etc. can be arbitrary values:
integers *in any order*, strings, complicated records, anything so long
as they support equality and are distinct. fail_priority must be set to
one of those values.

the variables critical_fails, high_fails, medium_fails, low_fails are
all counters that are etiher None, or the number of tests that were
failed.

Why set them to None when no tests failed? If no tests failed, why not
just use 0?

That is, instead of the counters being "the number of failures, or None",
they could be "the number of failures".