PEP: Adding Priority Scheduling feature to the subprocess

[darklord]

I am working on a PEP and would appreciate comment. The proposal is
available at

http://python.timehorse.com/PEP_-_Application_Priority.reST

and is repeated below:

---------

EP: <Unassigned>
:Title: Adding Priority Scheduling feature to the subprocess module
:Version: $Rev: 93 $
:Last Modified: $Date: 2008-02-20 08:37:00 -0500 (Wed, 20 Feb 2008) $
:Author: Jeffrey C. Jacobs
iscussions-To: python at timehorse dot com
:Status: Draft
:Type: Standards Track
:Content-Type: text/x-rst
:Created: 19-Feb-2008
ython-Version: 2.5.1
ost-History: <None>

----------

Introduction
~~~~~~~~~~~~

The ``subprocess`` module is intended to create a platform-independent
method of *spawning* or *popening* or *execing* an OS-level sub-
process. This module provides the ability to spawn processes on a
variety of platforms by creating an instance of the
``subprocess.Popen`` class.

Currently, only the Windows version of the ``Popen.__init__`` method
can set the OS priority of a spawned process directly using the
``createFlags`` parameter. Clients operating under UNIX must use some
version of the ``preexec_fn`` to either wrap a C-call to the
``setpriority`` POSIX function or do so by prepending ``nice <u>`` to
the ``args`` or in the ``executable``, which would only work for sub-
processes spawned from a shell. Under a Windows environment, however,
the ``preexec_fn`` parameter is ignored and there is no ``nice`` shell
program. This thus precludes the easy setting of a spawned process's
priority in the UNIX environment, as well as creates a disconnect in
terms of the way priorities are set between Windows and UNIX
platforms, which somewhat defeats the purpose of ``subprocess.py`` in
terms of being relatively platform-independent.

In addition to these issues, because Windows processes are referenced
by handles, not by process IDs, although this would divide the UNIX-
specific and Windows specific code, the Handle for the windows would
none the less be useful in the ``win32process`` module and therefore
it is proposed that this handle be officially exposed from the
``Popen`` instance via the name ``handle``. Currently, this property
is unofficially accessible via the name ``_handle``, so this proposes
to rename that value and make the name official for python running
under Windows. The value of ``handle`` for other platforms would then
be ``None``. In addition to officially exposing the Handle to
Process, it is proposed that the Main Thread Handle, also internally
exposed within the ``Popen.__init__`` process, be externally exposed
via the name ``hthread`` for use by ``win32process`` and the priority
methods.

Because UNIX uses priorities between +20 and -20 and Windows, via
Process and Thread priorities, allows settings between 0 and 31, a
uniform setting for each system should be derived. This would be
accomplished by giving process priority in terms of a floating-point
value between 0.0 and 1.0 for lowest and highest possible priority,
respectively. This priority would be added to the ``Popen``
constructor as an optional parameter, with a default value of
``None``. If and only if something other than ``None`` is set, that
value would override any setting in ``createFlags``, and the Process
Priority flags of ``createFlags`` would be masked off under the
Windows environment. In all cases a non-``None`` setting would cause
the spawned process to run at the indicated priority converted to the
platform-specific units.

The Macintosh OS X / BSD subsystem should be compatible with the UNIX-
specific sections of this proposal.

Specifics
~~~~~~~~~

The following steps would accomplish this proposal:

1) Add a C-Python wrapper for the POSIX ``getpriority`` /
``setpriority`` methods to the ``os`` module. These methods would
only be available in a UNIX environment and would mesh well with the
existing file/process functions specified therein. The
``win32process`` module already exposes the corresponding Windows
methods.

2) Officially expose the Process Handle as ``handle`` and the
Process's Main Thread Handle as ``hThread`` within the ``Popen``
instance interface.

3) Add ``setPriority`` / ``getPriority`` methods to the ``Popen``
class which will use the platform-specific priority function either
from ``os`` for UNIX or ``win32process`` for Windows. The input
priority to ``setPriority`` will be checked and raise a ``ValueError``
exception for values not in the range 0.0 to 1.0.

The 0.0 to 1.0 scale will be converted to the 32-point Windows
scale by linear mapping, e.g. for a floating-point priority *p*, the
Windows priority is given by:

``int(p*31 + .5)``

Windows is limited to a non-linear, 7-point scale for process
priorities, with thread priorities making up the difference relative
to the 32-point internal priority scale. Thus, the Windows versions
of these functions will use a combination of ``SetPriorityClass`` and
``SetThreadPriority`` in order to set the requested process priority,
using the ``handle`` and ``hThread`` properties of the ``Popen``
instance.

The 0.0 to 1.0 scale will be converted to the 41-point UNIX scale
(values between +20 and -20, inclusive) via an inverted linear
mapping, e.g. for a floating-point priority *p*, the UNIX priority is
given by:

``20 - int(p*40 + .5)``

Under the UNIX platform, only the super-user (root) is allowed to
set priorities below 0 (*p* > 0.5). This condition will not be
checked in the ``setPriority`` code; it is the responsibility of the
caller to ensure the script making the call to ``setpriority`` has
sufficient user privileges, i.e. via the ``os.setuid`` function.

The conversion from the platform-specific Priority scale to the
floating-point scale are used by ``getPriority`` and are as follows
for a given UNIX priority *u* and Windows priority *w*:

``(20 - u) / 40.0``

``w / 31.0``

As with the ``setPriority`` method for windows, the ``getPriority``
method shall use a combination of ``GetPriorityClass`` /
``GetThreadPriority`` to determine the true Windows priority based on
the 32-point scale before converting to floating point. The specific
numeric conversion for the Windows ``getPriority`` method is given in
`Table 1`_.

See the `Appendix`_ for details about the 32-point Windows scale.
Note that some priorities have equivalent settings. In this case, raw
Process Priority Class (without regard for Foreground and Background
Process settings) will have preferential setting and Thread Priority
Levels shall be used to make up the difference. E.g. priorities from
1 to 4 will have ``IDLE_PRIORITY_CLASS``, 5 to 7 will have
``BELOW_NORMAL_PRIORITY_CLASS``, 8 and 9 will have
``NORMAL_PRIORITY_CLASS``, 10 to 12 will have
``ABOVE_NORMAL_PRIORITY_CLASS``, 13 to 15 will have
``HIGH_PRIORITY_CLASS`` and 16 to 31 will be accessed through
``REALTIME_PRIORITY_CLASS``, potentially with a non-standard Thread
Priority level. Also note that Priority 0 is reserved for the Zero-
Page Thread, which is responsible for zeroizing memory when no other
process is running. It is impossible for a user process to have this
priority. Therefore, on Windows, any priority set request below about
0.016 will be treated as if it was requesting Windows Priority Level
1.

4) Add a priority parameter to the ``Popen.__init__`` method that
shall be the final parameter of the function and default to ``None``
in order to preserve backwards compatibility. When priority is not
``None``, it shall be used in order to set the priority of the spawned
process either at the time of the creation, or soon after. The
setting of priority may be accomplished via a ``nice <u>`` prefix to
shell commands (``shell = True``) or via a call to ``setpriority``
after the process is spawned. When a priority is set via the Windows
version, a mask is applied to the ``createFlags`` parameter to remove
any priority related bits. The calculated priority is then used to
determine a new priority in the 7-point Windows Process scale and set
the corresponding flag in the ``createFlags`` parameter. Then
``SetThreadPriority`` shall be called on the returned Main Thread
Handle with the remaining priority points to further adjust the
priority toward the 32-point scale.

Opened Questions
~~~~~~~~~~~~~~~~

1) Should ``Popen.__init__`` for UNIX check for shell-based spawns and
automatically apply ``nice`` or should it always first spawn the
process and then set its priority via ``setpriority`` once created.
Note, the later technique would technically not be available using
``preexec_fn``.

2) The ``startinfo`` parameter may also need adjusting to set whether
the process is spawned as a background or foreground process (See
`Table 1`_).

3) Should windows use the Priority Boost APIs:
``SetProcessPriorityBoost``, ``GetProcessPriorityBoost``,
``SetThreadPriorityBoost`` and ``GetThreatPriorityBoost``?

4) How does one determine if a given Windows process with handle
``handle`` is running as a *Background* or a *Foreground* process at
any given time, for use in the Windows version of ``getPriority``?

5) With the current version of ``subprocess`` I have been unable to
spawn a process with a higher priority than ``NORMAL_PRIORITY_CLASS``
under Windows; python is always running as ``NORMAL_PRIORITY_CLASS``.

Appendix
~~~~~~~~

... _Table 1:

+----+----------------------------------
+-------------------------------+
| Pt | Process Priority Class | Thread Priority
Level |
+====+==================================
+===============================+
| 1 | IDLE_PRIORITY_CLASS |
THREAD_PRIORITY_IDLE |
+----+----------------------------------
+-------------------------------+
| 1 | BELOW_NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_IDLE |
+----+----------------------------------
+-------------------------------+
| 1 | NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_IDLE |
+----+----------------------------------
+-------------------------------+
| 1 | ABOVE_NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_IDLE |
+----+----------------------------------
+-------------------------------+
| 1 | HIGH_PRIORITY_CLASS |
THREAD_PRIORITY_IDLE |
+----+----------------------------------
+-------------------------------+
| 2 | IDLE_PRIORITY_CLASS |
THREAD_PRIORITY_LOWEST |
+----+----------------------------------
+-------------------------------+
| 3 | IDLE_PRIORITY_CLASS |
THREAD_PRIORITY_BELOW_NORMAL |
+----+----------------------------------
+-------------------------------+
| 4 | IDLE_PRIORITY_CLASS |
THREAD_PRIORITY_NORMAL |
+----+----------------------------------
+-------------------------------+
| 4 | BELOW_NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_LOWEST |
+----+----------------------------------
+-------------------------------+
| 5 | IDLE_PRIORITY_CLASS |
THREAD_PRIORITY_ABOVE_NORMAL |
+----+----------------------------------
+-------------------------------+
| 5 | BELOW_NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_BELOW_NORMAL |
+----+----------------------------------
+-------------------------------+
| 5 | Background NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_LOWEST |
+----+----------------------------------
+-------------------------------+
| 6 | IDLE_PRIORITY_CLASS |
THREAD_PRIORITY_HIGHEST |
+----+----------------------------------
+-------------------------------+
| 6 | BELOW_NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_NORMAL |
+----+----------------------------------
+-------------------------------+
| 6 | Background NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_BELOW_NORMAL |
+----+----------------------------------
+-------------------------------+
| 7 | BELOW_NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_ABOVE_NORMAL |
+----+----------------------------------
+-------------------------------+
| 7 | Background NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_NORMAL |
+----+----------------------------------
+-------------------------------+
| 7 | Foreground NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_LOWEST |
+----+----------------------------------
+-------------------------------+
| 8 | BELOW_NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_HIGHEST |
+----+----------------------------------
+-------------------------------+
| 8 | NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_ABOVE_NORMAL |
+----+----------------------------------
+-------------------------------+
| 8 | Foreground NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_BELOW_NORMAL |
+----+----------------------------------
+-------------------------------+
| 8 | ABOVE_NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_LOWEST |
+----+----------------------------------
+-------------------------------+
| 9 | NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_HIGHEST |
+----+----------------------------------
+-------------------------------+
| 9 | Foreground NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_NORMAL |
+----+----------------------------------
+-------------------------------+
| 9 | ABOVE_NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_BELOW_NORMAL |
+----+----------------------------------
+-------------------------------+
| 10 | Foreground NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_ABOVE_NORMAL |
+----+----------------------------------
+-------------------------------+
| 10 | ABOVE_NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_NORMAL |
+----+----------------------------------
+-------------------------------+
| 11 | Foreground NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_HIGHEST |
+----+----------------------------------
+-------------------------------+
| 11 | ABOVE_NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_ABOVE_NORMAL |
+----+----------------------------------
+-------------------------------+
| 11 | HIGH_PRIORITY_CLASS |
THREAD_PRIORITY_LOWEST |
+----+----------------------------------
+-------------------------------+
| 12 | ABOVE_NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_HIGHEST |
+----+----------------------------------
+-------------------------------+
| 12 | HIGH_PRIORITY_CLASS |
THREAD_PRIORITY_BELOW_NORMAL |
+----+----------------------------------
+-------------------------------+
| 13 | HIGH_PRIORITY_CLASS |
THREAD_PRIORITY_NORMAL |
+----+----------------------------------
+-------------------------------+
| 14 | HIGH_PRIORITY_CLASS |
THREAD_PRIORITY_ABOVE_NORMAL |
+----+----------------------------------
+-------------------------------+
| 15 | HIGH_PRIORITY_CLASS |
THREAD_PRIORITY_HIGHEST |
+----+----------------------------------
+-------------------------------+
| 15 | HIGH_PRIORITY_CLASS |
THREAD_PRIORITY_TIME_CRITICAL |
+----+----------------------------------
+-------------------------------+
| 15 | IDLE_PRIORITY_CLASS |
THREAD_PRIORITY_TIME_CRITICAL |
+----+----------------------------------
+-------------------------------+
| 15 | BELOW_NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_TIME_CRITICAL |
+----+----------------------------------
+-------------------------------+
| 15 | NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_TIME_CRITICAL |
+----+----------------------------------
+-------------------------------+
| 15 | ABOVE_NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_TIME_CRITICAL |
+----+----------------------------------
+-------------------------------+
| 16 | REALTIME_PRIORITY_CLASS | THREAD_PRIORITY_IDLE |
+----+----------------------------------
+-------------------------------+
| 17 | REALTIME_PRIORITY_CLASS | -7 |
+----+----------------------------------
+-------------------------------+
| 18 | REALTIME_PRIORITY_CLASS | -6 |
+----+----------------------------------
+-------------------------------+
| 19 | REALTIME_PRIORITY_CLASS | -5 |
+----+----------------------------------
+-------------------------------+
| 20 | REALTIME_PRIORITY_CLASS | -4 |
+----+----------------------------------
+-------------------------------+
| 21 | REALTIME_PRIORITY_CLASS | -3 |
+----+----------------------------------
+-------------------------------+
| 22 | REALTIME_PRIORITY_CLASS | THREAD_PRIORITY_LOWEST |
+----+----------------------------------
+-------------------------------+
| 23 | REALTIME_PRIORITY_CLASS |
THREAD_PRIORITY_BELOW_NORMAL |
+----+----------------------------------
+-------------------------------+
| 24 | REALTIME_PRIORITY_CLASS | THREAD_PRIORITY_NORMAL |
+----+----------------------------------
+-------------------------------+
| 25 | REALTIME_PRIORITY_CLASS |
THREAD_PRIORITY_ABOVE_NORMAL |
+----+----------------------------------
+-------------------------------+
| 26 | REALTIME_PRIORITY_CLASS | THREAD_PRIORITY_HIGHEST |
+----+----------------------------------
+-------------------------------+
| 27 | REALTIME_PRIORITY_CLASS | 3 |
+----+----------------------------------
+-------------------------------+
| 28 | REALTIME_PRIORITY_CLASS | 4 |
+----+----------------------------------
+-------------------------------+
| 29 | REALTIME_PRIORITY_CLASS | 5 |
+----+----------------------------------
+-------------------------------+
| 30 | REALTIME_PRIORITY_CLASS | 6 |
+----+----------------------------------
+-------------------------------+
| 31 | REALTIME_PRIORITY_CLASS |
THREAD_PRIORITY_TIME_CRITICAL |
+----+----------------------------------
+-------------------------------+

Table 1: 32-Point Windows Priority Scale [#MSPriorityTable]_.

 

[Terry Reedy]

| Because UNIX uses priorities between +20 and -20 and Windows, via
| Process and Thread priorities, allows settings between 0 and 31, a
| uniform setting for each system should be derived. This would be
| accomplished by giving process priority in terms of a floating-point
| value between 0.0 and 1.0 for lowest and highest possible priority,
| respectively.

I would rather that the feature use the -20 to 20 priorities and map that
appropriately to the Windows range on Windows.

 

[TimeHorse]

| Because UNIX uses priorities between +20 and -20 and Windows, via
| Process and Thread priorities, allows settings between 0 and 31, a
| uniform setting for each system should be derived.  This would be
| accomplished by giving process priority in terms of a floating-point
| value between 0.0 and 1.0 for lowest and highest possible priority,
| respectively.

I would rather that the feature use the -20 to 20 priorities and map that
appropriately to the Windows range on Windows.

The problem as I see it is that -20 to +20 is only just over 5 bits of
precision and I can easily imagine an OS with many more than just 5
bits to specify a process priority. Of course, the os.getpriority and
os.setpriority, being specific to UNIX, WOULD use the -20 to +20
scale, it's just the generic subprocess that would not. But for a
generic priority, I like floating point because it gives 52 bits of
precision on most platforms. This would allow for the most
flexibility. Also, 0.0 to 1.0 is in some ways more intuitive to new
programmers because it can be modeled as ~0% CPU usage vs. ~100% CPU
usage, theoretically. Users not familiar with UNIX might OTHO be
confused by the idea that a lower priority number constitutes a
"higher priority".

Of course, the scale used for p in Popen(...).setPriority(p) is really
not an important issue to me as long as it makes sense in the context
of priorities. Given that os.setpriority and Popen(...).setPriority
have virtually the same name, it would probably be better to rename
the later to something a bit less prone to confusion. Alternatively,
it would not be unreasonable to design setPriority (and getPriority
correspondingly) such that under UNIX it takes 1 parameter, -20 to +20
and under Windows it takes 2 parameters, second one optional, where
the Windows API priorities are directly passed to it (for getPriority,
Windows would return a Tuple pair corresponding to Priority Class and
Main Thread Priority). However, I personally prefer a unified
definition for subprocess.py's Priority since there already is or will
be direct os-level methods to accomplish the same thing in the os-
native scale.

Anyway, thanks for the input and I will make a note of it in the PEP.
Other than the generic Property ranges, do you see any other issues
with my proposal?

Jeffrey.

 

[Dennis Lee Bieber]

The problem as I see it is that -20 to +20 is only just over 5 bits of
precision and I can easily imagine an OS with many more than just 5
bits to specify a process priority. Of course, the os.getpriority and

Why imagine... AmigaOS ran -128..+127 (though in practice, one never
went above +20 as the most time critical system processes ran at that
level; User programs ran at 0, the Workbench [desktop] ran at +1... I
think file system ran +5 and disk device handlers ran +10)

On the other side, VMS ran 0..31, with 16..31 being fixed realtime
priorities, and 0..15 being variable (they don't drop below the process
base, but can and are boosted higher the longer they are preempted)
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[TimeHorse]

        Why imagine... AmigaOS ran -128..+127 (though in practice, one never
went above +20 as the most time critical system processes ran at that
level; User programs ran at 0, the Workbench [desktop] ran at +1... I
think file system ran +5 and disk device handlers ran +10)

        On the other side, VMS ran 0..31, with 16..31 being fixed realtime
priorities, and 0..15 being variable (they don't drop below the process
base, but can and are boosted higher the longer they are preempted)

Thanks for the info! Actually, many peices of the WinNT kernel were
based on VMS, interestingly enough (go back to the NT 3.5 days and you
can see this) and of course Windows 2000, XP and Vista are all derived
from NT. So I guess one long-time holdover from the VMS days of NT is
just as you say, it's still true in Windows: processes run 0..15 for
anything but realtime, even if the thread is boosted to Time Critical,
but in realtime the priorities go from 16..31.

Anyway, on the one hand AmigaOS support where -128 -> p = 0.0 and +127
-> p = 1.0 would be a good example of why simply using a 41 point UNIX
scale is defecient in representing all possible priorities, but apart
from the support AmigaOS argument, you bring up another issue which
may be dangerous: are priorities linear in nature? For instance, a
with-Focus Windows App running Normal-Normal runs at priority 9, p ~=
0.29 (as well as likely for VMS), where as UNIX and Amiga have 0 for
normal processes, p ~= 0.50. In many ways, the "Normal" mode being
the epoch makes sense, but this clearly cannot be done on a linear
scale. Perhaps I should modify the PEP to instead having the generic
priorities from 0.0 to 1.0, to have them go from -1.0 to +1.0, with
0.0 considered normal priority. Then, the negative and positive
regions can be considered linearly but not necessairily with the same
spacing since on Windows -1.0 to 0.0 spans 0 to 9 priorities and 0.0
to +1.0 spans 10 to 31 priorities. And then, since +20 is the highest
AmigaOS priority in practice, yet the scale goes up to +127, the would
mean that from p ~= +0.16 to p ~= -1.0 you get obscenely high
priorities which do not seem practical.

I will need to think about this problem so more. I'd hate to think
there might be a priority scale based on a Normal Distribution! I'm
already going bald; I can't afford to loose any more hair!

Anyway, thanks for info Dennis; you've given me quite a bit to think
about!

Jeffrey.

 

[Nick Craig-Wood]

Anyway, on the one hand AmigaOS support where -128 -> p = 0.0 and +127
-> p = 1.0 would be a good example of why simply using a 41 point UNIX
scale is defecient in representing all possible priorities, but apart
from the support AmigaOS argument, you bring up another issue which
may be dangerous: are priorities linear in nature?

Interestingly enough this was changed in recent linux kernels.
Process levels in linus kernels are logarithmic now, whereas before
they weren't (but I wouldn't like to say exactly what!).

If I run two CPU intensive jobs to busy out both my CPUs, then using
kernel 2.6.22, run

$ python -c 'from time import time
t = time()
for i in xrange(10000000):
pass
print time()-t'
1.36508607864

$ nice -n 10 python -c 'from time import time
t = time()
for i in xrange(10000000):
pass
print time()-t'
4.27783703804

$ nice -n 20 python -c 'from time import time
t = time()
for i in xrange(10000000):
pass
print time()-t'
36.9293899536

You can see that the levels are not linear! A nice 10 job gets 32% of
the CPU of a nice 0 job, but a nice 20 job gets only 4%.

I think you are on to a loser here trying to normalise it across
OSes unfortunately :-(

 

[TimeHorse]

Interestingly enough this was changed in recent linux kernels.
Process levels in linus kernels are logarithmic now, whereas before
they weren't (but I wouldn't like to say exactly what!).

Wow! That's a VERY good point. I ran a similar test on Windows with
the 'start' command which is similar to nice but you need to specify
the Priority Class by name, e.g.

start /REALTIME python.exe bench1.py

Now, in a standard operating system you'd expect some variance between
runs, and I did find that. So I wrote a script to compute the Mode
(but not the Standard Deviation as I didn't have time for it) for each
Priority Class, chosen each run at random, accumulated the running
value for each one. Now, when I read the results, I really wish I'd
computed the Chi**2 to calculate the Standard Deviation because the
results all appeared within very close relation to one another, as if
the Priority Class had overall very little effect. In fact, I would
be willing to guess that say NORMAL and ABOVENORMAL lie with one
Standard Deviation of one another!

That having been said, the tests all ran in about 10 seconds so it may
be that the process was too simple to show any statistical results. I
know for instance that running ffmpeg as NORMAL or REALTIME makes a
sizable difference.

So, I concede the "Unified Priority" may indeed be dead in the water,
but I am thinking of giving it once last go with the following
suggestion:

0.0 == Zero-Page (Windows, e.g. 0) / +20 (Unix)
1.0 == Normal (Foreground) Priority (Windows, e.g. 9) / 0 (Unix)
MAX_PRIORITY == Realtime / Time Critical (Windows, e.g. 31) / -20
(Unix)

With the value of MAX_PRIORITY TBD. Now, 0.0 would still represent
(relatively) 0% CPU usage, but now 1.0 would represent 100% of
'Normal' priority. I would still map 0.0 - 1.0 linearly over the
scale corresponding to the given operating system (0 - 9, Window; +20
- 0, Unix), but higher priorities would correspond to > 1.0 values.

The idea here is that most user will only want to lower priority, not
raise it, so it makes lowering pretty intuitive. As for the linear
mapping, I would leave a note in the documentation that although the
scale is "linear", the operating system may choose to behave as if the
scale is linear and that the user should consult the documentation for
their OS to determine specific behavior. This is similar to the
documentation of the file time-stamps in os.stat, since their
granularity differs based on OS. Most users, I should think, would
just want to make their spawn "slower" and use the scale do determine
"how much" in a relative fashion rather than expecting hard-and-fast
numbers for the actually process retardation.

Higher than Normal priorities may OTHO, be a bit harder to deal with.
It strikes me that maybe the best approach is to make MAX_PRIORITY
operating system dependent, specifically 31 - 9 + 1.0 = +23.0 for
Windows and -20 - 0 + 1.0 = +21.0 for Unix. This way, again the
priorities map linearly and in this case 1:1. I think for most users,
they would choose a "High Priority" relative to MAX_PRIORITY or just
choose a small increment about 1.0 to add just a little boost.

Of course, the 2 biggest problems with this approach are, IMHO, a) the
< Normal scale is percent but the > Normal scale is additive.
However, there is no "Simple" definition of MAX_PRIORITY, so I think
using the OS's definition is natural. b) This use of the priority
scale may be confusing to Unix users, since 1.0 now represents
"Normal" and +21, not +/-20 represents Max Priority. However, the
definition of MAX_PRIORITY would be irrelevant to the definition of
setPriority and getPriority, since each would, in my proposal, compute
for p > 1.0:

Windows: 9 + int((p - 1) / (MAX_PRIORITY - 1) * 22 + .5)
Unix: -int((p - 1) / (MAX_PRIORITY - 1) * 20 + .5)

Anyway, that's how I'd propose to do the nitty-gritty. But, more than
anything, I think the subprocess 'priority' methods should use a
priority scheme that is easy to explain. And by that, I propose:

1.0 represents normal priority, 100%. Any priority less than 1
represents a below normal priority, down to 0.0, the lowest possible
priority or 0%. Any priority above 1.0 represents an above normal
priority, with MAX_PRIORITY being the highest priority level available
for a given os.

Granted, it's not much simpler than 0 is normal, -20 is highest and
+20 is lowest, except in so far as it being non-intuitive to consider
a lower priority number representing a higher priority. Certainly, we
could conform all systems to the +20.0 to -20.0 floating point system,
but I prefer not to bias the methods and honestly feel percentage is
more intuitive.

So, how does that sound to people? Is that more palatable?

Thanks again for all the input!

Jeffrey.

 

[Nick Craig-Wood]

Wow! That's a VERY good point. I ran a similar test on Windows with
the 'start' command which is similar to nice but you need to specify
the Priority Class by name, e.g.

start /REALTIME python.exe bench1.py

Now, in a standard operating system you'd expect some variance between
runs, and I did find that. So I wrote a script to compute the Mode
(but not the Standard Deviation as I didn't have time for it) for each
Priority Class, chosen each run at random, accumulated the running
value for each one. Now, when I read the results, I really wish I'd
computed the Chi**2 to calculate the Standard Deviation because the
results all appeared within very close relation to one another, as if
the Priority Class had overall very little effect. In fact, I would
be willing to guess that say NORMAL and ABOVENORMAL lie with one
Standard Deviation of one another!

That having been said, the tests all ran in about 10 seconds so it may
be that the process was too simple to show any statistical results. I
know for instance that running ffmpeg as NORMAL or REALTIME makes a
sizable difference.

You need to run N x tasks at normal priority which just use up CPU, eg

python -c "while 1: pass"

N needs to be the number of CPUs that you have.

If you don't do that then your time test prog will just run at the
full CPU speed and not test the scheduling at all!

So, I concede the "Unified Priority" may indeed be dead in the water,
but I am thinking of giving it once last go with the following
suggestion:

0.0 == Zero-Page (Windows, e.g. 0) / +20 (Unix)
1.0 == Normal (Foreground) Priority (Windows, e.g. 9) / 0 (Unix)
MAX_PRIORITY == Realtime / Time Critical (Windows, e.g. 31) / -20
(Unix)

With the value of MAX_PRIORITY TBD. Now, 0.0 would still represent
(relatively) 0% CPU usage, but now 1.0 would represent 100% of
'Normal' priority. I would still map 0.0 - 1.0 linearly over the
scale corresponding to the given operating system (0 - 9, Window; +20
- 0, Unix), but higher priorities would correspond to > 1.0 values.

The idea here is that most user will only want to lower priority, not
raise it, so it makes lowering pretty intuitive. As for the linear
mapping, I would leave a note in the documentation that although the
scale is "linear", the operating system may choose to behave as if the
scale is linear and that the user should consult the documentation for
their OS to determine specific behavior. This is similar to the
documentation of the file time-stamps in os.stat, since their
granularity differs based on OS. Most users, I should think, would
just want to make their spawn "slower" and use the scale do determine
"how much" in a relative fashion rather than expecting hard-and-fast
numbers for the actually process retardation.

Higher than Normal priorities may OTHO, be a bit harder to deal with.
It strikes me that maybe the best approach is to make MAX_PRIORITY
operating system dependent, specifically 31 - 9 + 1.0 = +23.0 for
Windows and -20 - 0 + 1.0 = +21.0 for Unix. This way, again the
priorities map linearly and in this case 1:1. I think for most users,
they would choose a "High Priority" relative to MAX_PRIORITY or just
choose a small increment about 1.0 to add just a little boost.

Of course, the 2 biggest problems with this approach are, IMHO, a) the
< Normal scale is percent but the > Normal scale is additive.
However, there is no "Simple" definition of MAX_PRIORITY, so I think
using the OS's definition is natural. b) This use of the priority
scale may be confusing to Unix users, since 1.0 now represents
"Normal" and +21, not +/-20 represents Max Priority. However, the
definition of MAX_PRIORITY would be irrelevant to the definition of
setPriority and getPriority, since each would, in my proposal, compute
for p > 1.0:

Windows: 9 + int((p - 1) / (MAX_PRIORITY - 1) * 22 + .5)
Unix: -int((p - 1) / (MAX_PRIORITY - 1) * 20 + .5)

Anyway, that's how I'd propose to do the nitty-gritty. But, more than
anything, I think the subprocess 'priority' methods should use a
priority scheme that is easy to explain. And by that, I propose:

1.0 represents normal priority, 100%. Any priority less than 1
represents a below normal priority, down to 0.0, the lowest possible
priority or 0%. Any priority above 1.0 represents an above normal
priority, with MAX_PRIORITY being the highest priority level available
for a given os.

Granted, it's not much simpler than 0 is normal, -20 is highest and
+20 is lowest, except in so far as it being non-intuitive to consider
a lower priority number representing a higher priority. Certainly, we
could conform all systems to the +20.0 to -20.0 floating point system,
but I prefer not to bias the methods and honestly feel percentage is
more intuitive.

So, how does that sound to people? Is that more palatable?

I think it might give people the idea that if you set it to 0.1 you'll
get 1/10 of the cpu than normal. Likewise 2.0 etc. There is
something to be said for the deliberately somewhat arbitrary nice
levels in the unix world!