Getting rid of bitwise operators in Python 3?

[Carl Banks]

Anyone with me here? (I know the deadline for P3 PEPs has passed; this
is just talk.)

Not many people are bit-fiddling these days. One of the main uses of bit
fields is flags, but that's not often done in Python because of keyword
arguments and dicts, which are lot more versatile. Another major use,
talking to hardware, is not something oft done in Python either.

It seems like this occasional usage wouldn't justify having built-in
operators on its own. And with the int/long unification, it makes a bit
less sense to think of integers as a bit field. Python has these
operators because of its heritage, but Python continues to move away from
the bad habits of its ancestors (integer division and so on), and I
wonder if this isn't another one.

Of course I'm not suggesting to get rid of bitwise operations altogether;
just make them builtin functions: "x & 1" becomes "bitwise_and(x,1)" and
so on.

Is it worth it to make such a change? It would remove a lot of operators
(11 by my count), vastly simplifying syntax, Which, IMHO, is no small
thing. New numerical types would have fewer operations to support. And
let's face it: unlike arithmetic opertaions, there's not a lot of
different meanings for bit operations. And it would also, um, make new
special characters available *cough*.

Obviously, how widespread their usage is would matter. But keep in mind
it would also be easy to convert the code automatically, because the
Python parser could reliably find all bitwise operations reliably. (The
problem would be types that overloaded them to be something other than
bitwise operations: I'm looking at you, set. That could very well be a
deal breaker. I've never approved of things with completely different
meanings being deliberately overloaded to have the same spelling; this is
one reason why.)

If anyone says, "But that takes away an easy test for oddness (x&1)!",
or, "But you can multiply powers of two using left shift! Isn't that
cool?", I'm not buying it. Those are gimmicks. Arithmetic operations
should be done with arithmetic operators. The bitwise operators make the
code much less readable, especially to people unfamiliar with this usage.


the-above-pun-was-intended-ly yr's,

Carl Banks

 

[Stargaming]

Anyone with me here? (I know the deadline for P3 PEPs has passed; this
is just talk.)

Not many people are bit-fiddling these days.

Why did we invent the `binary literals`_ (0b101) then?

One of the main uses of
bit fields is flags, but that's not often done in Python because of
keyword arguments and dicts, which are lot more versatile. Another
major use, talking to hardware, is not something oft done in Python
either.

Talking to hardware, parsing files or network streams and all this stuff
where you cannot rely on 8 bit encoding so well.

It seems like this occasional usage wouldn't justify having built-in
operators on its own. And with the int/long unification, it makes a bit
less sense to think of integers as a bit field. Python has these
operators because of its heritage, but Python continues to move away
from the bad habits of its ancestors (integer division and so on), and I
wonder if this isn't another one.

Heh, ``0b1 / 0b11`` would be 0.33333333333333331 then. Particularly
funny.

Of course I'm not suggesting to get rid of bitwise operations
altogether; just make them builtin functions: "x & 1" becomes
"bitwise_and(x,1)" and so on.

Is it worth it to make such a change? It would remove a lot of
operators (11 by my count), vastly simplifying syntax, Which, IMHO, is
no small thing. New numerical types would have fewer operations to
support. And let's face it: unlike arithmetic opertaions, there's not a
lot of different meanings for bit operations. And it would also, um,
make new special characters available *cough*.

You're right about the simplification here perhaps. But nobody is forced
to either learn or use (please, don't invert this argument justifying "we
can implement any feature then -- don't use it if you don't like it!" )
them. But it's just a little bit sugar -- like, say, decorators -- and I
like that.

Obviously, how widespread their usage is would matter. But keep in mind
it would also be easy to convert the code automatically, because the
Python parser could reliably find all bitwise operations reliably. (The
problem would be types that overloaded them to be something other than
bitwise operations: I'm looking at you, set. That could very well be a
deal breaker. I've never approved of things with completely different
meanings being deliberately overloaded to have the same spelling; this
is one reason why.)

A quick `google code search`_ failed but IMO those operators give you
some freedom to make *your* classes more syntactical-sugar'ly. ``<<`` is
used in PyParsing, as far as I know.

If anyone says, "But that takes away an easy test for oddness (x&1)!",
or, "But you can multiply powers of two using left shift! Isn't that
cool?", I'm not buying it. Those are gimmicks. Arithmetic operations
should be done with arithmetic operators. The bitwise operators make
the code much less readable, especially to people unfamiliar with this
usage.

Full acknowledgement! But in bit-wise environments, they can be
particularly useful.

the-above-pun-was-intended-ly yr's,

Carl Banks

... _binary literals: http://www.python.org/dev/peps/pep-3127/
... _google code search: http://www.google.com/codesearch?hl=en&lr=&q=lang%
3Apython+%3C%3C+%3E%3E+%7E+%26+%5C%7C+%5C%5E

 

[Grant Edwards]

Anyone with me here? (I know the deadline for P3 PEPs has passed; this
is just talk.)

Not me.

Not many people are bit-fiddling these days.

I do.

One of the main uses of bit fields is flags, but that's not
often done in Python because of keyword arguments and dicts,
which are lot more versatile. Another major use, talking to
hardware, is not something oft done in Python either.

I do it all the time.

Bit fiddling is also required for implementing communications
prototocols.

It seems like this occasional usage wouldn't justify having built-in
operators on its own. And with the int/long unification, it makes a bit
less sense to think of integers as a bit field. Python has these
operators because of its heritage, but Python continues to move away from
the bad habits of its ancestors (integer division and so on), and I
wonder if this isn't another one.

Of course I'm not suggesting to get rid of bitwise operations altogether;
just make them builtin functions: "x & 1" becomes "bitwise_and(x,1)" and
so on.

Is it worth it to make such a change? It would remove a lot of operators
(11 by my count), vastly simplifying syntax, Which, IMHO, is no small
thing.

If you don't want to use the bitwise operations, then ignore
them. Presto! Simpler syntax.

Obviously, how widespread their usage is would matter. But keep in mind
it would also be easy to convert the code automatically, because the
Python parser could reliably find all bitwise operations reliably.

The resulting code wold be fugly.

 

[Carl Banks]

Why did we invent the `binary literals`_ (0b101) then?

Hah, good point.

Ironic that base-2 literals are going into Python, but have never been a
standard part of C where they would be a LOT more useful.


Carl Banks

 

[John Machin]

Anyone with me here? (I know the deadline for P3 PEPs has passed; this
is just talk.)

Not many people are bit-fiddling these days. One of the main uses of bit
fields is flags, but that's not often done in Python because of keyword
arguments and dicts, which are lot more versatile. Another major use,
talking to hardware, is not something oft done in Python either.

You need to get out more often

Python's bitwise operators are useful for and *used* for:

(1) packing and unpacking arcane structs used in communication protocols
and in proprietary file formats

(2) trial implementations of pseudocode from academic papers, before
transliterating to pyrex or C

E.g. main loop for a bit-parallel O(N) algorithm for a Levenshtein distance:
for c in t:
pm_new = pm[ord(c)]
d0 = (((pm_new & vp) + vp) ^ vp) | pm_new | vn
hp = vn | ~(d0 | vp)
hn = d0 & vp
if hp & mask:
x += 1
if hn & mask:
x -= 1
hp = (hp << 1) | 1
vp = (hn << 1) | ~(d0 | hp)
vn = d0 & hp
return x

It seems like this occasional usage wouldn't justify having built-in
operators on its own. And with the int/long unification, it makes a bit
less sense to think of integers as a bit field.

"to think of integers as a bit field" makes no sense to me. Possibly you
meant "to think of an integer as a collection of bit fields" -- but this
is putting the cart before the horse. Simply: if the data is a
collection of bit fields, and the data is held in an int or a long or
the unification thereof, then you need bitwise operators to extract/pack
the bit fields.

Python has these
operators because of its heritage, but Python continues to move away from
the bad habits of its ancestors (integer division and so on), and I
wonder if this isn't another one.

Of course I'm not suggesting to get rid of bitwise operations altogether;
just make them builtin functions: "x & 1" becomes "bitwise_and(x,1)" and
so on.

Function call overhead?? No thanks. E.g. from real working code:

r = Rowinfo()
# Using upkbits() is far too slow on a file
# with 30 sheets each with 10K rows :-(
# upkbits(r, bits2, (
# ( 0, 0x00000007, 'outline_level'),
# ( 4, 0x00000010, 'outline_group_starts_ends'),
# ( 5, 0x00000020, 'hidden'),
# ( 6, 0x00000040, 'height_mismatch'),
# ( 7, 0x00000080, 'has_default_xf_index'),
# (16, 0x0FFF0000, 'xf_index'),
# (28, 0x10000000, 'additional_space_above'),
# (29, 0x20000000, 'additional_space_below'),
# ))
# So:
r.outline_level = bits2 & 7
r.outline_group_starts_ends = (bits2 >> 4) & 1
r.hidden = (bits2 >> 5) & 1
r.height_mismatch = (bits2 >> 6) & 1
r.has_default_xf_index = (bits2 >> 7) & 1
r.xf_index = (bits2 >> 16) & 0xfff
r.additional_space_above = (bits2 >> 28) & 1
r.additional_space_below = (bits2 >> 29) & 1

Is it worth it to make such a change? It would remove a lot of operators
(11 by my count), vastly simplifying syntax, Which, IMHO, is no small
thing. New numerical types would have fewer operations to support.

Specious argument. A new numerical type doesn't have to support bitwise
operations now, if they make no sense for that type.

And
let's face it: unlike arithmetic opertaions, there's not a lot of
different meanings for bit operations. And it would also, um, make new
special characters available *cough*.

Obviously, how widespread their usage is would matter. But keep in mind
it would also be easy to convert the code automatically, because the
Python parser could reliably find all bitwise operations reliably. (The
problem would be types that overloaded them to be something other than
bitwise operations: I'm looking at you, set. That could very well be a
deal breaker. I've never approved of things with completely different
meanings being deliberately overloaded to have the same spelling; this is
one reason why.)

Too late. That instance of Pandora's box had its lid nailed open ab initio.

If anyone says, "But that takes away an easy test for oddness (x&1)!",
or, "But you can multiply powers of two using left shift! Isn't that
cool?", I'm not buying it. Those are gimmicks. Arithmetic operations
should be done with arithmetic operators. The bitwise operators make the
code much less readable, especially to people unfamiliar with this usage.

Agreed, but this is beside the point. People will write obfuscated code
using any and every operator or built-in that you give them.

In case you hadn't guessed by now: -1

Cheers,
John

 

[Carl Banks]

Not me.


I do.

Your anecdotal evidence is noted and given all the consideration it's due.

I do it all the time.

Anecdotes again. Do you have any stats on the usage of hardware
interfacing in Python? That would be a lot more useful.

Bit fiddling is also required for implementing communications
prototocols.

Fair enough. That is a big issue, and defintely common on Python.

If you don't want to use the bitwise operations, then ignore them.
Presto! Simpler syntax.

Until people no longer need to read code written by others, this argument
is utter bull.

It's completely besides the point, anyways. The point is to make the
language core smaller. It would shift code implementing bitwise logic
out the core parts of Python, and localize it in modulespace. It's not
about trying to make the mental footprint of the language smaller; in
fact, wouldn't do that at all.

The resulting code wold be fugly.

Of course it would be. The reason I mention it is that automatic
convertibility is a key factor in whether a change can make it into
Python 3.


Carl Banks

 

[Robert Kern]

Anyone with me here? (I know the deadline for P3 PEPs has passed; this
is just talk.)

I'm not.

We use them in numpy. The bitwise operations on int and float arrays aren't all
that useful, but they are very useful for bool arrays. We can't use the
"and"/"or"/"not" keywords since they can't be overloaded, but they bitwise
operators fill the need fairly well. We've had
logical_and()/logical_or()/logical_not() functions since forever, too, and
they're a pain in the ass.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Bjoern Schliessmann]

Your anecdotal evidence is noted and given all the consideration
it's due.

Being funny, or being arrogant? Many today's network protocols work
with bit flags. It is /not/ anecdotal.

Anecdotes again. Do you have any stats on the usage of hardware
interfacing in Python? That would be a lot more useful.

You want to take features out just because they are not widely used?
Let's get rid of metaclasses and operator overloading too then.
Let's create special object types for implementing this and pack
them into seperate modules.

Until people no longer need to read code written by others, this
argument is utter bull.

What is your goal already? Holy war against usage of bitwise
operators? I can't think of any other reason.

It's completely besides the point, anyways. The point is to make
the language core smaller. It would shift code implementing
bitwise logic out the core parts of Python, and localize it in
modulespace.

And seriously, what is the point of this? Why do you want to enforce
banning bitwise operations?

Regards,


Björn

 

[Grant Edwards]

Until people no longer need to read code written by others, this argument
is utter bull.

It's completely besides the point, anyways. The point is to
make the language core smaller.

Since when has the size of Python been a concern?

It would shift code implementing bitwise logic out the core
parts of Python, and localize it in modulespace. It's not
about trying to make the mental footprint of the language
smaller; in fact, wouldn't do that at all.

I disagree. Making the mental footprint of the language
smaller is far more important. The small size of the mental
footprint is what helps readability, maintainability, and
productivity. Space in my head is more valuable than space in
my computer.

Of course it would be. The reason I mention it is that
automatic convertibility is a key factor in whether a change
can make it into Python 3.

It matters not whether fugly code is automatically generated or
manually generated. It's still hard to read and maintain.

 

[Paddy]

Sorry Carl,
I think *you* may not have much need for bitwise operators but others,
including myself do. No matter what the usage found, I would think
replacing bitwise operators by function calls a retrograde step.

- Paddy.

 

[Bryan Olson]

Not many people are bit-fiddling these days. One of the main uses of bit
fields is flags, but that's not often done in Python because of keyword
arguments and dicts, which are lot more versatile. Another major use,
talking to hardware, is not something oft done in Python either. [...]
Of course I'm not suggesting to get rid of bitwise operations altogether;
just make them builtin functions: "x & 1" becomes "bitwise_and(x,1)" and
so on.

Based on one informal survey (of my recollection), many
Pythoneers bit-twiddle, and all of them also know C. For
this population, Python's adoption of C's bitwise operators
helps avoid user-out-of-memory errors. Python even kept C's
counter-intuitive low precedence for shifts.

One surprising result was that more of the Python
programmers surveyed use bitwise operators than are aware
of the exponentiation operator, which C does not offer.

Possibly the study used a bias sample.

 

[mensanator]

Anyone with me here? (I know the deadline for P3 PEPs has passed; this
is just talk.)

Are you a loony?

Python doesn't have enough bit operations.

I'm always using the gmpy module's bit functions:

digits(...)
digits(x[,base]): returns Python string
representing x in the given base (2 to 36,
default 10 if omitted or 0); leading '-'
present if x<0, but no leading '+' if x>=0.
x must be an mpz, or else gets coerced into one.

getbit(...)
getbit(x,n): returns 0 or 1, the bit-value
of bit n of x; n must be an ordinary Python
int, >=0; x is an mpz, or else gets coerced
to one.

hamdist(...)
hamdist(x,y): returns the Hamming distance
(number of bit-positions where the bits
differ) between x and y. x and y must be
mpz, or else get coerced to mpz.

lowbits(...)
lowbits(x,n): returns the n lowest bits of
x; n must be an ordinary Python int, >0;
x must be an mpz, or else gets coerced to one.

numdigits(...)
numdigits(x[,base]): returns length of string
representing x in the given base (2 to 36,
default 10 if omitted or 0); the value returned
may sometimes be 1 more than necessary; no
provision for any 'sign' characte, nor leading
'0' or '0x' decoration, is made in the returned
length. x must be an mpz, or else gets coerced
into one.

popcount(...)
popcount(x): returns the number of 1-bits set
in x; note that this is 'infinite' if x<0,
and in that case, -1 is returned. x must be
an mpz, or else gets coerced to one.

scan0(...)
scan0(x, n=0): returns the bit-index of the
first 0-bit of x (that is at least n); n must
be an ordinary Python int, >=0. If no more
0-bits are in x at or above bit-index n
(which can only happen for x<0, notionally
extended with infinite 1-bits), None is
returned. x must be an mpz, or else gets
coerced to one.

scan1(...)
scan1(x, n=0): returns the bit-index of the
first 1-bit of x (that is at least n); n
must be an ordinary Python int, >=0. If no
more 1-bits are in x at or above bit-index
n (which can only happen for x>=0, notionally
extended with infinite 0-bits), None is returned.
x must be an mpz, or else gets coerced to one.

setbit(...)
setbit(x,n,v=1): returns a copy of the value
of x, with bit n set to value v; n must be
an ordinary Python int, >=0; v, 0 or !=0;
x must be an mpz, or else gets coerced to one.

And I don't do hardware, I do math research where
things like Hamming distance, popcount and scan1
are used extensively along with Python's bit
operators like >>.

Hey, I never use classes, maybe we should get rid of
them also, eh?

 

[richyjsm]

One surprising result was that more of the Python
programmers surveyed use bitwise operators than are aware
of the exponentiation operator, which C does not offer.

On that subject, I'd suggest that the pow() builtin (not the **
operator - just the pow() function) should also be a candidate for
removal...

Richard

 

[Paul Rubin]

If anyone says, "But that takes away an easy test for oddness (x&1)!",
or, "But you can multiply powers of two using left shift! Isn't that
cool?", I'm not buying it. Those are gimmicks. Arithmetic operations
should be done with arithmetic operators. The bitwise operators make the
code much less readable, especially to people unfamiliar with this usage.

The bitwise operators are used for set operations, which were
added fairly recently, if that hasn't been mentioned yet. They
make good sense for that.

 

[Carl Banks]

It matters not whether fugly code is automatically generated or manually
generated. It's still hard to read and maintain.

By my understanding, it really doesn't matter that much.

The transition tool's purpose is to allow a subset of 2.x code to run on
3.x, so that developers don't have to maintain separate codebases to
support both. The 3.x code that is produced isn't intended to be read or
maintained. So no big deal if it's a little ugly. (But there's still
tracebacks to read and regressions to debug, so I guess it can't be too
ugly.)


Carl Banks

 

[Carl Banks]

Sorry Carl,
I think *you* may not have much need for bitwise operators but others,
including myself do. No matter what the usage found, I would think
replacing bitwise operators by function calls a retrograde step.

Well, if people are going to take the suggestion this personally, maybe
that's reason enough not to bother.


Carl Banks

 

[Terry Reedy]

| | Is it worth it to make such a change? It would remove a lot of
operators
| (11 by my count), vastly simplifying syntax, Which, IMHO, is no small
| thing. New numerical types would have fewer operations to support.

Py3 adds an abstract base class module. Bit-integers (with the above
operations) are a separate subclass of abstract int/rats (forget the
detail), so new numerical types do not have to support those ops unless
declared as deriving from the bit-int class.

Some people consider having lots of operators available to overload in
their classes to be a good thing. Operator notation is much nicer that
function notation for binary functions.mapping pairs of objects of a class
to an object of the class.

tjr

 

[Bryan Olson]

On that subject, I'd suggest that the pow() builtin (not the **
operator - just the pow() function) should also be a candidate for
removal...

A darn good candidate, I'd say. I'm biased because I once took
quite a while to figure out that importing the math module's
'pow' was how I broke my code.

The operator module offers pow(). Is there any good reason for
pow() as a built-in?

 

[Grant Edwards]

By my understanding, it really doesn't matter that much.

The transition tool's purpose is to allow a subset of 2.x code to run on
3.x, so that developers don't have to maintain separate codebases to
support both. The 3.x code that is produced isn't intended to be read or
maintained. So no big deal if it's a little ugly.

Eh? If you remove bitwise operators, then doing bitwise
operations is going to be ugly, hard to write, and hard to read
and maintain. I don't see what 2.x vs. 3.x vs. a "transition
tool" has to do with it.

 

[Marc 'BlackJack' Rintsch]

The operator module offers pow(). Is there any good reason for
pow() as a built-in?

The `operator.pow()` is just the function for ``**``, it lacks the
optional third argument of the built in `pow()`.

Ciao,
Marc 'BlackJack' Rintsch