Dynamic features used

[bearophileHUGS]

I often use Python to write small programs, in the range of 50-500
lines of code. For example to process some bioinformatics data,
perform some data munging, to apply a randomized optimization
algorithm to solve a certain messy problem, and many different things.
For that I often use several general modules that I have written, like
implementation of certain data structures, and small general "utility"
functions/classes, plus of course several external modules that I keep
updated.

Compared to other languages Python generally allows me to write a
correctly working program in the shorter time (probably because of the
Python shell, the built-in safeties, the doctests, the clean and short
and handy syntax, the quick write-run-test cycle, the logical design,
its uniformity, the availability of standard data structures and many
standard or external modules, and the low number (compared to other
languages) of corner cases and tricky semantic corners).

Today Python is defined a dynamic language (and not a scripting
language, a term that few languages today seem to want attached to
them) but being dynamic isn't a binary thing, it's an analog quality,
a language can be less or be more dynamic. For example I think Ruby is
more dynamic than Python, Python is more dynamic than CLisp, CLips
seems more dynamic than C#/Java, Java is more dynamic than D, and D is
more dynamic than C++. Often such differences aren't sharp, and you
can find ways to things more dynamically, even with a less nice syntax
(or creating less idiomatic code). (In C#4 they have even added a
dynamic feature that may make languages like IronPython/Boo faster and
simpler to write on the dotnet).

In the last two years I have seen many answers in the Python
newsgroups, and I have seen that some of the dynamic features of
Python aren't much used/appreciated:
- Metaclasses tricks
- exec and eval
- monkey patching done on classes
- arbitrary cmp among different types removed from Python 3
While some new static-looking/related features are being introduced:
- ABCs and function signatures added
- More tidy exception tree

So it seems that while C#/D are becoming more dynamic, Python/
JavaScript are becoming a little less dynamic looking (and this I
think this is positive, because too much dynamism turns code into a
swamp, and too much rigid systems lead to bloat and other problems.
Note that there are another orthogonal solution: to use an advanced
flexible and handy static type system, as in Haskell).

I have seen that in lot of those little programs of mine, or in a
significant percentage of their lines, I often don't use the dynamic
features of Python (this means that the same code can be written with
static types, especially if you can use templates like in C++/D, or a
flexible type system like in Haskell, and it also means that lot of
those small programs can be compiled by ShedSkin/Cython, with usually
a sharp decrease of running time).

What are the dynamic features of Python more used in your programs?
(From this set try to remove the things that can be done with a
flexible static template system, like the D one, that for some things
is more handy and powerful than the C++ template system, and for other
things less powerful).

If very little or no dynamic features are used in a program it may
seem a "waste" to use Python to write the code, because the final
program may be quite slow with no gain from the other features of
Python. (I think in such situations Python can be a good choice
anyway, because it's good to write working prototypes in a short
time). (The large number of solution of this page shows how a certain
class of Python programmers want more speed from their programs:
http://scipy.org/PerformancePython and note that page misses many
other solutions, like SIP, Boost Python, ShedSkin, Cinpy, Cython,
RPython, and so on).

In the last year I have found two situations where exec/eval is a way
to reduce a lot of the complexity of the code, so if used with care
the dynamic features can be quite useful.

Before ending this partially incoherent post, I'd also like to briefly
remind how the dynamic features are used in the Boo language: Boo
programs are generally statically typed, but duck types are used once
in a while to reduce the "pressure" of the static type system. You can
find more info on this on the Boo site. (Note that I have never seen a
good set of speed benchmarks to compare the performance of CPython to
Boo).

Bye,
bearophile

 

[Rafe]

I often use Python to write small programs, in the range of 50-500
lines of code. For example to process some bioinformatics data,
perform some data munging, to apply a randomized optimization
algorithm to solve a certain messy problem, and many different things.
For that I often use several general modules that I have written, like
implementation of certain data structures, and small general "utility"
functions/classes, plus of course several external modules that I keep
updated.

Compared to other languages Python generally allows me to write a
correctly working program in the shorter time (probably because of the
Python shell, the built-in safeties, the doctests, the clean and short
and handy syntax, the quick write-run-test cycle, the logical design,
its uniformity, the availability of standard data structures and many
standard or external modules, and the low number (compared to other
languages) of corner cases and tricky semantic corners).

Today Python is defined a dynamic language (and not a scripting
language, a term that few languages today seem to want attached to
them) but being dynamic isn't a binary thing, it's an analog quality,
a language can be less or be more dynamic. For example I think Ruby is
more dynamic than Python, Python is more dynamic than CLisp, CLips
seems more dynamic than C#/Java, Java is more dynamic than D, and D is
more dynamic than C++. Often such differences aren't sharp, and you
can find ways to things more dynamically, even with a less nice syntax
(or creating less idiomatic code). (In C#4 they have even added a
dynamic feature that may make languages like IronPython/Boo faster and
simpler to write on the dotnet).

In the last two years I have seen many answers in the Python
newsgroups, and I have seen that some of the dynamic features of
Python aren't much used/appreciated:
- Metaclasses tricks
- exec and eval
- monkey patching done on classes
- arbitrary cmp among different types removed from Python 3
While some new static-looking/related features are being introduced:
- ABCs and function signatures added
- More tidy exception tree

So it seems that while C#/D are becoming more dynamic, Python/
JavaScript are becoming a little less dynamic looking (and this I
think this is positive, because too much dynamism turns code into a
swamp, and too much rigid systems lead to bloat and other problems.
Note that there are another orthogonal solution: to use an advanced
flexible and handy static type system, as in Haskell).

I have seen that in lot of those little programs of mine, or in a
significant percentage of their lines, I often don't use the dynamic
features of Python (this means that the same code can be written with
static types, especially if you can use templates like in C++/D, or a
flexible type system like in Haskell, and it also means that lot of
those small programs can be compiled by ShedSkin/Cython, with usually
a sharp decrease of running time).

What are the dynamic features of Python more used in your programs?
(From this set try to remove the things that can be done with a
flexible static template system, like the D one, that for some things
is more handy and powerful than the C++ template system, and for other
things less powerful).

If very little or no dynamic features are used in a program it may
seem a "waste" to use Python to write the code, because the final
program may be quite slow with no gain from the other features of
Python. (I think in such situations Python can be a good choice
anyway, because it's good to write working prototypes in a short
time). (The large number of solution of this page shows how a certain
class of Python programmers want more speed from their programs:http://scipy.org/PerformancePython and note that page misses many
other solutions, like SIP, Boost Python, ShedSkin, Cinpy, Cython,
RPython, and so on).

In the last year I have found two situations where exec/eval is a way
to reduce a lot of the complexity of the code, so if used with care
the dynamic features can be quite useful.

Before ending this partially incoherent post, I'd also like to briefly
remind how the dynamic features are used in the Boo language: Boo
programs are generally statically typed, but duck types are used once
in a while to reduce the "pressure" of the static type system. You can
find more info on this on the Boo site. (Note that I have never seen a
good set of speed benchmarks to compare the performance of CPython to
Boo).

Bye,
bearophile

http://scipy.org/PerformancePython is loading to slowly for my
connection. How ironic (though it probably isn't Pythons fault).

 

[Almar Klein]

<snip>

You're right (I think), but I fail to see the point you're trying
to make or the question you're asking...

I use python for scientific research too, and for me speed can be
an issue too sometimes. By using numpy and scipy I have
an environment similar to Matlab in terms of speed and
functionality (but with a language thats much easier to code in).
C# or C++ etc would be faster, yes...
But what I find even more important is that my code is interpreterd,
so I can run some code from my editor, check some results,
run another piece of code, or make changes to the code and try
again. I've tried using C# for doing my stuff, but then the
compile-> run step takes ages just too long... Plus in python you
can introspect all your varialbes using the python prompt.

Cheers,
Almar

 

[bearophileHUGS]

Almar Klein:

but I fail to see the point you're trying
to make or the question you're asking...

It's not easy to define what my point was I try again, but the
following questions don't cover all the points:
- What are the dynamic features of Python that you use in your code?
(excluding ones that can can be done with a good static template
system).
- Are them worth the decrease in running speed?
- Is it good for Python to become two languages in one, a fast
statically typed one and a dynamically one, like pypy shows to like
with RPython, or is it better to go the way of the Boo language, that
(while being mostly static) is mixing dynamic and static typing in the
same code, but that must rely on a very complex virtual machine to
work?
- Or maybe is it better to find other ways like CLips ones, that allow
to mix dynamic and static features, generally keeping programs fast
enough (Lisp-like syntax can lead to high performance too, as shown by
the Stalin Scheme compiler http://en.wikipedia.org/wiki/Stalin_(Scheme_implementation)
).

Bye,
bearophile

 

[Tim Chase]

- What are the dynamic features of Python that you use in your code?

(excluding ones that can can be done with a good static template
system).

introspection & dynamic properties are a big one. Having
functions and classes as higher-order objects that can be passed
around like any other variable is quite handy. There may other
"dynamic" features that I use but don't have mentally filed as
"dynamic" so they've been omitted here.

- Are them worth the decrease in running speed?

heck yeah. I hardly notice any decrease in speed for using
Python -- most of my bottlenecks are I/O (usually disk, network,
or database) related and those that are computationally bound,
it's often a matter of choosing a better algorithm (see several
threads in the last couple months where the OP was using an
O(N^2) algorithm that was fairly simply replaced with an O(N)
algorithm).

For those using a best-case algorithm that are still CPU-bound, I
haven't found great orders of magnitude speedup in converting
them to a compiled/non-dynamic language. I might shave a couple
seconds off runtime of a couple minutes, but the development
speed using Python more than makes up for those couple minutes
most times (most of them are one-off scripts that don't run more
than a couple times), but my O(N^2) algorithms that can't be
reduced to exact-matching-using-sets solutions still take
correspondingly long no matter the language.

- Is it good for Python to become two languages in one, a fast
statically typed one and a dynamically one, like pypy shows to like
with RPython, or is it better to go the way of the Boo language, that
(while being mostly static) is mixing dynamic and static typing in the
same code, but that must rely on a very complex virtual machine to
work?

I think Python walks this line very well, being readily
embeddable in other programs, and allowing native code-modules to
be included for inner-loop processes. While there are some times
it would be handy to have some static features such as
compile-time checking, I would only want to turn them on
optionally, not have them enforced by the language.

My $0.02 taxed for bail-out money...

-tkc

 

[pruebauno]

What are the dynamic features of Python that you use in your code?

The main ones is using configuration files that are plain Python
instead of XML and not having to wait 5 minutes to compile larger
programs. I also prefer structural typing over nominative typing and
duck typing is closer to structural than nominative typing with the
difference that compilation doesn’t take forever but the disadvantage
that the checking doesn’t happen until runtime. Hopefully they will
keep improving pylint to generate warnings for all those.

As far as speed goes, there might be 5% of my code that I would be
willing to run through a slow compiler to speed it up. For the rest it
is probably not worth it.

 

[George Sakkis]

It's not easy to define what my point was I try again, but the
following questions don't cover all the points:
- What are the dynamic features of Python that you use in your code?
(excluding ones that can can be done with a good static template
system).

Off the top of my head, getattr/setattr are the most frequent dynamic
features I use.

- Are them worth the decrease in running speed?
- Is it good for Python to become two languages in one, a fast
statically typed one and a dynamically one, like pypy shows to like
with RPython, or is it better to go the way of the Boo language, that
(while being mostly static) is mixing dynamic and static typing in the
same code, but that must rely on a very complex virtual machine to
work?
- Or maybe is it better to find other ways like CLips ones, that allow
to mix dynamic and static features, generally keeping programs fast
enough (Lisp-like syntax can lead to high performance too, as shown by
the Stalin Scheme compilerhttp://en.wikipedia.org/wiki/Stalin_(Scheme_implementation)

Very valid points, and I also often think that dynamic typing is
overrated; most programs don't need to add or remove attributes at
will or change the class hierarchy. I don't know which of the
alternatives you mention would be better but I would welcome changes
towards the "static by default" direction, provided that (1) it *is*
still possible to write dynamic code if necessary and (2) the extra
effort in writing and reading it is not off-putting (e.g. no C++
template metaprogramming atrocities)

George

 

[Aaron Brady]

On Nov 21, 3:17 am, (e-mail address removed) wrote:
snip

Compared to other languages Python generally allows me to write a
correctly working program in the shorter time (probably because of the
Python shell, the built-in safeties, the doctests, the clean and short
and handy syntax, the quick write-run-test cycle, the logical design,
its uniformity, the availability of standard data structures and many
standard or external modules, and the low number (compared to other
languages) of corner cases and tricky semantic corners).

I like the brevity (conciseness). The Python implementation of a
program is shorter on average than the C++ implementation. You don't
have to declare your types, which is just extra words and handcuffs,
looking at everything but performance. Some of the things that C++
can do are irrelevant to some programs, and Python keeps them off your
mind. C++ gives you much finer control of a machine, but you don't
always need it.

I think the central decision-making has the advantage that there is
only one Python, whereas C has many versions. It makes it easier to
share and communicate about programs in the language. Plus I don't
even need new files to cut-and-paste a snippet from the newsgroup into
the interpreter.

Today Python is defined a dynamic language (and not a scripting
language, a term that few languages today seem to want attached to
them) but being dynamic isn't a binary thing, it's an analog quality,
a language can be less or be more dynamic. For example I think Ruby is
more dynamic than Python, Python is more dynamic than CLisp, CLips
seems more dynamic than C#/Java, Java is more dynamic than D, and D is
more dynamic than C++. Often such differences aren't sharp, and you
can find ways to things more dynamically, even with a less nice syntax
(or creating less idiomatic code). (In C#4 they have even added a
dynamic feature that may make languages like IronPython/Boo faster and
simpler to write on the dotnet).

Python can be written in C. Thus, anything Python can do, C can do,
not to mention the equal expressiveness of the languages. But I have
a non-trivial observation.

With an STL 'map' and a union, you could make a quick and dirty quasi-
Python in C++. Dynamic assignment is just a __dict__ member anyway.

Functions (and classes) are first-class objects, but the 'functor'
pattern can approximate it, even including decorators. Variable-
length arguments are just an STL 'vector', and keyword arguments are
just an STL dict. You can't get the same flexibility in C++ though,
without parsing the text of a function, and storing argument info
about it. Syntax can't leave the realm of C++, but just a few classes
could be able to bring C++ a lot closer to Python.

In the last two years I have seen many answers in the Python
newsgroups, and I have seen that some of the dynamic features of
Python aren't much used/appreciated:
- Metaclasses tricks
- exec and eval
- monkey patching done on classes
- arbitrary cmp among different types removed from Python 3
While some new static-looking/related features are being introduced:
- ABCs and function signatures added
- More tidy exception tree snip
What are the dynamic features of Python more used in your programs?
(From this set try to remove the things that can be done with a
flexible static template system, like the D one, that for some things
is more handy and powerful than the C++ template system, and for other
things less powerful).

You didn't mention 'metaprogramming', which is writing a program to
generate the text code of a program, which you then compile/
interpret. I think 'exec' enables you to keep such a program in one
piece. For a rough example, a C++ program might have a dozen
repetitive classes, which you would have to write by hand or generate
externally. The Python program can just execute the definitions in
place. That is, if 'setattr' doesn't already do the trick.

The ability to return a class or a function obsoletes some of the
Gamma et al Design Patterns, such as the Factory pattern, IIRC.

And you didn't mention garbage collection.

If very little or no dynamic features are used in a program it may
seem a "waste" to use Python to write the code, because the final
program may be quite slow with no gain from the other features of
Python. (I think in such situations Python can be a good choice
anyway, because it's good to write working prototypes in a short
time).

Some of the value of that comes from the instant gratification that
Python gives one. 'Now' is a good time to test your code. 'Later' is
ok. I'm thinking of writing graphics with pygame, which lacks the
same quality in C++. Python is eloquent, whereas I'd feel I'd just
beaten a dead horse by the time I got the C++ equivalents up and
running, though perhaps less so with practice.

snip

In the last year I have found two situations where exec/eval is a way
to reduce a lot of the complexity of the code, so if used with care
the dynamic features can be quite useful.

snip

I'm not sure how much of my like of Python comes from the pioneer's
spirit, the feeling that it's new and hot and secret, undiscovered
country, like I'm out in the wilderness. But nifty tricks such as
with decorators are part of the fun. The coolest two lines of Python
are pretty cool.