python simply not scaleable enough for google?

P

Paul Rubin

Robert P. J. Day said:

I'd bet it's not just about multicore scaling and general efficiency,
but also the suitability of the language itself for large, complex
projects. It's just not possible to be everything for everybody.
Python is beginner-friendly, has a very fast learning curve for
experienced programmers in other languages, and is highly productive
for throwing small and medium sized scripts together, that are
debugged through iterated testing. One might say it's optimized for
those purposes. I use it all the time because a lot of my programming
fits the pattern. The car analogy is the no-frills electric commuter
car, just hop in and aim it where you want to go; if you crash it,
brush yourself off and restart. But there are times (large production
applications) when you really want the Airbus A380 with the 100's of
automatic monitoring systems and checkout procedures to follow before
you take off, even if the skill level needed to use it is much higher
than the commuter car.
 
V

Vincent Manis

I'd bet it's not just about multicore scaling and general efficiency,
but also the suitability of the language itself for large, complex
projects. It's just not possible to be everything for everybody.
Python is beginner-friendly, has a very fast learning curve for
experienced programmers in other languages, and is highly productive
for throwing small and medium sized scripts together, that are
debugged through iterated testing. One might say it's optimized for
those purposes. I use it all the time because a lot of my programming
fits the pattern. The car analogy is the no-frills electric commuter
car, just hop in and aim it where you want to go; if you crash it,
brush yourself off and restart. But there are times (large production
applications) when you really want the Airbus A380 with the 100's of
automatic monitoring systems and checkout procedures to follow before
you take off, even if the skill level needed to use it is much higher
than the commuter car.

OK. The quoted link deals with Unladen Swallow, which is an attempt to deal with the
very real performance limitations of current Python systems. The remarks above deal with
productivity scalability, which is a totally different matter. So...

People can and do write large programs in Python, not just `throwing...medium sized
scripts together'. Unlike, say, Javascript, it has the necessary machinery to build very
large programs that are highly maintainable. One can reasonably compare it with Java, C#,
and Smalltalk; the facilities are comparable, and all of those (as well as Python) are
used for building enterprise systems.

I believe that the A380's control software is largely written in Ada, which is a
perfectly fine programming language that I would prefer not to write code in. For
approximately 10 years, US DOD pretty much required the use of Ada in military (and
aerospace) software (though a a couple of years ago I discovered that there is still
one remaining source of Jovial compilers that still sells to DOD). According to a
presentation by Lt. Colonel J. A. Hamilton, `Programming Language Policy in the DOD:
After The Ada Mandate', given in 1999, `We are unlikely to see a return of a programming
language mandate' (www.drew-hamilton.com/stc99/stcAda_99.pdf). As I understand it,
the removal of the Ada mandate came from the realization (20 years after many computer
scientists *told* DOD this) that software engineering processes contribute more to
reliability than do programming language structures (c.f. Fred Brooks, `No Silver
Bullet').

So: to sum up, there are lots of large systems where Python might be totally appropriate,
especially if complemented with processes that feature careful specification and strong
automated testing. There are some large systems where Python would definitely NOT be
the language of choice, or even usable at all, because different engineering processes
were in place.

From a productivity viewpoint, there is no data to say that Python is more, less, or equally
scalable than <Language X> in that it produces correctly-tested, highly-maintainable programs
at a lower, higher, or equal cost. I would appreciate it if people who wanted to comment on
Python's scalability or lack thereof would give another programming language that they would
compare it with.

-- v
 
A

Aahz


Haven't seen this elsewhere in the thread:

http://dalkescientific.com/writings/diary/archive/2009/11/15/100000_tasklets.html
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it." --Brian W. Kernighan
 
R

Robin Becker

Aahz said:


I looked at this and it looks very good in that stackless appears twice as fast
as go(lang) (I used to be in the department of computing at Imperial so I
suppose I have to side with McCabe).

Anyhow, my reading of why Pike was so proud of his set up and tear down of the
tasks example was that these were real threads.

Presumably that means they could potentially run in parallel on the 100000 cpu
machines of the future.

I'm not so clear on whether the threadless tasklets will run on separate cpus.
 
S

sturlamolden

Presumably that means they could potentially run in parallel on the 100000 cpu
machines of the future.

I'm not so clear on whether the threadless tasklets will run on separate cpus.

You can make a user-space scheduler and run a 100000 tasklets on a
threadpool. But there is a GIL in stackless as well.

Nobody wants 100000 OS threads, not with Python, not with Go, not with
C.

Also note that Windows has native support for "taskelets", regardless
of language. They are called "fibers" (as opposed to "threads") and
are created using the CreateFiber system call. I would not be
surprised if Unix'es has this as well. We do not need Stackless for
light-weight threads. We can just take Python's threading modules' C
code and replace CreateThread with CreateFiber.
 
A

Aahz

Also note that Windows has native support for "taskelets", regardless
of language. They are called "fibers" (as opposed to "threads") and are
created using the CreateFiber system call. I would not be surprised if
Unix'es has this as well. We do not need Stackless for light-weight
threads. We can just take Python's threading modules' C code and
replace CreateThread with CreateFiber.

Are you advocating a high-fiber diet?
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it." --Brian W. Kernighan
 
S

sturlamolden

Are you advocating a high-fiber diet?

Only if you are a ruminant.

No really...

Windows has user-space threads natively. But you must reserve some
stack space for them (from virtual memory), which mainly makes them
useful on 64 bit systems.
 
N

Nobody

You can make a user-space scheduler and run a 100000 tasklets on a
threadpool. But there is a GIL in stackless as well.

Nobody wants 100000 OS threads, not with Python, not with Go, not with
C.

Also note that Windows has native support for "taskelets", regardless
of language. They are called "fibers" (as opposed to "threads") and
are created using the CreateFiber system call. I would not be
surprised if Unix'es has this as well. We do not need Stackless for
light-weight threads. We can just take Python's threading modules' C
code and replace CreateThread with CreateFiber.

POSIX.1-2001 and POSIX.1-2004 have makecontext(), setcontext(),
getcontext() and swapcontext(), but obsoleted by POSIX.1-2008.

They are available on Linux; I don't know about other Unices.
 
R

Robin Becker

sturlamolden said:
You can make a user-space scheduler and run a 100000 tasklets on a
threadpool. But there is a GIL in stackless as well.

Nobody wants 100000 OS threads, not with Python, not with Go, not with
C.

Also note that Windows has native support for "taskelets", regardless
of language. They are called "fibers" (as opposed to "threads") and
are created using the CreateFiber system call. I would not be
surprised if Unix'es has this as well. We do not need Stackless for
light-weight threads. We can just take Python's threading modules' C
code and replace CreateThread with CreateFiber.
........

not really sure about all the parallelism that will actually be achievable, but
apparently the goroutines are multiplexed onto native threads by the run time.
Apparently each real thread is run until it blocks and then another goroutine is
allowed to make use of the thread. Apparently the gccgo runtime has 1 goroutine
per thread and is different to the fast compilers.
 
R

Robin Becker

sturlamolden said:
You can make a user-space scheduler and run a 100000 tasklets on a
threadpool. But there is a GIL in stackless as well.

Nobody wants 100000 OS threads, not with Python, not with Go, not with
C.

Also note that Windows has native support for "taskelets", regardless
of language. They are called "fibers" (as opposed to "threads") and
are created using the CreateFiber system call. I would not be
surprised if Unix'es has this as well. We do not need Stackless for
light-weight threads. We can just take Python's threading modules' C
code and replace CreateThread with CreateFiber.
........

not really sure about all the parallelism that will actually be achievable, but
apparently the goroutines are multiplexed onto native threads by the run time.
Apparently each real thread is run until it blocks and then another goroutine is
allowed to make use of the thread. Apparently the gccgo runtime has 1 goroutine
per thread and is different to the fast compilers.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,990
Messages
2,570,211
Members
46,796
Latest member
SteveBreed

Latest Threads

Top