Python Operating System???

[Paul Rubin]

An OS is NOT an application. It is a completely different kind of program.
Do you guys understand the difference between user and kernel mode? Do you
know what address spaces and hardware interrupts are? Python is not
equipped to handle these things. You would end up doing so much in C
extensions that it could barely be called Python.

You'd need some library functions to let Python access device registers
and you'd need some low level code to dispatch interrupts into Python code.

I am not trying to be insulting... but unless someone would like to educate
me otherwise, the idea of an OS written in Python is almost ludicrous.

Is an OS written in Lisp also ludicrous? Because it's been done.

When Unix was first written, people thought implementing an OS in C
was ludicrous. Everyone knew OS's had to be written in assembler.

Also I am no threading expert, but I would imagine it would be very hard to
write a task scheduler in Python given that it has the whole GIL thing. (As
I said that isn't my domain of expertise but I'm sure someone here could
expound on that.)

The GIL is not part of the Python language. It's just an artifact of
one particular implementation.

 

[David Brown]

So how would I make an OS Shell?

 

[Peter Hansen]

When Unix was first written, people thought implementing an OS in C
was ludicrous. Everyone knew OS's had to be written in assembler.

Actually, when Unix was first written that belief was entirely
correct, and OSes *did* have to be written in assembler.

That is, pure C did not have the capability to handle all
that was required. I recall it taking a good decade before it
became *common* to find C compilers with, for example, @interrupt
support to let the compiler generate code that properly saved
all registers and used RTI instead of RTS (or whatever it might
have been called one one's particular flavour of CPU).

Now, once you added a few tiny interrupt handlers, and some context
switching (stack manipulation) routines, pretty much everything else
*could* be done in C, but that doesn't invalidate the point.

I think it's safe to say that none of pure C, pure Lisp, or pure Python
are really capable of being used as the *sole* language to build
an operating system.

It's also safe to say that this is a largely irrelevant point.
It would probably only be of academic interest to try to do
something like that. Any practical attempt would not think more
than twice of resorting to a little "glue" in assembler or in
the form of "canned" byte sequences built by hand and stuck
into the appropriate places in memory...

-Peter

 

[Arich Chanachai]

Actually, when Unix was first written that belief was entirely
correct, and OSes *did* have to be written in assembler.
*nods*


That is, pure C did not have the capability to handle all
that was required. I recall it taking a good decade before it
became *common* to find C compilers with, for example, @interrupt
support to let the compiler generate code that properly saved
all registers and used RTI instead of RTS (or whatever it might
have been called one one's particular flavour of CPU).

If my memory serves me, you are entirely correct.

Now, once you added a few tiny interrupt handlers, and some context
switching (stack manipulation) routines, pretty much everything else
*could* be done in C, but that doesn't invalidate the point.
*nods*


I think it's safe to say that none of pure C, pure Lisp, or pure Python
are really capable of being used as the *sole* language to build
an operating system.
*nods*


It's also safe to say that this is a largely irrelevant point.
It would probably only be of academic interest to try to do
something like that. Any practical attempt would not think more
than twice of resorting to a little "glue" in assembler or in
the form of "canned" byte sequences built by hand and stuck
into the appropriate places in memory...

-Peter

Indeed indeed. Once must remain focused and ask oneself what he/she is
attempting to achieve. In all likelihood, a little asm/c/python glue
work won't hurt the objective, especially given that doing otherwise
would entail a very _low level_ reworking of Python that would take as
much and likely more effort and skill than to resort to asm/c when
Python falls incapable. One could do a Python-Lisp OS w/ Lisp used for
the Python incapable areas, but you would near entirely loose the
benefits of Lisp in that respect--- it would have to be so low-level,
could you call it Lisp? (A question someone else here posed in this
thread.)

- Arich

 

[Roose]

Is an OS written in Lisp also ludicrous? Because it's been done.

Can you point me to this? I'd like to see how "truly" Lisp it is.

My first guess would be -- not very. And I'd like to install it on my PC.

 

[Roose]

Well can you describe what kind of things you want to do exactly?

My guess is you are not out to develop a new algorithm for virtual memory or
task scheduling.

There are many parts to an OS shell. An example is the command line, i.e.
bash and the like in Unix, and cmd.exe in Windows. In Windows, Windows
Explorer could be considered part of the shell, as well as the start menu
and all that user-specific stuff.

Basically you need to pick your OS, and then find out what the API to
program the shell to would be. e.g. in Windows you would do it with the
Win32 API. This will let you do things like delete and create files,
interact with user structures, the registry, etc.

OR, as a first stab -- I would just write a prototype. i.e., don't tie it
to a real OS. Just pretend you have your own "users", your own "file
system", your own display space, etc. This will help you get a much better
idea of what you want to do.

 

[jtauber]

My experiment, Cleese, was making progress before I got distracted by
other things.

The approach was a micro-kernel in C made up of the CPython bytecode
interpreter with the file-related calls to libc ripped out and some
bare-metal port read/writes and memory operations exposed to Python as
built-in functions.

Everything else could then be written in pure Python. Dave Long was
able to write a VGA driver in Python and an implementation of Sokoban
that used it. You could then boot your machine to Sokoban

I should probably get back to it at some stage.

see http://cleese.sourceforge.net/
James Tauber
http://jtauber.com/blog/

 

[Paul Rubin]

Can you point me to this? I'd like to see how "truly" Lisp it is.
http://en.wikipedia.org/wiki/Lisp_machine

My first guess would be -- not very. And I'd like to install it on my PC.

Although written with federal funds at a university, it was never
released to the public, but was instead offered for sale from some
companies. The conflicts over this led to the birth of the free
software movement.

Also, it was never intended to run on PC's (which didn't exist at that
time). It needed special hardware that was built for the purpose of
running Lisp. Lately there are people trying to program PC's to
simulate the Lisp hardware and to get the Lisp Machine software
released (now that the commercial market for it has long since dried
up). However, both of those projects have a ways to go.

 

[Bulba!]

I am not trying to be insulting... but unless someone would like to educate
me otherwise, the idea of an OS written in Python is almost ludicrous. As I
said, I think you might mean an OS SHELL, which would be a reasonable
(although maybe unconventional) thing to write in python.

That's a crazy idea I once had - what if a user could run Python as
a shell of Linux/other free U**x, and access every single subsystem
and piece of data in the object-like manner?

Sure, things like pipes and immediate access to files without having
to open() them and the whole filesystem hierarchy would have to be
added. Quite a lot of plumbing work would have to be done,
but once it were done, it would be like an object operating
system, everything organized nicely in predictable object hierarchies
with dot access.

System scripts would no longer be a mess that breaks every now and
then depending on slight differences in filesystem hierarchy. Python
dictionaries with their ability to quickly bind names to various
objects would make it possible to keep such hierarchy stable, even
with data moving around on different physical disk partitions and
folders. Instead of saying:

# cp /etc/postfix/main.cf /mnt/floppy

the user could say e.g.

..>>> copy system[config][mta] floppy

or

..>>> copy system.config.mta floppy

In an "object system" like Python it could be reasonable to
consider writing a class that would implement at least a
subset of generic interface to the equivalent functionalities
of various MTAs. Maybe. With traditional filesystems trying
smth like that is not really reasonable, because 99% of
the time it would work, but 1% of the time it would fail
and that 1% would ruin the whole venture, as handling
low-level OS complexity for a non-tech user has to be
more reliable than that - in two cases when I installed
Linux for free for someone the installation was finally
deleted by those users because the desktop shortcuts
to devices like floppy and CD ultimately failed to work
for some reason and using 'mount -t iso9660 ...' was
not an option that they would consider learning and
using really.

Think about all the potential! On top of clean, stable
system interface a user could use map, filter and reduce
functions on various system objects. Handlers for
files with known structure could be written, so
dot or list or dictionary access would allow manipulating
data inside the files, like file[rownumber]='spam', or
spreadsheet_file['A2', 100]. Or you could issue commands
like browser.go('somewhere') or desktop.edit('hello.txt').

I was considering writing smth like this myself but it's
way over my head re low-level OS and kernel stuff. :-(

IMHO Linux problem has long been that in its lower level
interface (above kernel and below GUI) is way too much like
traditional UNIX. Sure someone with sysadmin experience
can use it - but in my experience it's way over the head of
the typical user. Trying to teach them about mknod and
/dev/* hierarchy is hopeless and too hard for them.

 

[Andrew Dalke]

Paul Rubin:

Lately there are people trying to program PC's to
simulate the Lisp hardware and to get the Lisp Machine software
released (now that the commercial market for it has long since dried
up). However, both of those projects have a ways to go.

There's a video of someone demoing how to use a Lisp Machine at
http://lemonodor.com/archives/000103.html

Andrew
(e-mail address removed)

 

[Bengt Richter]

Actually, when Unix was first written that belief was entirely
correct, and OSes *did* have to be written in assembler.

That is, pure C did not have the capability to handle all
that was required. I recall it taking a good decade before it
became *common* to find C compilers with, for example, @interrupt
support to let the compiler generate code that properly saved
all registers and used RTI instead of RTS (or whatever it might
have been called one one's particular flavour of CPU).

Now, once you added a few tiny interrupt handlers, and some context
switching (stack manipulation) routines, pretty much everything else
*could* be done in C, but that doesn't invalidate the point.

I think it's safe to say that none of pure C, pure Lisp, or pure Python
are really capable of being used as the *sole* language to build
an operating system.

It's also safe to say that this is a largely irrelevant point.
It would probably only be of academic interest to try to do
something like that. Any practical attempt would not think more
than twice of resorting to a little "glue" in assembler or in
the form of "canned" byte sequences built by hand and stuck
into the appropriate places in memory...

Well, usually there's more than one tool in the chain from source
to bootable loader and OS image, even if you are writing in C and assembler,
and there's usually more to an OS than the bootable image contains.

So ISTM "writing an OS in <language>" is pretty loose talk ;-)

Regards,
Bengt Richter

 

[John Roth]

Well, usually there's more than one tool in the chain from source
to bootable loader and OS image, even if you are writing in C and
assembler,
and there's usually more to an OS than the bootable image contains.

So ISTM "writing an OS in <language>" is pretty loose talk ;-)

Most people who work in the field know what it means, though.
One of the touchstones of language design is whether you can
write the compiler and runtime in that language. PyPy is working
on that for Python, Squeak Smalltalk is written in Smalltalk and
then converted to C for compilation of the runtime. Both of them
are completely virtualizable: you can run them (very slowly) under
themselves for testing.

Given that everything in Python is an object, I see no reason
why one couldn't create a set of objects that are appropriate
for an operating system kernel and can also be converted
cleanly to the operating system subset of C, and write a converter
that does it.

That's the easy part. The hard part is writing the operating
system. Look at the (lack of) progress the GNU HURD
project has been making recently.

John Roth

 

[John Roth]

My experiment, Cleese, was making progress before I got distracted by
other things.

The approach was a micro-kernel in C made up of the CPython bytecode
interpreter with the file-related calls to libc ripped out and some
bare-metal port read/writes and memory operations exposed to Python as
built-in functions.

Everything else could then be written in pure Python. Dave Long was
able to write a VGA driver in Python and an implementation of Sokoban
that used it. You could then boot your machine to Sokoban

I should probably get back to it at some stage.

As my ex-wife was fond of saying, "I wish you'd have
told me it was impossible before I did it."

John Roth

 

[Roose]

OK I've heard of that. But the original poster didn't ask to make a Python
machine and then a Python OS.

My point was that you can't do a lot of hardware interface programming in
pure Python -- there would be so much non-trivial code in C that it would be
hard to call it a Python OS.

So this basically proves my point -- that you need different hardware
altogether in order to make an OS in a high level language like Lisp or
Python.

And if that Lisp OS on a PC simulator works... that still wouldn't disprove
what I'm saying. What does the PC simulator run on? An OS? or it part of
the OS? Either way you've got tons of non-trivial code in C/C++/assembly.

 

[Paul Rubin]

My point was that you can't do a lot of hardware interface programming in
pure Python -- there would be so much non-trivial code in C that it would be
hard to call it a Python OS.

Why do you say that? Is the same thing not true of C, where you need
some assembly code at the lowest levels? Because Linux has some
assembly code, would you say that it's not written in C?

So this basically proves my point -- that you need different hardware
altogether in order to make an OS in a high level language like Lisp or
Python.

It doesn't prove anything of the sort. The Lisp hardware was needed
for performance reasons, back in the day.

 

[Bengt Richter]

Most people who work in the field know what it means, though.

Even if they know the whole story intimately, IMO it is not unlikely
that they are talking about particular chapters, and it may not
be clear whether they are thinking totally radically of rewriting
ALL software in the system in Python, visualizing a holy grail of
saying goodby to both GNU and M$ and having a purely Python-defined
system (which would mean Python tool chains that could prepare binary
images (no theoretical problem) for a computer's boot BIOS and on
from there, as a tour de force project, or whether they want e.g.
to integrate Python front end to gcc (or at one remove via C as is easier),
and not reinvent so many wheels.

One of the touchstones of language design is whether you can
write the compiler and runtime in that language. PyPy is working
on that for Python, Squeak Smalltalk is written in Smalltalk and
then converted to C for compilation of the runtime. Both of them

This "runtime" is then represented as an executable file that an
OS shell you are using will ask its host OS kernel to load and execute
as an OS-managed user process, right? Or is it bootable? If so,
in what format that what loader understands?

are completely virtualizable: you can run them (very slowly) under
themselves for testing.

But what binary is running? The compiled C output of the backend?
Interpeting the Python source of itself to spit out the same C that it itself
was compiled from? That's cool. And it shows that you can prepare
a binary with the help of the C compiler+linker (most likely with gcc libraries)
to produce a userland program executable.

But I suspect some are thinking that Python source should implement
the whole environment it runs in. That's more radical ;-)
I would guess a cross development system hosted on a normal PC OS
but targeting some limited embedded environment would be
where I'd expect to see this kind of thing first. But still
going via C tools to the end product most likely, since C is so ubiquitous.

Given that everything in Python is an object, I see no reason
why one couldn't create a set of objects that are appropriate
for an operating system kernel and can also be converted
cleanly to the operating system subset of C, and write a converter
that does it. OSCTypes?

That's the easy part. The hard part is writing the operating
system. Look at the (lack of) progress the GNU HURD
project has been making recently.

There are more interesting things to do than reinvent wheels though,
even if it is fun doing it in Python ;-)

Regards,
Bengt Richter

 

[Roose]

Why do you say that? Is the same thing not true of C, where you need
some assembly code at the lowest levels? Because Linux has some
assembly code, would you say that it's not written in C?

It's a difference of degree, but an important difference. I haven't looked
at Linux or Windows NT source, but my guess is the assembly used is just
small functions for accessing special CPU instructions for atomicity,
context switching, and the like.

I KNOW they don't have huge amounts of assembly simply because they run on
different architectures.

But are you really going to write a virtual memory system in Python? Are
you going to write a file system, and a task scheduler in Python? Are you
going to have people write device drivers in Python? I'm not saying it
can't be done, but it would be a poor engineering decision, and the
rationale thus far seems to be "Python is cool, I like OSes, let's write a
whole OS in Python". If that's not the case then let me know what your
rationale is.

The right tool for the right job. I love Python probably more than any
other language. But it's not the right tool for every job. I don't know
why people would think it is, just as C is not the right tool for every job
(which is one of the reasons I learned Python, being a C programmer first).

It doesn't prove anything of the sort. The Lisp hardware was needed
for performance reasons, back in the day.

OK, then give me an example of Lisp OS that runs on a PC. I would like to
install it on my PC tomorrow. Or maybe my Mac. That was your whole point,
originally, that since it could be done in Lisp, why not Python?

 

[Paul Rubin]

But are you really going to write a virtual memory system in Python? Are
you going to write a file system, and a task scheduler in Python? Are you
going to have people write device drivers in Python?

Do you know how virtual memory systems, file systems, task schedulers,
and device drivers actually work? Have you ever written any of them?
What exactly do you think the obstacles are to writing them in Python?

I've written file systems in Python, and task schedulers in
Javascript, and they were fine for their purposes. Virtual memory and
device drivers are just more of the same.

OK, then give me an example of Lisp OS that runs on a PC. I would like to
install it on my PC tomorrow. Or maybe my Mac. That was your whole point,
originally, that since it could be done in Lisp, why not Python?

Huh? That's a non-sequitur, nothing prevents you from running Lisp on
your PC or Mac. The same issues issues that apply to OS code, also
apply to user code. The Lisp machine hardware wasn't needed only to
make the OS run fast. The Lisp machine was developed so that people
could deploy large user-level applications written in Lisp, and the
hardware was there to support those applications. And given such a
good Lisp environment, there was no reason to think of writing the OS
in anything other than Lisp.

In fact, the reason the Lisp machine died off was because general
purpose workstation hardware (and later, PC-class hardware) became
fast enough to run Lisp applications without needing special purpose
CPU's. That same PC hardware speed is what makes it possible to run
user applications in Python.

 

[Arich Chanachai]

As my ex-wife was fond of saying, "I wish you'd have
told me it was impossible before I did it."

John Roth

Is that why she divorced you?


You will notice that tauber says he stopped working on Cleese due to
"distraction" not a lack of progress or a notion of impending
doom/"impossibility".

 

[Arich Chanachai]

....

It's a difference of degree, but an important difference. I haven't looked
at Linux or Windows NT source, but my guess is the assembly used is just
small functions for accessing special CPU instructions for atomicity,
context switching, and the like.

I KNOW they don't have huge amounts of assembly simply because they run on
different architectures.


But are you really going to write a virtual memory system in Python? Are
you going to write a file system, and a task scheduler in Python? Are you
going to have people write device drivers in Python?
Cleese!!

I'm not saying it
can't be done, but it would be a poor engineering decision, and the
rationale thus far seems to be "Python is cool, I like OSes, let's write a
whole OS in Python". If that's not the case then let me know what your
rationale is.