Wrapper round x86 Assembler

[Fuzzyman]

There might be a really good reason why this hasn't been done *or*
someone might have done it and I just can't find it..... *but*

what about a wrapper to an assembler (presumably for x86 assembly !)
!!
I just wrote some code doing binary operations which would have been
about a zillion times faster in a few lines of assembly code.

I also have fond memories of programming in BBC Basic which had an
inline assembler - so you could wrap your assembly program in Basic.
It meant some commercial games started with Basic !

Anyway - it would be easy to reserve some memory with a string like
object to pass to an 'assembly object' and allow some really nifty
(and fast) stuff ?? For simple algorithms it would be very neat.
Avoiding memory overflow etc would be up to the assembly code 'chunk'
of course.

Regards,


Fuzzy

http://www.voidspace.org.uk/atlantibots/pythonutils.html

 

[Peter Hansen]

There might be a really good reason why this hasn't been done *or*
someone might have done it and I just can't find it..... *but*

what about a wrapper to an assembler (presumably for x86 assembly !)
!!
I just wrote some code doing binary operations which would have been
about a zillion times faster in a few lines of assembly code.

I refer you to "Pyrex"...

-Peter

 

[Harald Massa]

what about a wrapper to an assembler (presumably for x86 assembly !)
I just wrote some code doing binary operations which would have been
about a zillion times faster in a few lines of assembly code.

import psyco
psyco.full()

psyco will automagically and transparently translate python to X86 assembly
where it is usefull.

And psyco is Open Source, so you can dig HOW Dr. Armin Rigo implemented the
"insert assembler code".

Probably there is allready a solution within the psyco source.

Harald

 

[Fuzzyman]

I refer you to "Pyrex"...

-Peter

If I understand correctly (which is probably not likely) - Pyrex is a
compiled language with some C syntax and some Python syntax.

Because the result is compiled, and a step in the compile chain is
assembly, you could include inline assembly in the same way that you
could with C ??

Never having done any C programming that stretches the limit of my
knowledge.

*However* it's not quite what I had in mind. What I'd like to be able
to do is to access machine code subroutines from within normal python.

Not that exploring pyrex is a bad idea - I probably should.
I don't think my 'idea' would be hard to achieve, since effectively
all that would be needed is a wrapper to an existing freeware
assembler *and* an interface to the assembled 'objects'. Ho hum - a
bit beyond me at the moment and until I have a real burning need for
it will have to remain an idea......


Regards,


Fuzzy

http://www.voidspace.org.uk/atlantibots/pythonutils.html

 

[Peter Hansen]

If I understand correctly (which is probably not likely) - Pyrex is a
compiled language with some C syntax and some Python syntax.

Pretty much.

Because the result is compiled, and a step in the compile chain is
assembly, you could include inline assembly in the same way that you
could with C ??

Having assembly as one step in the compile chain is a somewhat
dated concept. Many compilers go directly to object code.

Never having done any C programming that stretches the limit of my
knowledge.

*However* it's not quite what I had in mind. What I'd like to be able
to do is to access machine code subroutines from within normal python.

Ah, you didn't say that. You were asking about performance,
and certainly in any place where resorting to assembly would
give you that, you should be able to get it from Pyrex as well.

Pyrex might also let you interface with "machine code subroutines",
however, provided they support the standard C calling conventions
(with respect to passing args, saving registers and such).

My main reason for pushing Pyrex as opposed to discussing
options for assembly programming is that there's very little
reason for people to be doing assembly in the world these days.
C is higher level yet with an optimizing compiler allows better
performance in all but a few rare cases (meaning a few rare
problem domains and a few rare individuals).

Assembly is also the most platform-specific thing going, whereas
C is "quite" portable in comparison, whether we're talking about
the code itself or the technical knowledge of the programmer.

If you have specific code that exists and does not follow C
calling conventions, then if Pyrex won't let you call it (and
I'm guessing it won't) you could still whip up a C extension
that would, using some inline assembly there. Or if you're
good enough at assembly to prefer it to C, you could probably
just write a Python extension in assembly directly.

-Peter

 

[John Roth]

There might be a really good reason why this hasn't been done *or*
someone might have done it and I just can't find it..... *but*

what about a wrapper to an assembler (presumably for x86 assembly !)
!!
I just wrote some code doing binary operations which would have been
about a zillion times faster in a few lines of assembly code.

I also have fond memories of programming in BBC Basic which had an
inline assembler - so you could wrap your assembly program in Basic.
It meant some commercial games started with Basic !

Anyway - it would be easy to reserve some memory with a string like
object to pass to an 'assembly object' and allow some really nifty
(and fast) stuff ?? For simple algorithms it would be very neat.
Avoiding memory overflow etc would be up to the assembly code 'chunk'
of course.

Regards,


Fuzzy

You should be able to write a relatively simple C language extension
that would let you do this. The interface to extensions isn't all that
hard to work with. It would probably be pretty useful for some kinds
of intensive crunching. There's one thing to watch out for: Intel
CPU's are going to a model where dynamically generated code won't
work unless you provide the correct incantation. This is to eliminate a
large class of viri.

John Roth

 

[Michael Geary]

My main reason for pushing Pyrex as opposed to
discussing options for assembly programming is that
there's very little reason for people to be doing
assembly in the world these days. C is higher level yet
with an optimizing compiler allows better
performance in all but a few rare cases (meaning a
few rare problem domains and a few rare individuals).

On modern CPUs, unless you really know what you're doing, hand-written
assembly code is likely to be slower than code generated by a good
optimizing C/C++ compiler. If your assembly code is at all straightforward,
it will definitely be slower than compiled code, because you'll be stalling
the CPU all over the place when one instruction waits for the results of a
previous one.

-Mike

 

[Paddy McCarthy]

There might be a really good reason why this hasn't been done *or*
someone might have done it and I just can't find it..... *but*

what about a wrapper to an assembler (presumably for x86 assembly !)
!!
I just wrote some code doing binary operations which would have been
about a zillion times faster in a few lines of assembly code.

I also have fond memories of programming in BBC Basic which had an
inline assembler - so you could wrap your assembly program in Basic.
It meant some commercial games started with Basic !

Anyway - it would be easy to reserve some memory with a string like
object to pass to an 'assembly object' and allow some really nifty
(and fast) stuff ?? For simple algorithms it would be very neat.
Avoiding memory overflow etc would be up to the assembly code 'chunk'
of course.

Regards,


Fuzzy

http://www.voidspace.org.uk/atlantibots/pythonutils.html

I.m sure that what I am about to suggest isn't as high tech as you
were expecting ,
but I thought that Python already has a way of accessing functions in
a Unix
shared library, or Windows DLL.
If you cancompile Python on the platform than you no doubt have an
assembler handy too.
So, you could put your assemble language in a string; write the string
to a file;
assemble the file to create a shered library or DLL, then call the
function from Python!

I don't think I'd do it that way though

Cheers, Pad.

 

[Josiah Carlson]

On modern CPUs, unless you really know what you're doing, hand-written

assembly code is likely to be slower than code generated by a good
optimizing C/C++ compiler. If your assembly code is at all straightforward,
it will definitely be slower than compiled code, because you'll be stalling
the CPU all over the place when one instruction waits for the results of a
previous one.

Yes and no. With the existance of the Tomasulo algorithm for register
renaming and out-of-order execution, you can pick up quite a bit of the
ILP without even programming it that way. Of course, there is only so
much the Tomasulo algorithm can do for crappy assembly.

- Josiah

 

[Andrew MacIntyre]

*However* it's not quite what I had in mind. What I'd like to be able
to do is to access machine code subroutines from within normal python.

Not that exploring pyrex is a bad idea - I probably should.
I don't think my 'idea' would be hard to achieve, since effectively
all that would be needed is a wrapper to an existing freeware
assembler *and* an interface to the assembled 'objects'. Ho hum - a
bit beyond me at the moment and until I have a real burning need for
it will have to remain an idea......

At the moment you have several alternatives available that I can see:

- build your assembly routines into a DLL/.so and use Thomas Heller's
excellent ctypes module.

- build your assembly routines into a library or DLL/.so and use Pyrex
to call them.

- find a C compiler that understands how to build asm files to objects,
then tweak the Distutils support to call the compiler when it an
assembly source file is included in a Distutils'ified package. On
Windows MinGW should get the job done; gcc can do so for posix
platforms.

- do it the old fashioned way: create a makefile that builds the
components and links to the Python core.

If you want to avoid dealing with the necessary Python C API boilerplate
required to allow Python to call your routine, you'll find the ctypes
solution by far the easiest as your code only has to have a calling
interface compatible with a C compiler. The Pyrex solution could work as
well, but introduces an extra language layer into the situation.

The other 2 approaches require that your code deals with the Python C API,
and you would have to create the necessary include files to replicate
"Python.h" appropriately.

The only downside to the ctypes approach may be performance, as each call
may have a high overhead cost compared to the other approaches.

 

[Peter Hansen]

On modern CPUs, unless you really know what you're doing, hand-written
assembly code is likely to be slower than code generated by a good
optimizing C/C++ compiler. If your assembly code is at all straightforward,
it will definitely be slower than compiled code, because you'll be stalling
the CPU all over the place when one instruction waits for the results of a
previous one.

I think that's what I just said...

 

[Fuzzyman]

[snip..]

Having assembly as one step in the compile chain is a somewhat
dated concept. Many compilers go directly to object code.

Right ? I got confused by the mention that they could handle inline
assembler.... so many of the compilers include an assembler as well...
purely for 'inline assembly'.... ??

Or exceeds it. <wink>

*ahem* - well yeah.
I've never coded in C, but I've done some assembly language
programming for the Amiga (the 68000 series processor - great to
program). I had to learn a little bit about C because I had to build
C structures to pass to the operating system which was written in C
(well, written in B first, and then recoded in C ). That was all a
while ago though.

Ah, you didn't say that. You were asking about performance,
and certainly in any place where resorting to assembly would
give you that, you should be able to get it from Pyrex as well.

No... but I said something similar ;-)

Pyrex might also let you interface with "machine code subroutines",
however, provided they support the standard C calling conventions
(with respect to passing args, saving registers and such).

Right - if I was to pursue this I would certainly need to use
something to build a wrapper interface.

My main reason for pushing Pyrex as opposed to discussing
options for assembly programming is that there's very little
reason for people to be doing assembly in the world these days.
C is higher level yet with an optimizing compiler allows better
performance in all but a few rare cases (meaning a few rare
problem domains and a few rare individuals).

However, I was *hoping* for a more dynamic solution than the 'compiler
chain' one.

I recently did some coding that involved classic 'bit-twiddling' (for
want of a better expression). One textbook I found showed a similar
example and said 'this is a good example of code that would be much
better written in assembly'.

My past experience of assembly was using an operating system that
handled multitasking 'behind the scenes' (pre-emptive multitasking) -
so there was no performance hit on the rest of your system from
writing crappy assembly - and the internal architecture of the
processor was useful enough to make coding fun.....

Assembly is also the most platform-specific thing going, whereas
C is "quite" portable in comparison, whether we're talking about
the code itself or the technical knowledge of the programmer.

If you have specific code that exists and does not follow C
calling conventions, then if Pyrex won't let you call it (and
I'm guessing it won't) you could still whip up a C extension
that would, using some inline assembly there. Or if you're
good enough at assembly to prefer it to C, you could probably
just write a Python extension in assembly directly.

-Peter

Hmmm..... oh well. Maybe my idea of dynamically assembled 'assembly
subroutines' for high speed number crunching is a pipe-dream. It
seemed like a good idea.

It does occur to me that for a certain subset of a 'pseudo assembly
language' a 'cross-platform assembler' could work (dynamically
assemble directly to machine code - easy enough to build binaries per
platform) and include the alogrithms mentioned by the other
posters.... Nice idea - well beyond me though. It's easy enough to
have ideas that other people ought to do.

Regards,

Fuzzy

http://www.voidspace.org.uk/atlantibots/pythonutils.html

 

[Peter Hansen]

Right ? I got confused by the mention that they could handle inline
assembler.... so many of the compilers include an assembler as well...
purely for 'inline assembly'.... ??

That's correct. Of course, many also do the CC+AS thing, but
I think that's no longer the most common approach.

My past experience of assembly was using an operating system that
handled multitasking 'behind the scenes' (pre-emptive multitasking) -
so there was no performance hit on the rest of your system from
writing crappy assembly - and the internal architecture of the
processor was useful enough to make coding fun.....

The 68K was certainly assembly-friendly, compared to anything
yet produced by Intel.

It does occur to me that for a certain subset of a 'pseudo assembly
language' a 'cross-platform assembler' could work

I really think that in most ways that matter (not quite all, but most)
this is exactly what C was designed to be and is. It is only a very
thin layer on top of assembly, really. You can think of it is a
pseudo-high-level language designed to allow assembly programmers to
write code roughly the way they would if they were not just hacking.
The Amiga's huge variety of STRUCT macros, for example, are
practically field-for-field identical to how C structs work, mostly
just providing an efficient method for specifying the offsets of
each field. Pointers are nothing more than a more concise way of
expressing the "indirect" addressing mode. Heck, C even has (or had)
the "goto" statement and labels! And the for loop syntax is basically
just how assembly loops are done, but again in a more concise and
readable fashion.

I think anyone who attempted to do a "cross-platform assembler"
would quickly find themselves heading in the direction of something
like C. (I know such things have actually been attempted, and even
exist, by the way. They just aren't mainstream and clearly C has
greater success.) Any attempt to stay at the level of hardware
registers would immediately prevent the term "cross-platform" from
applying.

-Peter

 

[Fuzzyman]

I.m sure that what I am about to suggest isn't as high tech as you
were expecting ,
but I thought that Python already has a way of accessing functions in
a Unix
shared library, or Windows DLL.
If you cancompile Python on the platform than you no doubt have an
assembler handy too.
So, you could put your assemble language in a string; write the string
to a file;
assemble the file to create a shered library or DLL, then call the
function from Python!

I don't think I'd do it that way though

Cheers, Pad.

Ha - in someways this is more like what I envisioned. What would have
been nice would have been to have kept the string in memory and called
it directly....(a nice dynamic solution - for a nice dynamic language)
- but if modern OS / processor combinations have real issues with
jumping directly into 'unoptimized' code - then it may remain a
pipe-dream... *sigh*

Regards,


Fuzzy

 

[Fuzzyman]

That's correct. Of course, many also do the CC+AS thing, but
I think that's no longer the most common approach.


The 68K was certainly assembly-friendly, compared to anything
yet produced by Intel.


I really think that in most ways that matter (not quite all, but most)
this is exactly what C was designed to be and is. It is only a very
thin layer on top of assembly, really. You can think of it is a
pseudo-high-level language designed to allow assembly programmers to
write code roughly the way they would if they were not just hacking.
The Amiga's huge variety of STRUCT macros, for example, are
practically field-for-field identical to how C structs work, mostly
just providing an efficient method for specifying the offsets of
each field. Pointers are nothing more than a more concise way of
expressing the "indirect" addressing mode. Heck, C even has (or had)
the "goto" statement and labels! And the for loop syntax is basically
just how assembly loops are done, but again in a more concise and
readable fashion.

I think anyone who attempted to do a "cross-platform assembler"
would quickly find themselves heading in the direction of something
like C. (I know such things have actually been attempted, and even
exist, by the way. They just aren't mainstream and clearly C has
greater success.) Any attempt to stay at the level of hardware
registers would immediately prevent the term "cross-platform" from
applying.

-Peter

In summary - I'm better off learning C and how to write Python
extensions !!
It's just a very 'undynamic solution'.

Ho hum.... I've been trying to avoid delving into the arcane
makefile/linker/compiler chain mysteries... probably be more useful
anyway.

I will be using MinGw and gcc I think.... great fun - any suggestions
of good resources to start learning :
a) The mechanics and plumbing of using MinGw to compile for the win32
platform (building makefiles, system configuration etc)
b) learning C

I'm 'aware' of the concepts of memory allocation, garbage collection
etc - but I'm sure there's plenty more to learn..........

Regards,


Fuzzy

http://www.voidspace.org.uk/atlantibots/pythonutils.html

 

[Peter Hansen]

In summary - I'm better off learning C and how to write Python
extensions !!
It's just a very 'undynamic solution'.

Ho hum.... I've been trying to avoid delving into the arcane
makefile/linker/compiler chain mysteries... probably be more useful
anyway.

That is where Pyrex comes in. You aren't supposed to have to know
all those gory details then. Just some of them.

I'm 'aware' of the concepts of memory allocation, garbage collection
etc - but I'm sure there's plenty more to learn..........

I doubt you'll need to encounter those if you're just trying to
squeeze performance out of some bit-twiddling code.

-Peter

 

[Michael Hudson]

In summary - I'm better off learning C and how to write Python
extensions !!

I'm a bit lost on where this thread is at, but I've written Python
extensions in PPC assembly (more to see if I could than for any real
reason...).

Cheers,
mwh

 

[Fuzzyman]

That is where Pyrex comes in. You aren't supposed to have to know
all those gory details then. Just some of them.


I doubt you'll need to encounter those if you're just trying to
squeeze performance out of some bit-twiddling code.

-Peter

More trying to learn some new techniques / tools. The bit twiddling
was just a very good example where assembler might actually have been
the *simplest* solution.

Anyway thanks for your help - maybe I'll look into pyrex as a way of
coding python extensions.

The only compielr I have is gcc under mingw - and I'm not at all sure
it's set up right. Does pyrex still need a C-compiler or *is* pyrex a
compiler ? Oh well... I'm sure I'll work it out.

Thanks

Fuzzy

http://www.voidspace.org.uk/atlantibots/pythonutils.html

 

[Thomas Heller]

There might be a really good reason why this hasn't been done *or*
someone might have done it and I just can't find it..... *but*

what about a wrapper to an assembler (presumably for x86 assembly !)
!!
I just wrote some code doing binary operations which would have been
about a zillion times faster in a few lines of assembly code.

I also have fond memories of programming in BBC Basic which had an
inline assembler - so you could wrap your assembly program in Basic.
It meant some commercial games started with Basic !

Anyway - it would be easy to reserve some memory with a string like
object to pass to an 'assembly object' and allow some really nifty
(and fast) stuff ?? For simple algorithms it would be very neat.
Avoiding memory overflow etc would be up to the assembly code 'chunk'
of course.

The following has been reposted by Bradley Schatz to the ctypes mailing
list, it may indeed contain what you want, and it *is* very
interesting. The original post was to the bugtraq mailing list (or so):

from (e-mail address removed)

Today marks another solar cycle I've spent on this planet. To celebrate I'd
like to share one of my toys with all of you.

Adder is a tool I wrote for myself, so that I could experiment with runtime
modification of binary applications. I've found it really useful for
prototyping run-time patches, understanding the effects and possibilities of
call-hooking and other run-time program tweaks; that sort of thing. I hope
you might find it useful too...


Binary:
http://www.rootkit.com/vault/x3nophi1e/adder-0.3.3-win32.zip
( NT 4 / 2000 / XP / 2003 )

Source:
http://www.rootkit.com/vault/x3nophi1e/adder-0.3.3-src.zip

Documentation:
http://www.rootkit.com/vault/x3nophi1e/adder-manual.zip
( please read the installation instructions in here. )


The way it works is fairly simple. Adder allows you to inject a python
interpreter into any win32 process. That interpreter then runs a script
within the context of your target process which is able to instrument and
modify the target in any way it sees fit. Included are a extensions to the
python language that provide:

- safe pointer support
- execution path hooking in python and C++. Hooks can be installed at
something close to instruction granularity.
- x86 instruction manipulation. (based on z0mbie's ADE32 engine)
- programmable x86 instruction disassembler. (a win32 port of libdisasm from
The Bastard)
- x86 assembler. (Dave Aitel's Mosdef 1.1)

These features make it easy to play with the deep majik of really low-level
code hacking in an efficient, sophisticated, high-level language. So adder
is a sort of meta-tool which you might use to script things like:

- dynamic analysis. Hook every function in jscript.dll and graph which ones
execute when a HTML page's script runs.
- API interception. Should IE really be allowed to open an .exe straight of
the web?
- run-time patching. Get rid of those pesky bugs.
- binary forensics. Packers aren't so hard to crack when they run.

Performance and stability are pretty good at this point. Since it's a tool I
wrote for my own use, there are lots of rough edges that need to be cleaned
up. I've been waiting to find the time to fix these for ages and never seem
to. So you'll excuse the occasional glitch. Please tell me if you find
something really horrid.

Hope you all find this interesting, and maybe even useful.

~x

----------

 

[Greg Ewing]

Intel
CPU's are going to a model where dynamically generated code won't
work unless you provide the correct incantation. This is to eliminate a
large class of viri.

As long as Microsoft continue to do things like providing
a nice cross-platform virus environment in their word
processors and mail handlers, I can't see this helping much...