I believe that both work fine with current JVM's.
If everything you're doing (except the OS) is in the JVM, it works fine.
If what's in the JVM is quite small, it works fine. But if the JVM heap
grows to say a gig or two and you have Firefox, Photoshop, and a dozen
other big applications open as well, the JVM has a tendency to get
swapped out, and when it's used again and a major collection is
triggered, ouch. Nonresponsive for literally *minutes*.
The G1 collector that's supposed to be in JDK7 (thus bringing this topic
full-circle) might alleviate that somewhat, given what I understand about
how it will work. But the optimum logically must be if the paging and the
garbage collector cooperated in some way, which pretty much requires at
the very least adding some kind of specific API support for garbage
collection to the OS.
Ultimately, though, there are security, stability, portability, and other
benefits to standardizing on a VM as user-mode application host and then
performance benefits to moving that VM into the OS and much OS
functionality (however much is not kernel-mode) into the VM and API
library.