java, multithreading, fork() - performance problem on Solaris

[bugbear]

I'm using tomcat on Solaris with around 100 threads.
This is perfectly normal and mundane, and works well.

However, I have now noticed that when the JSP's
call system() (either via runtime.exec or new processBuilder)
the whole thing slows down a lot, and the kernel
usage goes crazy. Idle time goes UP (which is odd).
JSP response tanks.

No, I can't stop shelling out from tomcat :-(

I have traced this problem to fork(), to whit:

The fork() and fork1() functions suspend
all threads in the process before proceeding.

(ref:
http://docs.sun.com/app/docs/doc/816-0212/6m6nd4n9e?a=view
)

They're all started up again after the fork(), of course, and
I have confirmed this diagnosis by crawling over an ENORMOUS
truss output log.

Now, with 100 tomcat threads, this is a bit of an overhead (*)

It appears that multithreaded Java and fork()
on Solaris have a rather deep incompatibility.

Has any body else seen/experienced this, and
has anybody found a solution/workround?

BugBear

(*) British understatement.

 

[Tim Bradshaw]

Has any body else seen/experienced this, and
has anybody found a solution/workround?

It might be that posix_spawn has different behaviour, though this
seems unlikely to me. Otherwise the obvious thing would be to have a
separate, single-threaded, process which does the forks to which your
huge JVM can talk using some IPC mechanism.

 

[Casper H.S. Dik]

It might be that posix_spawn has different behaviour, though this
seems unlikely to me. Otherwise the obvious thing would be to have a
separate, single-threaded, process which does the forks to which your
huge JVM can talk using some IPC mechanism.

System and posix_spawn both use vfork(); vfork() also needs to
suspend all threads, pretty much by definition.

Note that fork/fork1 are expensive in requiring all page
table entries for the process to be marked "copy on write".

vfork() is generally cheap because it only needs to suspend the
parent (but that is quite different when the parent has 100s
of threads). (While vfork() is documented as not being thread
safe, the restrictions on its use are known and fixed so we
do know how to safely use it in the implementation of system(),
popen() and posix_spawn)


The fork() process creation model and the POSIX threads in a single
address space model don't really work well together and the
combination should be avoided.

Casper

 

[bugbear]

It might be that posix_spawn has different behaviour, though this
seems unlikely to me. Otherwise the obvious thing would be to have a
separate, single-threaded, process which does the forks to which your
huge JVM can talk using some IPC mechanism.

Yes, thank you.

That solution is quite natural, but would complicate
out software architecture (and hence deployment).

I was hoping to avoid such a solution if possible...
(which it may not be :-( )

It appears that (in theory) the problem
is independant of Java; any strongly multi threaded
code would have "issues" with shell out.

It's just that Java tends to *be* strongly
multi-threaded.

BugBear

 

[Arne Vajhøj]

I'm using tomcat on Solaris with around 100 threads.
This is perfectly normal and mundane, and works well.

However, I have now noticed that when the JSP's
call system() (either via runtime.exec or new processBuilder)
the whole thing slows down a lot, and the kernel
usage goes crazy. Idle time goes UP (which is odd).
JSP response tanks.

No, I can't stop shelling out from tomcat :-(

[other have commented on the fork part]

You may need to run external commands, but could
you instead of starting new processes have a
pool of permanent processes you just communicated
with ?

Arne