JDBC - Out of memory - null pointer exception?

[Istvan]

Dear Group,

I have the following problem:

In a Java program I reuse the same connection (native driver) to
connect to a MySQL database. I have decided to test the program with
large inserts, that's when I encountered this problem: after a number
of - nonconsistent - number of inserts the program throws a null
pointer exception.

I pass instances of my classes to the database insert method, which
unpacks the attributes of the objects, inserts them into an sql string
and sends it off to the database. Immediately after it reads the new
identity, and uses it to read the attributes back into the passed
object and returns it.

In my test method I then use this object to do some operations and
then loop.

The database connection class is fairly large (perhaps bloated) and
each of these passed objects have approximately 10 associated objects.

Anyway, after approx. 550 inserts the returned object is null and when
I call the aforementioned operations, quite understandably, the test
method throws a NullPointerException.

As I use Eclipse, I have tried to run the code in debug mode, there I
got to over 1000 successful inserts (and then aborted, as it took a
long time). Also, if I insert loops (up to e.g. Integer.MAX_VALUE/10)
for say each 200th insert I can get a lot more inserts, still, it
still throws that NullPointerException at some insert.

I suspect that VM runs out of memory, in some curious way. I have also
tried to call finlize() and gc() for each 200th loop, with no success.

Any thoughts? Is my code crappy so that it keeps collecting objects
somewhere without dereferencing them, thus filling up memory? Is it
Java not doing the garbagecollection thing very well (as mentioned
earlier, it helps to put in long loops or running in debug mode which
is a lot slower than normal execution)? Is it something else?

Any help would be greatly appreciated. Just please don't tell me to
use smaller/fewer objects - it's not really an option.

Thanks in advance,

Istvan

 

[Ike]

I suspect your server closes the connection after a finite amount of time.
Have you tried the notion of connection pooling instead? -Ike

 

[Istvan]

I suspect your server closes the connection after a finite amount of time.

Have you tried the notion of connection pooling instead? -Ike

Thanks for the answer.

I use a self implemented connection pool, it even tests whether the
connection is functioning or not before returning it to the class that
uses it (otherwise it creates a new one and returns that).

The call chain looks sort of like this

Lots of transport classes <-- TestMethod --> DBConnector -->
ConnectionPool

The test method creates transport objects, passes them to the a
DBConnector object, this aquires a native (Java implementation of
MySQL JDBC driver, no ODBC) connection object from the ConnectionPool
and uses it for one insert, one identity request and one read, after
which it releases this connection back to the ConnectionPool.

The test method loops and off we go again.

The connection should not be closed, when it's used, as it's tested
before use. Also, I don't get exceptions from the try-cathc
surrounding my insert method, rather from the test method when it
apparently tries to call a method on a returned transport object that
had worked fine the previous 550 times, but which in this is case
null.

Also, as mentioned in the previous post, if I insert delaying loops,
or even better: just run in the slower debug mode, I get more inserts!
An increase from 550 to 1300 successful inserts means that I use at
least twice as much time - connected - as before. Not to forget that
each loop/insert takes somewhat longer in debug mode.

However, I have seen that JDBC has itself implemented a connection
pool - I would like to give it a try - if I just could get my hands on
some simple and good documentation...

Thanks again for the suggestion,

Istvan

 

[Ike]

If you discover what the cause is, if you could kindly email me to let me
know, I am very curious about this. THanks, Ike

 

[Sudsy]

If you discover what the cause is, if you could kindly email me to let me
know, I am very curious about this. THanks, Ike

If you're performing bulk inserts, would it be fair to assume that
you've done Connection.setAutoCommit( false )? Did you know that
the database could be running out of space on the rollback segment
if you perform too many inserts without interspersed commits? If
you start having problems after 550 inserts, have you tried doing
a commit after every 500?
Bulk inserts are always tricky. In places where I've worked, when
it came time to perform these operations extra space would have
to be allocated to the rollback segments, indexes and constraints
removed from the table, etc. Depending on the particular RDBMS it
can be a lot more efficient to create an index after all the data
is loaded than to have each row insert be indexed.
YMMV

ps. That's why I consider unbridled access to a database without
some knowledge of the underlying structure to be a "bad thing".
Ever see an ad-hoc query which involves an outer join and takes
24 hours to complete? Ugh!

 

[Istvan]

Thanks for the suggestions.

If you're performing bulk inserts, would it be fair to assume that
you've done Connection.setAutoCommit( false )?

Inserts are performed without autocommit: that's true. But since I am
using
a class that inserts data, *Each* inserted row is committed before
control returns the test method, which then sends a new set of data in
the long loop.

With other words: It can hardly be the problem.

Ever see an ad-hoc query which involves an outer join and takes
24 hours to complete? Ugh!

No...

I still suspect that the cause to this problem is in Java: it probably
runs out of memory...


To Ike:
No solutions discovered yet... :-|

Thanks for all answers,
still hoping for a solution,
Istvan

 

[Mike]

(e-mail address removed) (Istvan) wrote in message
Try to avoid the JVM "-server" option.

Try to use the incremental garbage collector, like so:
java -Xincgc -jar something.jar

IC works wonders.