Handling OutOfMemory Error

[ruds]

Hi,
I'm operating on a 600MB file to extract some data from it.
I'm using ArrayList and HashTable for storing the values extracted
from the file and manipulating the data obtained.
When I try executing the program I'm getting the following error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.expandCapacity(Unknown
Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuffer.append(Unknown Source)
at
Analysis_Summary_Variable.getdata(Analysis_Summary_Variable.java:61)
at
Analysis_Summary_Variable.main(Analysis_Summary_Variable.java:549)

How should I increase the JVM's memory to handle such large amount of
data?
Is there any other way around it?

 

[softwarepearls_com]

Hi,
I'm operating on a 600MB file to extract some data from it.
I'm using ArrayList and HashTable for storing the values extracted
from the file and manipulating the data obtained.
When I try executing the program I'm getting the following error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Unknown Source)
        at java.lang.AbstractStringBuilder.expandCapacity(Unknown
Source)
        at java.lang.AbstractStringBuilder.append(Unknown Source)
        at java.lang.StringBuffer.append(Unknown Source)
        at
Analysis_Summary_Variable.getdata(Analysis_Summary_Variable.java:61)
        at
Analysis_Summary_Variable.main(Analysis_Summary_Variable.java:549)

How should I increase the JVM's memory to handle such large amount of
data?
Is there any other way around it?

The -Xmx command line flag lets you set the heap size. See java.exe
docs dor details..

 

[EJP]

I'm using ArrayList and HashTable for storing the values extracted
from the file and manipulating the data obtained.

Don't.

Use a database.

 

[Tom Anderson]

I'm operating on a 600MB file to extract some data from it. I'm using
ArrayList and HashTable for storing the values extracted from the file
and manipulating the data obtained. When I try executing the program I'm
getting the following error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuffer.append(Unknown Source)
at Analysis_Summary_Variable.getdata(Analysis_Summary_Variable.java:61)
at Analysis_Summary_Variable.main(Analysis_Summary_Variable.java:549)

How should I increase the JVM's memory to handle such large amount of
data?

-Xmx, as others have mentioned.

Is there any other way around it?

Firstly, the stacktrace there is from a StringBuffer which is trying to
expand itself. If you can work out ahead of time how big that StringBuffer
eventually needs to be, or even put a useful upper bound on it, then you
can create the StringBuffer with that much capacity in the first place,
which will avoid the need to expand it, and might avoid that failure. It
should certainly improve performance.

I'd also look at whether you need to use that buffer at all - you're not
doing something like reading the entire file line by line and putting it
in the buffer, are you? If you are, find an alternative! In general, a
great way to reduce memory use is to find ways of doing things
incrementally, so you don't need to have all your data in memory at once.
For instance, if you were adding up all the numbers in a file, you might
do:

make an empty list of numbers
for line in file:
parse the line to a number
put the number in the list
set the total to 0
for number in list:
add the number to the total
report the total

But you'd use far less memory like this:

set the total to 0
for line in file:
parse the line to a number
add the number to the total
report the total

That's an obvious and trivial and example, and i imagine your example does
not admit such easy improvements. However, with the application of a
sufficient amount of cleverness, some degree of incrementalisation is
often possible.

Otherwise, whilst ArrayList is fine, the HashMap would worry me slightly
in terms of space. How are you using it? Is the set of keys the same or
similar between records? Could you use an object instead of a hashmap?

If you tell us more about your program, we can give you more specific
help.

tom