java.lang.OutOfMemoryError: Java heap space

[Goofball]

Does anyone have any ideas on why that happened:

[java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap
space

Explaining the situation. I have such code:

for (int i=0; i<10000-1; i++) {
for (int j=i+1; j<10000; j++) {
compareElements(arr, arr[j]);
}
}

Element of the array is a simple class that has two fields (one is
string and another is BitVector) and getters and setters for those
fields. Function compareElements compares the BitVectors of two
elements.

Now the problem: when I run the code, program runs only till then 1334
element in the first cycle. Then I receive the above exception.

Does anyone know, what can that be? Thanks for any help.

 

[Brandon McCombs]

Does anyone have any ideas on why that happened:

[java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap
space

Explaining the situation. I have such code:

for (int i=0; i<10000-1; i++) {
for (int j=i+1; j<10000; j++) {
compareElements(arr, arr[j]);
}
}

Element of the array is a simple class that has two fields (one is
string and another is BitVector) and getters and setters for those
fields. Function compareElements compares the BitVectors of two
elements.

Now the problem: when I run the code, program runs only till then 1334
element in the first cycle. Then I receive the above exception.

Does anyone know, what can that be? Thanks for any help.

Yeah, your JVM is allocated a certain amount of memory from the OS and
you are creating so many objects at a certain size that your JVM is
running out of memory, hence the OutOfMemoryError exception. You can
tune the size of the memory for the JVM by passing in various parameters
to the java command. I believe they look similar to -X256m or something
like that. You could also think about how your programming is allocating
the data and possibly restructure your algorithm or, if possibly, when
you are done working with the data then get rid of it so the GC can
reclaim the memory.

 

[Daniel Pitts]

Does anyone have any ideas on why that happened:

[java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap
space

Explaining the situation. I have such code:

for (int i=0; i<10000-1; i++) {
for (int j=i+1; j<10000; j++) {
compareElements(arr, arr[j]);
}
}

Element of the array is a simple class that has two fields (one is
string and another is BitVector) and getters and setters for those
fields. Function compareElements compares the BitVectors of two
elements.

Now the problem: when I run the code, program runs only till then 1334
element in the first cycle. Then I receive the above exception.

Does anyone know, what can that be? Thanks for any help.

If you post an SSCCE (http://www.physci.org/codes/sscce/), we would be
better able to help you.

 

[Goofball]

I was experimenting with gc() in different ways. I was callling it
after processing 1000 elements. However, that didn't help and the
problem remained the same: after processing 1334 (or 1335) elements I
received the OutOfMemory error.

Does anyone have any ideas on why that happened:

[java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap
space

Explaining the situation. I have such code:

for (int i=0; i<10000-1; i++) {
for (int j=i+1; j<10000; j++) {
compareElements(arr, arr[j]);
}
}

Element of the array is a simple class that has two fields (one is
string and another is BitVector) and getters and setters for those
fields. Function compareElements compares the BitVectors of two
elements.

Now the problem: when I run the code, program runs only till then 1334
element in the first cycle. Then I receive the above exception.

Does anyone know, what can that be? Thanks for any help.

Yeah, your JVM is allocated a certain amount of memory from the OS and
you are creating so many objects at a certain size that your JVM is
running out of memory, hence the OutOfMemoryError exception. You can
tune the size of the memory for the JVM by passing in various parameters
to the java command. I believe they look similar to -X256m or something
like that. You could also think about how your programming is allocating
the data and possibly restructure your algorithm or, if possibly, when
you are done working with the data then get rid of it so the GC can
reclaim the memory.

 

[Goofball]

Don't know, how much is that an SSCCE, but I am posting you my example:

import java.util.ArrayList;
import java.util.BitSet;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class Example {

private class ArrElem {

private String id;
private BitSet fp;

public BitSet getFp() { return fp; }
public void setFp(BitSet fp) { this.fp = fp; }
public String getId() { return id; }
public void setId(String id) { this.id = id;}
}

private List<ArrElem> arr = new ArrayList<ArrElem>();

private Map<ArrElem, Collection<ArrElem>> rez = new HashMap<ArrElem,
Collection<ArrElem>>();

public void processArr() {
//fillArr(10000);
for (int i = 0; i < 10000-1; i++) {
for (int j = i+1; j < 10000; j++) {
compareElements(arr.get(i), arr.get(j));
}
}
}

private void compareElements(ArrElem elem1, ArrElem elem2) {
int bitM = elem1.getFp().cardinality();
int bitN = elem2.getFp().cardinality();

BitSet commonBits = elem1.getFp();
commonBits.and(elem2.getFp());
int bitCommon = commonBits.cardinality();

double index = bitCommon/(bitM+bitN+bitCommon);
if (index < 0.8) {
if (! rez.containsKey(elem1))
rez.put(elem1, new ArrayList<ArrElem>());
rez.get(elem1).add(elem2);
}
}
}

Does anyone have any ideas on why that happened:

[java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap
space

Explaining the situation. I have such code:

for (int i=0; i<10000-1; i++) {
for (int j=i+1; j<10000; j++) {
compareElements(arr, arr[j]);
}
}

Element of the array is a simple class that has two fields (one is
string and another is BitVector) and getters and setters for those
fields. Function compareElements compares the BitVectors of two
elements.

Now the problem: when I run the code, program runs only till then 1334
element in the first cycle. Then I receive the above exception.

Does anyone know, what can that be? Thanks for any help.

If you post an SSCCE (http://www.physci.org/codes/sscce/), we would be
better able to help you.

 

[Ingo Menger]

private void compareElements(ArrElem elem1, ArrElem elem2) {
int bitM = elem1.getFp().cardinality();
int bitN = elem2.getFp().cardinality();

BitSet commonBits = elem1.getFp();
commonBits.and(elem2.getFp());
int bitCommon = commonBits.cardinality();

double index = bitCommon/(bitM+bitN+bitCommon);
if (index < 0.8) {
if (! rez.containsKey(elem1))
rez.put(elem1, new ArrayList<ArrElem>());
rez.get(elem1).add(elem2);
}
}

There you have it.
The index will be 0.0 unless bitM and bitN are both 0. So you end up
creating at least a new ArrayList element in each cycle.

 

[Red Orchid]

---------- snip -------------

private List<ArrElem> arr = new ArrayList<ArrElem>();

private Map<ArrElem, Collection<ArrElem>> rez = new HashMap<ArrElem,
Collection<ArrElem>>();

public void processArr() {
//fillArr(10000);
for (int i = 0; i < 10000-1; i++) {
for (int j = i+1; j < 10000; j++) {
compareElements(arr.get(i), arr.get(j));
}
}
}

It is mere guesswork.


The above code calls 'compareElements' (10000 * 10000) times.
Namely, 100 M times.

private void compareElements(ArrElem elem1, ArrElem elem2) {
int bitM = elem1.getFp().cardinality();
int bitN = elem2.getFp().cardinality();

BitSet commonBits = elem1.getFp();
commonBits.and(elem2.getFp());
int bitCommon = commonBits.cardinality();

double index = bitCommon/(bitM+bitN+bitCommon);

Min of 'index' is 0.
Max of 'index' is smaller than 1.

if (index < 0.8) {

If 'cardinality()' is random,
the probability of calling the bellow code will be about 8/10.

That is, 80 M times (10000 * 10000 * 0.8)

if (! rez.containsKey(elem1))
rez.put(elem1, new ArrayList<ArrElem>());
rez.get(elem1).add(elem2);
}
}
}

The above code adds an element to 'rez' whenever she is executed.
A total of 80 M elements.

320 MBytes memory (80M * 4) will be required at least.

 

[Ingo Menger]

Min of 'index' is 0.
Max of 'index' is smaller than 1.

Yes, it's 0!
Proof:
case 1) bitM and bitM are 0. Then so will bitComnmon. This will result
in zero divide. (BTW, this is a programming error)
case 2) bitM or bitM are > 0. Then the integer division will result in
0.

Thus, index will never be anything else than 0.0. And the programm will
crash occasionally.

Thus, this is always true.

If 'cardinality()' is random,
the probability of calling the bellow code will be about 8/10.

1, to be exact.

The above code adds an element to 'rez' whenever she is executed.
A total of 80 M elements.

It adds an element everytime. A total of 100M elements.

 

[Dijon Yu]

I was experimenting with gc() in different ways. I was callling it
after processing 1000 elements. However, that didn't help and the
problem remained the same: after processing 1334 (or 1335) elements I
received the OutOfMemory error.

I have meet the problem too.
Don't depend on gc(), whatever 。
If you want resolve the problem, you can set the JVM.

You can use java -Xmx256m to avoid the problem.
But I think maybe you can refactor and redesign the algorithm.

 

[Goofball]

Thanks a lot. That helped. I redesigned the alsorithm. Now it works
fine. But when I test it on about 1 million of elements - it crashes.
Starting JVM with more memory helps, but that sometimes cannot be done
(because of the requirements). So, I decided to write my own
implementations of List and Map interfaces, that will flush the results