A
Alan Gutierrez
I'm implementing file storage for a B+Tree for deployment on Linux,
development on OS X, and wanted to use MappedByteBuffer. This seems like
an easy way to address large pages of data and not have implement
buffering myself.
I've done a little testing...
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.channels.FileChannel.MapMode;
public class MemoryMap {
public static void main(String[] args) throws IOException {
int count = Integer.parseInt(args[0]);
MappedByteBuffer[] mappedByteBuffers =
new MappedByteBuffer[count];
for (int i = 0; i < count; i++) {
mappedByteBuffers = opened(i);
}
}
public static MappedByteBuffer opened(int count)
throws IOException {
int size = 1024 * 1024 * 256; // * 1024; // 1 GB
RandomAccessFile raf =
new RandomAccessFile("hello" + count + ".raf", "rw");
try {
raf.setLength(size);
FileChannel channel = raf.getChannel();
try {
MappedByteBuffer map
= channel.map(MapMode.READ_WRITE, 0, size);
channel.close();
map.put(size - 1, (byte) 7);
} finally {
channel.close();
}
} finally {
raf.close();
}
raf = new RandomAccessFile("hello" + count + ".raf", "rw");
try {
FileChannel channel = raf.getChannel();
try {
MappedByteBuffer map
= channel.map(MapMode.READ_WRITE, 0, size);
channel.close();
if (map.get(size - 1) != 7) {
throw new IllegalStateException();
}
return map;
} finally {
channel.close();
}
} finally {
raf.close();
}
}
}
On a smallish Linode running Ubuntu 10.4 I can create an array of
hundreds of MappedByteBuffer. Here are some timings.
[alan@maribor ~]$ time java MemoryMap 24
real 0m0.119s
user 0m0.067s
sys 0m0.029s
[alan@maribor ~]$ time java MemoryMap 512
real 0m0.225s
user 0m0.150s
sys 0m0.092s
[alan@maribor ~]$
On OS X running Java 1.5 I can create an array of 5, the first pass is
very slow, the second pass is faster, I assume this is because OS X
isn't being clever about creating the empty file.
[alan@postojna ~]$ time java MemoryMap 5
real 0m20.920s
user 0m0.089s
sys 0m1.267s
[alan@postojna ~]$ time java MemoryMap 5
real 0m0.301s
user 0m0.091s
sys 0m0.044s
[alan@postojna ~]$
When I attempt to create an array of 6 250 MB `MappedByteBuffer`s I get:
[alan@postojna ~]$ java MemoryMap 6
Exception in thread "main" java.io.IOException: Cannot allocate memory
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:742)
at MemoryMap.opened(MemoryMap.java:23)
at MemoryMap.main(MemoryMap.java:12)
[alan@postojna ~]$
However:
[alan@postojna ~]$ time
/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java
MemoryMap 6
real 0m0.386s
user 0m0.354s
sys 0m0.087s
[alan@postojna ~]$
I've created up to 24 250MB buffers using Java 1.6 on OS X. It takes a
while to create them, but once created, performance seems reasonable.
[alan@postojna ~]$ time
/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java
MemoryMap 24
real 3m13.779s
user 0m0.358s
sys 0m5.543s
[alan@postojna ~]$ time
/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java
MemoryMap 24
real 0m1.646s
user 0m0.364s
sys 0m0.100s
[alan@postojna ~]$
I've tried the program running on Windows XP with JDK 1.6 running in
Virtual Box and I can't get an array of three pages.
All this leads me to the question: am I on the right track? It seems
like I'll be able to get the benefits I seek,
* paging managed by the operating system,
* a simple memory management strategy.
on Linux and OS X with JDK 1.6.
Or am I misunderstanding the usage of MappedByteBuffer?
development on OS X, and wanted to use MappedByteBuffer. This seems like
an easy way to address large pages of data and not have implement
buffering myself.
I've done a little testing...
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.channels.FileChannel.MapMode;
public class MemoryMap {
public static void main(String[] args) throws IOException {
int count = Integer.parseInt(args[0]);
MappedByteBuffer[] mappedByteBuffers =
new MappedByteBuffer[count];
for (int i = 0; i < count; i++) {
mappedByteBuffers = opened(i);
}
}
public static MappedByteBuffer opened(int count)
throws IOException {
int size = 1024 * 1024 * 256; // * 1024; // 1 GB
RandomAccessFile raf =
new RandomAccessFile("hello" + count + ".raf", "rw");
try {
raf.setLength(size);
FileChannel channel = raf.getChannel();
try {
MappedByteBuffer map
= channel.map(MapMode.READ_WRITE, 0, size);
channel.close();
map.put(size - 1, (byte) 7);
} finally {
channel.close();
}
} finally {
raf.close();
}
raf = new RandomAccessFile("hello" + count + ".raf", "rw");
try {
FileChannel channel = raf.getChannel();
try {
MappedByteBuffer map
= channel.map(MapMode.READ_WRITE, 0, size);
channel.close();
if (map.get(size - 1) != 7) {
throw new IllegalStateException();
}
return map;
} finally {
channel.close();
}
} finally {
raf.close();
}
}
}
On a smallish Linode running Ubuntu 10.4 I can create an array of
hundreds of MappedByteBuffer. Here are some timings.
[alan@maribor ~]$ time java MemoryMap 24
real 0m0.119s
user 0m0.067s
sys 0m0.029s
[alan@maribor ~]$ time java MemoryMap 512
real 0m0.225s
user 0m0.150s
sys 0m0.092s
[alan@maribor ~]$
On OS X running Java 1.5 I can create an array of 5, the first pass is
very slow, the second pass is faster, I assume this is because OS X
isn't being clever about creating the empty file.
[alan@postojna ~]$ time java MemoryMap 5
real 0m20.920s
user 0m0.089s
sys 0m1.267s
[alan@postojna ~]$ time java MemoryMap 5
real 0m0.301s
user 0m0.091s
sys 0m0.044s
[alan@postojna ~]$
When I attempt to create an array of 6 250 MB `MappedByteBuffer`s I get:
[alan@postojna ~]$ java MemoryMap 6
Exception in thread "main" java.io.IOException: Cannot allocate memory
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:742)
at MemoryMap.opened(MemoryMap.java:23)
at MemoryMap.main(MemoryMap.java:12)
[alan@postojna ~]$
However:
[alan@postojna ~]$ time
/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java
MemoryMap 6
real 0m0.386s
user 0m0.354s
sys 0m0.087s
[alan@postojna ~]$
I've created up to 24 250MB buffers using Java 1.6 on OS X. It takes a
while to create them, but once created, performance seems reasonable.
[alan@postojna ~]$ time
/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java
MemoryMap 24
real 3m13.779s
user 0m0.358s
sys 0m5.543s
[alan@postojna ~]$ time
/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java
MemoryMap 24
real 0m1.646s
user 0m0.364s
sys 0m0.100s
[alan@postojna ~]$
I've tried the program running on Windows XP with JDK 1.6 running in
Virtual Box and I can't get an array of three pages.
All this leads me to the question: am I on the right track? It seems
like I'll be able to get the benefits I seek,
* paging managed by the operating system,
* a simple memory management strategy.
on Linux and OS X with JDK 1.6.
Or am I misunderstanding the usage of MappedByteBuffer?
