File map performance

[George2]

Hello everyone,


I have several physical file and I want to use file map
(MapViewOfFileEx) to map the file into memory to improve performance.
Each file is about several hundred M bytes. All the memory mapped
files are kept open during my application.

The mapping is successful, but the strange thing is,

1. the performance to access the files which are opened at first is
very fast;
2. the performance to access the files which are opened later is
slower and slower (the performance to access the 10th file is very
bad).

Any ideas to improve performance?


thanks in advance,
George

 

[Victor Bazarov]

I have several physical file and I want to use file map
(MapViewOfFileEx) to map the file into memory to improve performance.

....whatever "file map" is...

Each file is about several hundred M bytes. All the memory mapped
files are kept open during my application.

The mapping is successful, but the strange thing is,

1. the performance to access the files which are opened at first is
very fast;
2. the performance to access the files which are opened later is
slower and slower (the performance to access the 10th file is very
bad).

Any ideas to improve performance?

You need to ask in the newsgroup where "file map" is on topic. It is
not a feature of the language, it's a feature of your OS.

V

 

[Jim Langston]

Hello everyone,


I have several physical file and I want to use file map
(MapViewOfFileEx) to map the file into memory to improve performance.
Each file is about several hundred M bytes. All the memory mapped
files are kept open during my application.

The mapping is successful, but the strange thing is,

1. the performance to access the files which are opened at first is
very fast;
2. the performance to access the files which are opened later is
slower and slower (the performance to access the 10th file is very
bad).

Any ideas to improve performance?

It sounds like you are running into disk swapping and/or thrashing issues.
Physical memory is limited (1GB 2GB is common) and is shared amoung
programs, the operating system, etc... As you request memory from the OS it
allocates the memory. :The more memory you request the more that gets
swapped to disk.

So basically what it sounds like is happening, you load a file into memory,
and keep loading files. The memory starts to get swapped to disk. As you
go back and read the memory the OS has to read it from the disk swap file
and write other memory to the swap file for what it's replacing. So now
instead of saving time from reading a file once, it has to write and read.

Several hundred MB of many files is too much to keep in physical memory at
one time. I'm thinking it may actually be faster to read the files as
needed counting on the OS's own file buffering to reduce physical disk I/O.

 

[Rahul]

Hello everyone,

I have several physical file and I want to use file map
(MapViewOfFileEx) to map the file into memory to improve performance.
Each file is about several hundred M bytes. All the memory mapped
files are kept open during my application.

The mapping is successful, but the strange thing is,

1. the performance to access the files which are opened at first is
very fast;
2. the performance to access the files which are opened later is
slower and slower (the performance to access the 10th file is very
bad).

Any ideas to improve performance?

thanks in advance,
George

Are you using mmap() and does your operating system support virtual
memory?
What is your objective and what kind of application are you developing?

 

[Rahul]

Several hundred MB of many files is too much to keep in physical memory at
one time. I'm thinking it may actually be faster to read the files as
needed counting on the OS's own file buffering to reduce physical disk I/O.

I really wonder how OS's own file buffering would improve performance
for the OP's scenario...
after all, it too would need to load the contents of the file into the
memory right? So i would imagine the same performance issues with OS's
mechanisms too...

may be the op's application can't read from a file but instead only
from the memory with the help of a pointer to the buffer...
I have faced some third party applications, which accepts data only in
a buffer and not in a file... i used mmap() in those case...

 

[Jim Langston]

I really wonder how OS's own file buffering would improve performance
for the OP's scenario...
after all, it too would need to load the contents of the file into the
memory right? So i would imagine the same performance issues with OS's
mechanisms too...

may be the op's application can't read from a file but instead only
from the memory with the help of a pointer to the buffer...
I have faced some third party applications, which accepts data only in
a buffer and not in a file... i used mmap() in those case...

Because then the OS would only have to buffer it once. Not the OS buffering
it, and him buffering it as is the current case. It depends on how well the
OS buffers files.

 

[Rahul]

Because then the OS would only have to buffer it once. Not the OS buffering
it, and him buffering it as is the current case. It depends on how well the
OS buffers files.

Nothing to comment until the OP tells about his implementation...
But i'm sure mmap() is a single buffer operation just like file
operation...

 

[James Kanze]

Nothing to comment until the OP tells about his implementation...
But i'm sure mmap() is a single buffer operation just like file
operation...

Probably. The real difference is that when you have to
explicitly read the data in in order to access it, you tend to
do everything you can on that specific data immediately, so you
don't have to read it in a second time. Where as with virtual
memory...

With regards to the initial poster: I'd first try to unload the
file images I didn't need. And then I'd attact locality: try to
ensure that you jump around in the mapped memory as little as
possible.