P
Peter Nolan
Hi All,
I have some software that currently loads database tables into sorted arrays
in memory to be binary searched by processes. (Long story as to why it does
this..)
Each process must either read the data from the in memory array or from the
database itself if the table is too large to be loaded into memory. (2GB is
usually the limit because the array is indexed by a 32 bit signed integer.
I have never had one of these tables go over 2GB, even when the client had
20M customers.)
This current set up is fine for companies with a few million customers and a
few million transactions per day. The software works just fine on win2000
and solaris/aix. None of my existing customers has large enough lookup
tables or so many transactions that it is a problem to load them into
memory.
However, one of the major 'upgrades' I want to make to this code is to
enable it to share these in-memory arrays between many processes. This
sharing of memory will make the product pretty much 'endlessly scalable',
and this is where I want to get it to.
The idea is that if the customer is a large telephone company then I can
split the Call Detail Records into say 12 input files and have 12 processes
run at the same time. By sharing the tables to be searched in memory I can
cut the memory requirement by a factor of 12. By putting all the data to be
looked up into memory the speed of processing the CDRs can be maximised
because a lookup to a database is at least 20 times slower than the lookup
to the array even if the data is in memory.
I have the code I need to do this under solaris.....but from the examples I
can see in VS.NET it looks like windows 2000 acts a little differently.
Questions....
Q1. Does anyone have a fragement of code for a producer/consumer for memory
mapped IO under win2000 that they are willing to share? Or is there a good
public example anywhere that someone here can point to? The examples I can
find with vs.net are in java or very old.
Q2. Is there a 2GB limit on creating a memory mapped file under win2000?
Q3. Will a memory mapped file like this run at the same speed as an in
memory array that is created by calloc like the current one? (Once the
producer has loaded the table into memory all other processes are
'read-only' and will be able to read the memory at the same time, no locks
required.)
Q4. Are there any other pointers the experts on this group might like to
pass along to someone going to do their very first Memory Mapped IO piece of
work? Are there any 'gotchas' for beginners in this area?
All assistance is much appreciated!!!
Best Regards
Peter Nolan
www.peternolan.com
I have some software that currently loads database tables into sorted arrays
in memory to be binary searched by processes. (Long story as to why it does
this..)
Each process must either read the data from the in memory array or from the
database itself if the table is too large to be loaded into memory. (2GB is
usually the limit because the array is indexed by a 32 bit signed integer.
I have never had one of these tables go over 2GB, even when the client had
20M customers.)
This current set up is fine for companies with a few million customers and a
few million transactions per day. The software works just fine on win2000
and solaris/aix. None of my existing customers has large enough lookup
tables or so many transactions that it is a problem to load them into
memory.
However, one of the major 'upgrades' I want to make to this code is to
enable it to share these in-memory arrays between many processes. This
sharing of memory will make the product pretty much 'endlessly scalable',
and this is where I want to get it to.
The idea is that if the customer is a large telephone company then I can
split the Call Detail Records into say 12 input files and have 12 processes
run at the same time. By sharing the tables to be searched in memory I can
cut the memory requirement by a factor of 12. By putting all the data to be
looked up into memory the speed of processing the CDRs can be maximised
because a lookup to a database is at least 20 times slower than the lookup
to the array even if the data is in memory.
I have the code I need to do this under solaris.....but from the examples I
can see in VS.NET it looks like windows 2000 acts a little differently.
Questions....
Q1. Does anyone have a fragement of code for a producer/consumer for memory
mapped IO under win2000 that they are willing to share? Or is there a good
public example anywhere that someone here can point to? The examples I can
find with vs.net are in java or very old.
Q2. Is there a 2GB limit on creating a memory mapped file under win2000?
Q3. Will a memory mapped file like this run at the same speed as an in
memory array that is created by calloc like the current one? (Once the
producer has loaded the table into memory all other processes are
'read-only' and will be able to read the memory at the same time, no locks
required.)
Q4. Are there any other pointers the experts on this group might like to
pass along to someone going to do their very first Memory Mapped IO piece of
work? Are there any 'gotchas' for beginners in this area?
All assistance is much appreciated!!!
Best Regards
Peter Nolan
www.peternolan.com