Large Buffer Memory Management

[Randy Yates]

Hi Folks,

This may be somewhat off-topic, but it sorta fits.

This is a C++ application that will be built using gcc and targeted
(via the wonderful wxWidgets library) for both Windoze and Linux.

Let's say I want to maintain an array of 16-bit integers that can be 100's
of MBs in length. What's the best way to do this? Should I simply
create the array and let the OS deal with swapping? Or should I
use a file as temporary space and fseek through it? Is there some
other better way?

Your thoughts would be appreciated.
--
% Randy Yates % "How's life on earth?
%% Fuquay-Varina, NC % ... What is it worth?"
%%% 919-577-9882 % 'Mission (A World Record)',
%%%% <[email protected]> % *A New World Record*, ELO
http://home.earthlink.net/~yatescr

 

[Victor Bazarov]

This may be somewhat off-topic, but it sorta fits.

It don't, really. If you're concerned with swapping, it's not
a language problem: C++ does not define swapping. :-/

This is a C++ application that will be built using gcc and targeted
(via the wonderful wxWidgets library) for both Windoze and Linux.

Then you need to ask in Windows or Linux newsgroups (or both).

Let's say I want to maintain an array of 16-bit integers that can be
100's of MBs in length. What's the best way to do this? Should I
simply
create the array and let the OS deal with swapping? Or should I
use a file as temporary space and fseek through it? Is there some
other better way?

Start from the simplest solution you can think of, and only if it does
*not* work for you, improve it.

I bet you if you ask in the newsgroup dedicated to your OS, you will
find much more information about how to conform to OS' functionality
related to maintaining large arrays, than in any language newsgroup.

V

 

[Randy Yates]

It don't, really. If you're concerned with swapping, it's not
a language problem: C++ does not define swapping. :-/


Then you need to ask in Windows or Linux newsgroups (or both).


Start from the simplest solution you can think of, and only if it does
*not* work for you, improve it.

I bet you if you ask in the newsgroup dedicated to your OS, you will
find much more information about how to conform to OS' functionality
related to maintaining large arrays, than in any language newsgroup.

Thanks for that suggestion, Victor. I will check there.

If, in the mean time, any other folks here have ideas, I'd like to hear them.
I'm not sure this problem is so far divorced from the language that this is
not a reasonable place to ask.
--
% Randy Yates % "Though you ride on the wheels of tomorrow,
%% Fuquay-Varina, NC % you still wander the fields of your
%%% 919-577-9882 % sorrow."
%%%% <[email protected]> % '21st Century Man', *Time*, ELO
http://home.earthlink.net/~yatescr

 

[Daniel T.]

Let's say I want to maintain an array of 16-bit integers that can be 100's
of MBs in length. What's the best way to do this?

IMHO std::deque is the best solution.

Should I simply create the array and let the OS deal with swapping?

What if the OS doesn't swap?

Or should I use a file as temporary space and fseek through it? Is there some
other better way?

I would only do the above if the deque throws a bad_alloc exception.

 

[Diego Martins]

IMHO std::deque is the best solution.

why using deque instead of vector?

 

[Victor Bazarov]

why using deque instead of vector?

Insertions and removals from both ends are faster, my guess would be.
Also, implementation of deque is likely a "chunk"-based indirect array,
so it would be likely better suited for OSes that swap data... But
that's all not really related to C++.

V

 

[Daniel T.]

why using deque instead of vector?

I liked Victor's response, but my reason was different. I say use a
deque for very large blocks because the deque chunks it's data which
will likely make it easer for the memory manager to fit all the data in
if there is no paging of memory to disk. A deque is likely to cause
fewer fragmentation problems.

 

[peter koch]

Diego Martins skrev:

why using deque instead of vector?

[snip]
Read Victor Bazarovs answer. Also, deque grows in a more gentle manner
.. without any reallocations. For that reason alone, some recommend
std::deque instead of std::vector. Actually, std::vector only has a
small constant speed advantage over std::deque (well - also contiguous
memory) so perhaps they are right?

Peter

 

[mlimber]

Diego Martins skrev:

Let's say I want to maintain an array of 16-bit integers that can be 100's
of MBs in length. What's the best way to do this?

IMHO std::deque is the best solution.

why using deque instead of vector?
[snip]

[snip]
Actually, std::vector only has a
small constant speed advantage over std::deque (well - also contiguous
memory) so perhaps they are right?

Recognizing the OP from comp.dsp, I might speculate that he'd prefer
contiguous memory since most signal processing libraries don't accept
iterators. Without iterator support, the relevant section(s) of data
would need to be copied to contiguous memory and then possibly copied
back after processing. So really, I think Randy will need to be more
specific about how he's going to use the large buffers before we can
speculate accurately on what would be best.

Cheers! --M

 

[Robbie Hatley]

This may be somewhat off-topic, but it sorta fits.

This is a C++ application that will be built using gcc and targeted
(via the wonderful wxWidgets library) for both Windoze and Linux.

Let's say I want to maintain an array of 16-bit integers that can be 100's
of MBs in length. What's the best way to do this? Should I simply
create the array and let the OS deal with swapping? Or should I
use a file as temporary space and fseek through it? Is there some
other better way?

I'd put that much data in a large number of small files, if possible,
rather than one huge file. Possibly keyed by value range. Then just
load one of the smaller files into memory at any one time, and close
the file when done with it for now.

If you do put a very large array (or container) of data in memory,
either declare it global, or "static" within a function, or dynamically
allocate it using new or malloc. I'd recommend against declaring it as
a simple local variable in a function, because that can cause stack
overflow:

void MyWeirdFunc()
{
static int Fred[100000000]; // probably ok, if you have 1 GB of RAM
int Jane[100000000]; // might overflow stack
// ... do stuff ...
return;
}

The "static" version also has the advantage of zeroing it's elements
when you program starts, unlike the non-static version.


--
Cheers,
Robbie Hatley
Tustin, CA, USA
lone wolf intj at pac bell dot net
(put "[usenet]" in subject to bypass spam filter)
home dot pac bell dot net slant earnur slant