is using tons of stl a correct method?

[Peter van Merkerk]

Joe C wrote:

You could use std::vector::reserve() to prevent reallocations. Or you
could use a different allocator with the vector; one that clears memory
before deallocating it. Such an allocator could also be used with other
container classes.

You might find this an interesting read:
http://eddeye.net/src/secalloc/secalloc.pdf

 

[Jerry Coffin]

in a project, I use many,many stl such as stack,list,vctor etc.

somewhere the vector's size is more than 2K. is this a efficient way?

A number like 2K is nearly meaningless by itself. Using 2K to store 80
bytes of information is a whole different story from using 2K to store
1.9K of information.

In the end, std::vector inevitably impose some overhead: if I'm not
mistaken, it's typically implemented as a small data structure
containing a few fixed items (e.g. the currently allocated size, the
current number of items in use) and a pointer to the actual data,
which is normally allocated from the free store. The size allocated
on the free store varies geometrically if you allow it to grow by
default. This limits wasted space to a specific percentage. In
addition, when/if you remove items from the vector, its size normally
does NOT shrink automatically, so if you erase a great deal from the
vector, its overhead can be quite high until/unless you shrink it back
down manually.

Ultimately, the standard library is intended to be useful across a
wide variety of situations. To accomplish that, it cannot concentrate
solely on minimizing memory usage. Rather, it has to balance memory
usage with CPU usage. In most cases, if there's a conflict between the
two, the current specification tends to favor the latter.

 

[rokia]

Thanks for your explanation.

"rokia" <[email protected]> wrote in message

A number like 2K is nearly meaningless by itself. Using 2K to store 80
bytes of information is a whole different story from using 2K to store
1.9K of information.

In the end, std::vector inevitably impose some overhead: if I'm not
mistaken, it's typically implemented as a small data structure
containing a few fixed items (e.g. the currently allocated size, the
current number of items in use) and a pointer to the actual data,
which is normally allocated from the free store. The size allocated
on the free store varies geometrically if you allow it to grow by
default. This limits wasted space to a specific percentage. In
addition, when/if you remove items from the vector, its size normally
does NOT shrink automatically, so if you erase a great deal from the
vector, its overhead can be quite high until/unless you shrink it back
down manually.

Ultimately, the standard library is intended to be useful across a
wide variety of situations. To accomplish that, it cannot concentrate
solely on minimizing memory usage. Rather, it has to balance memory
usage with CPU usage. In most cases, if there's a conflict between the
two, the current specification tends to favor the latter.

--
Later,
Jerry.

The universe is a figment of its own imagination.

 

[tom_usenet]

As an example of
where I do this, I have a data-holding class that over-writes the
memory (for security reasons, using memset()) in the destructor before
releasing the memory. When I was designing the class, I couldn't
think of an easy way to get a vector to do the same. Does this
particular requirement (that no data remains in ram once it is
released) necessitate the use of arrays, or can the same be achieved
using vectors?

Assuming your vector is called vec

memset(&vec[0], 0, vec.size()*sizeof vec[0]);

The trick is that &vec[0] can be used to get the address of the first vector
element (and also that the C++ standard guarantees that a vector elements
occupy contiguous memory).

john

Thanks John. What about if the vector reallocates itself (ie the data is
concatinable)? Is there a way to make certain that the pre-reallocated
vector is over-written?

The best thing would be to write your own allocator. The you can do
the memset in the deallocate function, and then there's no risk of
those precious bytes hanging around when you don't want them to,
regardless of the container you're using.

Tom

 

[J. Campbell]

You might find this an interesting read:
http://eddeye.net/src/secalloc/secalloc.pdf

Thanks Peter for the replies. These authors address the same issues I
was attempting to deal with. My understanding of the STL is too weak
to follow the advice for writing my own allocators for use with the
STL. My AlignInt class takes a file, string, or another class object
to initialize. The data is word-aligned in memory so that
word-oriented operations are guaranteed to work on the data, however a
pointer to the data as bytes is also offered to the class user. data
may be concatinated and assigned, and the class can return it's data
(or a subset of it's data) as a string. I realize that a weakness of
my approach is that it gives direct (public) access to the memory. I
chose to do things this way because I was writing the the class to
support a specific application, not as a general-use container.
However, I've now opted to use the container in several other
projects. I would love receive criticism/comments on how to
better-implement a data-container with these properties. I have
attached the declaration/implementation of the current iteration of my
container. I appologize that I didn't write/edit the code to make it
easy to read on the news-reader. Also...it may not be clear why I
have some particular functions in the class...they were all written to
address specific issues for a particular program, and it is likely
that better, more general solutions exist, which is why I post.



begin------->Align_int_class.h

// Class to put data on the heap and forces the data to be aligned
// with native words...then offer pointers to that data as either
native
// integer or byte. Class overwrites data before releasing so that
data
// does not linger in memory.

// I've tried to keep this class light, but have freely added methods
as/when it's
// been convenient.
#ifndef ALIGNINT
#define ALIGNINT

#include <string>
#include <stdint.h>

class AlignInt{
private:
bool haveMemory;
unsigned int wordsize;

public:
unsigned int len_c; // len of memory in bytes (char)
unsigned int len_l; // len of memory in words (long)
uint32_t* ul; // points to memory
char* c; // points to same memory

// Constructors
AlignInt(); // default constructor...uninitialized pointer
AlignInt(int byte_len); // new empty with space reserved =
byte_len
AlignInt(uint32_t* parray, unsigned int byte_len); // new
object points to preexisting memory. memory must be freed elsewhere
AlignInt(const std::string& stringin); // copy string to new
properly sized AlignInt object

// Copy Constructors
AlignInt(const AlignInt& rhs); //copy
constructor
AlignInt(const AlignInt& rhs1, const AlignInt& rhs2);
//overloaded copy constructor..concatinates 2 objects

// Overloaded operator
AlignInt operator=(const AlignInt&); // assign AlignInt
to AlignInt
AlignInt operator=(const std::string& rhs); // assign string
to AlignInt
AlignInt operator+(const AlignInt& rhs); // concatinate

// Destructor
~AlignInt(); // clears mem on
close

// Other Methods
void loadfile(const std::string& filespec); // puts file
into object
bool hasData(){return haveMemory;} // has 'new'
been called by this object? If yes, clear memory upon destruction.
void clearend(); // since data is
'int-aligned' there may be trailing bytes that are not part of the
real data. This function clears them.
std::string str(); // returns data
as new std::string
std::string str(int offset, int chars_to_get);// similar to
std::string.substr()
void resizeEmpty(int byte_len); // erases old
data and makes a new resized
};

#endif //ALIGNINT

end------->Align_int_class.h

begin------->Align_int_class.cpp

#include "align_int_class.h"
#include "file_ok.h"

#include <iostream>
#include <fstream>

using namespace std;
//constructors
// default new empty
AlignInt::AlignInt()
:
haveMemory(false)
,wordsize(sizeof(*ul))
,len_c(0)
,len_l(0)
,ul(NULL)
,c(NULL)
{}

// new empty of defined length
AlignInt::AlignInt(int byte_len)
:
haveMemory(true)
,wordsize(sizeof(*ul))
,len_c(byte_len)
,len_l((len_c / wordsize) + ((len_c % wordsize) > 0))
,ul(new uint32_t[len_l])
,c(reinterpret_cast<char*>(ul))
{
memset(c, 0, len_l * wordsize );
}

//prealigned memory must be deallocated elsewhere
AlignInt::AlignInt(uint32_t* parray, unsigned int byte_len)
:
haveMemory(false)
,wordsize(sizeof(*ul))
,len_c(byte_len)
,len_l((len_c / wordsize) + ((len_c % wordsize) > 0))
,ul(parray)
,c(reinterpret_cast<char*>(ul))
{
clearend();
}

AlignInt::AlignInt(const string& stringin) // copy string
:
haveMemory(true)
,wordsize(sizeof(*ul))
,len_c(stringin.size())
,len_l((len_c / wordsize) + ((len_c % wordsize) > 0))
,ul(new uint32_t[len_l])
,c(reinterpret_cast<char*>(ul))
{

for(unsigned int j = 0; j < len_c; ++j){
c[j] = stringin[j];
}
clearend();
}

// Copy Constructors
AlignInt::AlignInt(const AlignInt& rhs) //copy constructor
:
haveMemory(true)
,wordsize(sizeof(*ul))
,len_c(rhs.len_c)
,len_l(rhs.len_l)
,ul(new uint32_t[rhs.len_l])
,c(reinterpret_cast<char*>(ul))
{
memcpy(c, rhs.c, rhs.len_c);
clearend();
}

AlignInt::AlignInt(const AlignInt& rhs1, const AlignInt& rhs2)
//overloaded copy-constructor, used to concatinate
:
haveMemory(true)
,wordsize(sizeof(*ul))
,len_c(rhs1.len_c + rhs2.len_c)
,len_l((len_c / wordsize) + ((len_c % wordsize) > 0))
,ul(new uint32_t[len_l])
,c(reinterpret_cast<char*>(ul))
{
memcpy(c, rhs1.c, rhs1.len_c);
memcpy((c + rhs1.len_c), rhs2.c, rhs2.len_c);
clearend();
}

// Assignment operator
AlignInt AlignInt:perator=(const AlignInt& rhs){
if(this != &rhs){
if(haveMemory){ //indicated memory has been allocated
memset(c, 0, len_l * wordsize);
delete[] ul;
}
len_c = rhs.len_c;
len_l = rhs.len_l;
ul = new uint32_t[rhs.len_l];
c = reinterpret_cast<char*>(ul);
haveMemory = true;
wordsize = sizeof(*ul);
memcpy(c, rhs.c, len_c);
clearend();
}
return *this;
}

AlignInt AlignInt:perator=(const string& rhs){

if(haveMemory){ //indicated memory has been allocated
memset(c, 0, len_l * wordsize);
delete[] ul;
}
len_c = rhs.size();
len_l = (len_c / wordsize) + ((len_c % wordsize) > 0);
ul = new uint32_t[len_l];
c = reinterpret_cast<char*>(ul);
haveMemory = true;
wordsize = sizeof(*ul);
for(unsigned int i = 0; i < len_c; ++i){
c = rhs;
}
clearend();
return *this;
}

AlignInt AlignInt:perator+(const AlignInt& rhs){
AlignInt temp(this->len_c + rhs.len_c);
memcpy(temp.c, this->c, this->len_c);
memcpy(temp.c + this->len_c, rhs.c, rhs.len_c);
temp.clearend();
return temp;
}

// Destructors
AlignInt::~AlignInt(){
if(haveMemory){
memset(c, 0, len_l * wordsize);
delete[] ul;
}
}

void AlignInt::loadfile(const string& filespec){
if(haveMemory){
cout << "ERROR...memory redesignation ERROR 1" << endl;
}else{
haveMemory = fileOK(filespec);
if(haveMemory){
ifstream in (filespec.c_str(), ios::in | ios::binary);
in.seekg(0, ios::end);
len_c = in.tellg(); // file length in bytes
len_l = (len_c / wordsize) + ((len_c % wordsize) > 0);
//^^^adds one if remainder^^^//
ul = new uint32_t[len_l];
ul[len_l - 1] = 0; // clear last word
c = reinterpret_cast<char*>(ul);
in.seekg(0, ios::beg);
in.read(c, (len_c));
in.close();
}else cout << "FILE ERROR: Cannot load: " << filespec << endl;
}
}

void AlignInt::clearend(){
memset(c + len_c, 0, (len_l * wordsize) - len_c);
}

string AlignInt::str(){
return string(c, len_c);
}

string AlignInt::str(int offset, int chars_to_get){
return string(c + offset, chars_to_get);
}

void AlignInt::resizeEmpty(int byte_len){
if(haveMemory){
memset(c, 0, len_l * wordsize);
delete[] ul;
}
haveMemory = true;
len_c = byte_len;
len_l = (len_c / wordsize) + ((len_c % wordsize) > 0);
ul = new uint32_t[len_l];
c = reinterpret_cast<char*>(ul);
memset(c, 0, len_l * wordsize);
}

end------->Align_int_class.cpp

 

[Peter van Merkerk]

Thanks Peter for the replies. These authors address the same issues I
was attempting to deal with. My understanding of the STL is too weak
to follow the advice for writing my own allocators for use with the
STL.

I'm not exactly a STL or template guru myself, but time spend learning
and understanding the standard container classes is time well spend. The
authors of that article were quite ambitious. But if you take away the
policy stuff and settle for simply clearing memory before deallocation
it becomes a lot simpler:

#include <vector>
#include <memory>
#include <cstring>


template <typename T,
class SecureAllocator : public BASE
{
public:
typedef typename BASE:ointer pointer;
typedef typename BASE::size_type size_type;

void deallocate(pointer p, size_type n)
{
// Fill memory with zero's before releasing it.
memset(p, 0, n * sizeof(T));
return BASE::deallocate(p, n);
}

template<typename U>
struct rebind {
typedef SecureAllocator<U> other;
};
};

typedef std::vector<int, SecureAllocator<int> > SecureIntVector;


int main()
{
SecureIntVector sv;

// Every time when the vector need to reallocate its
// buffer to accomodate new elements, the old buffer
// will be filled with zero's before is it deallocated.
for(int i = 0; i < 100; ++i)
{
sv.push_back(i);
}

// When the vector is destroyed the memory for the
// elements will be with zero's before is it deallocated.
}

The SecureAllocator should work with other container classes too. As far
as commenting your AlignInt class...keep in mind when reading from files
typically buffers are created behind your back which may hold part of
the file. Chances are that the buffers are *not* cleared before they are
deallocated.

There are probably many more things that could be said about the
AlignInt class...sorry, maybe some other time.

Regards