memory pool?

[Uli Kunkel]

I have a situation where I'm getting a picture from a camera in small
chunks.
So the photo is a few MBytes and I have a callback function that returns
1KB until it finishes transfering the picture.
Any suggestions or examples how should I put a picture in one variable?
I heard something about using memory pools for this but I only need
something simple for this occasion.
Could I use CString?

Anyway thanks in advance for any suggestions.

 

[Pascal J. Bourguignon]

I have a situation where I'm getting a picture from a camera in small
chunks.
So the photo is a few MBytes and I have a callback function that
returns 1KB until it finishes transfering the picture.
Any suggestions or examples how should I put a picture in one variable?
I heard something about using memory pools for this but I only need
something simple for this occasion.
Could I use CString?

Not on comp.lang.c++. For CString, you should ask on comp.lang.c.

Anyway thanks in advance for any suggestions.

typedef unsigned char byte;
const int KByte=1024;
const int MByte=1024*KByte;
const int few=6;
std::vector<byte> image(few*MByte,0);

Then you can receive copy small buffers into it:


std::vector<byte>::iterator pos=image.begin();
std::vector<byte> buffer(1*KByte);

while(receive(&buffer)){
pos=std::copy(buffer.begin(),buffer.end(),pos);
}

 

[Juan Antonio Zaratiegui Vallecillo]

Uli Kunkel escribió:

I have a situation where I'm getting a picture from a camera in small
chunks.
So the photo is a few MBytes and I have a callback function that returns
1KB until it finishes transfering the picture.
Any suggestions or examples how should I put a picture in one variable?
I heard something about using memory pools for this but I only need
something simple for this occasion.
Could I use CString?

Anyway thanks in advance for any suggestions.

Use a vector, taking advantage from its reserve() function.

#include <vector>

(...)

std::vector<unsigned char> myPhoto;
myPhoto.reserve(1024*1024);

for (size_t part=0; part !=1024;++part) {
<read image data into &myPhoto(part*1024) >
}
(...)
<you now have your photo in myPhoto.size() sequential bytes at
&myPhoto[0]


Just customize such piece for your exact needs, substsituting all magic
constants with nice real constants.
This code skeleton takes as granted that char size is one byte, you may
need to correct the code it it is otherwise

best regards,

Zara

 

[Uli Kunkel]

Uli Kunkel escribió:

I have a situation where I'm getting a picture from a camera in small
chunks.
So the photo is a few MBytes and I have a callback function that
returns 1KB until it finishes transfering the picture.
Any suggestions or examples how should I put a picture in one variable?
I heard something about using memory pools for this but I only need
something simple for this occasion.
Could I use CString?

Anyway thanks in advance for any suggestions.

Use a vector, taking advantage from its reserve() function.

#include <vector>

(...)

std::vector<unsigned char> myPhoto;
myPhoto.reserve(1024*1024);

for (size_t part=0; part !=1024;++part) {
<read image data into &myPhoto(part*1024) >
}
(...)
<you now have your photo in myPhoto.size() sequential bytes at
&myPhoto[0]


Just customize such piece for your exact needs, substsituting all magic
constants with nice real constants.
This code skeleton takes as granted that char size is one byte, you may
need to correct the code it it is otherwise

best regards,

Zara

Thanks for the answers.
I'm wondering if I could use reserve to allocate a few MB and then if
the size is not enough that the vector grows automatically?
I read that vector resizing is very costly so I presume that reserving
just 1KB is not enough.

 

[peter koch]

Juan Antonio Zaratiegui Vallecillo wrote:

Uli Kunkel escribió:

Use a vector, taking advantage from its reserve() function.

#include <vector>

    std::vector<unsigned char> myPhoto;
    myPhoto.reserve(1024*1024);

    for (size_t part=0; part !=1024;++part) {
       <read image data into &myPhoto(part*1024) >
    }
(...)
    <you now have your photo in myPhoto.size() sequential bytes at
&myPhoto[0]

Just customize such piece for your exact needs, substsituting all magic
constants with nice real constants.
This code skeleton takes as granted that char size is one byte, you may
need to correct the code it it is otherwise

best regards,

Zara

Thanks for the answers.
I'm wondering if I could use reserve to allocate a few MB and then if
the size is not enough that the vector grows automatically?
I read that vector resizing is very costly so I presume that reserving
just 1KB is not enough.

std::vector resizes autmatically. As an alternative, you could also
use std::deque which memorywise is more optimal with regard to
reallocation but overall is slightly slower (does more allocations)
than a std::vector.

/Peter

 

[Uli Kunkel]

Not on comp.lang.c++. For CString, you should ask on comp.lang.c.




typedef unsigned char byte;
const int KByte=1024;
const int MByte=1024*KByte;
const int few=6;
std::vector<byte> image(few*MByte,0);

Then you can receive copy small buffers into it:


std::vector<byte>::iterator pos=image.begin();
std::vector<byte> buffer(1*KByte);

while(receive(&buffer)){
pos=std::copy(buffer.begin(),buffer.end(),pos);
}

I wrote some code and I have a problem that I can't open the picture.
Here is the simplified code:
----------------------------
//This part of the code pushes back every character to myPhoto vector

for(unsigned long i=0; i < Length; i++)
{
myPhoto.push_back(srcData);
}
if(complete) //Write myPhoto vector to a file
{
ofstream outf("C:\\test_0.jpg");
for ( pos=myPhoto.begin()
; pos!=myPhoto.end()
; ++pos)
{
outf << *pos;
}
outf.close();
}

//This part of the code append every chunk to the file
//It's for testing only
ofstream myFile ("C:\\test_1.jpg", ios:ut | ios::binary | ios::app);
myFile.write ((char *)pProgress->pbData, pProgress->lLength);
myFile.close();
-----------------------------
What happens is that the first photo has 8 or 9 KB more than the second
one and it can't be opened.
The second picture is ok but I don't want to do it this way.
Also I opened them and they seem the samo at the begining but the first
one is larger..

I tried 2 different ways to writing them but with the same effect.

Any suggestions?

 

[Rolf Magnus]

I wrote some code and I have a problem that I can't open the picture.
Here is the simplified code:
----------------------------
//This part of the code pushes back every character to myPhoto vector

for(unsigned long i=0; i < Length; i++)
{
myPhoto.push_back(srcData);
}
if(complete) //Write myPhoto vector to a file
{
ofstream outf("C:\\test_0.jpg");
for ( pos=myPhoto.begin()
; pos!=myPhoto.end()
; ++pos)
{
outf << *pos;
}
outf.close();
}

//This part of the code append every chunk to the file
//It's for testing only
ofstream myFile ("C:\\test_1.jpg", ios:ut | ios::binary | ios::app);
myFile.write ((char *)pProgress->pbData, pProgress->lLength);
myFile.close();

The first file is opened in text mode, while the second is opened in binary
mode. Depending on the platform you're using, that means that the first file
might get altered in some way while being written.

 

[Uli Kunkel]

I wrote some code and I have a problem that I can't open the picture.
Here is the simplified code:
----------------------------
//This part of the code pushes back every character to myPhoto vector

for(unsigned long i=0; i < Length; i++)
{
myPhoto.push_back(srcData);
}
if(complete) //Write myPhoto vector to a file
{
ofstream outf("C:\\test_0.jpg");
for ( pos=myPhoto.begin()
; pos!=myPhoto.end()
; ++pos)
{
outf << *pos;
}
outf.close();
}

//This part of the code append every chunk to the file
//It's for testing only
ofstream myFile ("C:\\test_1.jpg", ios:ut | ios::binary | ios::app);
myFile.write ((char *)pProgress->pbData, pProgress->lLength);
myFile.close();

The first file is opened in text mode, while the second is opened in binary
mode. Depending on the platform you're using, that means that the first file
might get altered in some way while being written.

That was the problem.
I didn't pay much attention on that part..
Thanks for the help.

 

[Juan Antonio Zaratiegui Vallecillo]

Juan Antonio Zaratiegui Vallecillo escribió:

Uli Kunkel escribió:


Use a vector, taking advantage from its reserve() function.

#include <vector>

(...)

std::vector<unsigned char> myPhoto;
myPhoto.reserve(1024*1024);

for (size_t part=0; part !=1024;++part) {
<read image data into &myPhoto(part*1024) >

ThankÅ› to all for the comments.

I really meant to say

}
(...)
<you now have your photo in myPhoto.size() sequential bytes at
&myPhoto[0]

(...)

And I keep on favouring the use of reserve, as it will be surely faster
than letting the container resize as needed, because we already know the
final size.

Best regards,

Zara

PS: Any error in this new message,, it is because it is too late in the
afternoon ;-)

 

[Uli Kunkel]

Juan Antonio Zaratiegui Vallecillo escribió:

Uli Kunkel escribió:


Use a vector, taking advantage from its reserve() function.

#include <vector>

(...)

std::vector<unsigned char> myPhoto;
myPhoto.reserve(1024*1024);

for (size_t part=0; part !=1024;++part) {
<read image data into &myPhoto(part*1024) >

ThankÅ› to all for the comments.

I really meant to say

}
(...)
<you now have your photo in myPhoto.size() sequential bytes at
&myPhoto[0]

(...)

And I keep on favouring the use of reserve, as it will be surely faster
than letting the container resize as needed, because we already know the
final size.

Best regards,

Zara

PS: Any error in this new message,, it is because it is too late in the
afternoon ;-)

In the end I did something like this:
First I reserved 2 MB. Then I check the capacyty and reserve a 1KB more
if needed:

if(myPhoto.size() + pProgress->lLength < myPhoto.capacity())
myPhoto.reserve(1024*1024);

Thank you all for your help.

 

[Default User]

Not on comp.lang.c++. For CString, you should ask on comp.lang.c.

No, he should not. CString is not a C construct, but rather an MFC
class.




Brian

 

[James Kanze]

Not on comp.lang.c++.  For CString,  you should ask on comp.lang.c.

You really want to see him flames, don't you?

Anytime you see a class whose name is prepended by a C, it's
probably something from MFC; it's a Microsoft convention,
anyway. So if he really wants to use CString for some reason, a
Microsoft group would be in order.

Of course, CString isn't any simpler than std::string.

    typedef unsigned char byte;
    const int KByte=1024;
    const int MByte=1024*KByte;
    const int few=6;
    std::vector<byte> image(few*MByte,0);

Then you can receive copy small buffers into it:

I'm not too sure what he's trying to do, but it sounds to me
that what he probably wants os for the callback function to
append to a vector. In which case, he could initialize the
vector with reserve() if he knew the size in advance (but for
relatively small buffers like a couple of meg, this is probably
overkill). His callback function just does push_back (or
insert() at the end).

 

[James Kanze]

Juan Antonio Zaratiegui Vallecillo wrote:
I'm wondering if I could use reserve to allocate a few MB and
then if the size is not enough that the vector grows
automatically? I read that vector resizing is very costly so
I presume that reserving just 1KB is not enough.

What you read was wrong. Resizing isn't that costly for
primitive object types (like unsigned char). There are other
reasons you might want to avoid it, but they probably don't
apply here.

 

[Uli Kunkel]

What you read was wrong. Resizing isn't that costly for
primitive object types (like unsigned char). There are other
reasons you might want to avoid it, but they probably don't
apply here.

--
James Kanze (GABI Software) email:[email protected]
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

I tried 2 things:

1. reserve 3MB and then push_back every unsigned char (or byte) to the
vector.
2. Reserve 1KB on every callback

U didn't measure the time precisely, but it is approximatly the same.
It takes very long to complete, around 10 seconds.
This is to much.I don't know what the problem is.

 

[peter koch]

I tried 2 things:

1. reserve 3MB and then push_back every unsigned char (or byte) to the
vector.
2. Reserve 1KB on every callback

That is the wrong approach. You either reserve once at the beginning
or you don't reserve at all. Reserving in small chunks will cause your
algorithm to run much slower than without any reserves at all.

U didn't measure the time precisely, but it is approximatly the same.
It takes very long to complete, around 10 seconds.
This is to much.I don't know what the problem is.

That is far to much - something else must be wrong. You can isolate
the vector code by simply appending "random data" to the vector
without reading it from the source. If that is not a snap, you should
post the codde here.

/Peter

 

[Uli Kunkel]

That is the wrong approach. You either reserve once at the beginning
or you don't reserve at all. Reserving in small chunks will cause your
algorithm to run much slower than without any reserves at all.

That is far to much - something else must be wrong. You can isolate
the vector code by simply appending "random data" to the vector
without reading it from the source. If that is not a snap, you should
post the codde here.

/Peter

I agree that it is better to allocate a couple of MB than to allocate
1KB so many times.
People on the group said that the latter isn't such a big deal.
Anyway the code is simple:
-----------------------------
//first I allocate 3MB in some initialization
myPhoto.reserve(3*1024*1024);

//then I assign one char at a time
for(unsigned long i=0; i < pProgress->lLength; i++)
{
myPhoto.push_back(pProgress->pbData);
}

 

[Uli Kunkel]

That is the wrong approach. You either reserve once at the beginning
or you don't reserve at all. Reserving in small chunks will cause your
algorithm to run much slower than without any reserves at all.

That is far to much - something else must be wrong. You can isolate
the vector code by simply appending "random data" to the vector
without reading it from the source. If that is not a snap, you should
post the codde here.

/Peter

I agree that it is better to allocate a couple of MB than to allocate
1KB so many times.
People on the group said that the latter isn't such a big deal.
Anyway the code is simple:
-----------------------------
//first I allocate 3MB in some initialization
myPhoto.reserve(3*1024*1024);

//then I assign one char at a time
for(unsigned long i=0; i < pProgress->lLength; i++)
{
myPhoto.push_back(pProgress->pbData);
}
-------------------------------

What do you think?
Maybe I should use insert, instead of putting one char at the time.
I tried this but pbData is unsigned char* type and could't do it...

I had one big oversight.
In the callback function I can define the length of each chunk.I thought
it is integer type.As I recall that is only 36635.
I check again and saw it is actually unsigned long.That was very stupid
of me.
Now I pass 1MB and it takes 2,3 seconds which is much better.

If I'm still doing something wrong I would appreciate any help.

 

[peter koch]

I agree that it is better to allocate a couple of MB than to allocate
1KB so many times.
People on the group said that the latter isn't such a big deal.
Anyway the code is simple:

//then I assign one char at a time
for(unsigned long i=0; i < pProgress->lLength; i++)
{
    myPhoto.push_back(pProgress->pbData);

No need to push_back each item individually. Better would be to use
std::vector.insert or std::copy with a back_inserter. Look these up.

I had one big oversight.
In the callback function I can define the length of each chunk.I thought
it is integer type.As I recall that is only 36635.

It is implementation defined, but i doubt max(int) is 36635 - it might
be 32767.

I check again and saw it is actually unsigned long.That was very stupid
of me.
Now I pass 1MB and it takes 2,3 seconds which is much better.

If I'm still doing something wrong I would appreciate any help.

See above.

/Peter

 

[Uli Kunkel]

peter koch wrote:
James Kanze wrote:
Juan Antonio Zaratiegui Vallecillo wrote:
I'm wondering if I could use reserve to allocate a few MB and
then if the size is not enough that the vector grows
automatically? I read that vector resizing is very costly so
I presume that reserving just 1KB is not enough.
What you read was wrong. Resizing isn't that costly for
primitive object types (like unsigned char). There are other
reasons you might want to avoid it, but they probably don't
apply here.
--
James Kanze (GABI Software) email:[email protected]
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
I tried 2 things:
1. reserve 3MB and then push_back every unsigned char (or byte) to the
vector.
2. Reserve 1KB on every callback
That is the wrong approach. You either reserve once at the beginning
or you don't reserve at all. Reserving in small chunks will cause your
algorithm to run much slower than without any reserves at all.
U didn't measure the time precisely, but it is approximatly the same.
It takes very long to complete, around 10 seconds.
This is to much.I don't know what the problem is.
That is far to much - something else must be wrong. You can isolate
the vector code by simply appending "random data" to the vector
without reading it from the source. If that is not a snap, you should
post the codde here.
/Peter
I agree that it is better to allocate a couple of MB than to allocate
1KB so many times.
People on the group said that the latter isn't such a big deal.
Anyway the code is simple:
-----------------------------
//first I allocate 3MB in some initialization
myPhoto.reserve(3*1024*1024);
//then I assign one char at a time
for(unsigned long i=0; i < pProgress->lLength; i++)
{
myPhoto.push_back(pProgress->pbData);

No need to push_back each item individually. Better would be to use
std::vector.insert or std::copy with a back_inserter. Look these up.

I had one big oversight.
In the callback function I can define the length of each chunk.I thought
it is integer type.As I recall that is only 36635.

It is implementation defined, but i doubt max(int) is 36635 - it might
be 32767.

I check again and saw it is actually unsigned long.That was very stupid
of me.
Now I pass 1MB and it takes 2,3 seconds which is much better.

If I'm still doing something wrong I would appreciate any help.

See above.

/Peter

I have a silly problem.
Insert takes const byte&. And variable pProgress->pbData that I pass is
unsigned char*.
Anyway it inserts an empty value, something is not right..

 

[Richard Herring]

[...]

I agree that it is better to allocate a couple of MB than to allocate
1KB so many times.
People on the group said that the latter isn't such a big deal.
Anyway the code is simple:
-----------------------------
//first I allocate 3MB in some initialization
myPhoto.reserve(3*1024*1024);
//then I assign one char at a time
for(unsigned long i=0; i < pProgress->lLength; i++)
{
myPhoto.push_back(pProgress->pbData);

No need to push_back each item individually. Better would be to use
std::vector.insert or std::copy with a back_inserter. Look these up.

}

[...]

See above.

I have a silly problem.
Insert takes const byte&. And variable pProgress->pbData that I pass is
unsigned char*.

Use the two-iterator form of insert:

myPhoto.insert(myPhoto.end(), pProgress->pBData, pProgress->pBData+pProgress->lLength);

Pointers are iterators.