recording massive data stream

[Lorn]

i have a very large data stream, on the order of 1.5 million records an
hour (32 bytes per/record), which i would like to try and record to
disk. it doesn't need to be iserted into a db, so flat files or
binaries will probably be the end result. what would be the most
efficient way of acheiving this with c++? the data is arriving via the
internet on a broadband connection 3mbps and the hardware is an
unraided uata-133 with a 1.4ghz athlon and 512mb of ram. if any more
information is needed to base your input on, please let me know. any
help would be greatly appreciated.

many thanks,
lorn

 

[ciorga]

i have a very large data stream, on the order of 1.5 million records an
hour (32 bytes per/record), which i would like to try and record to
disk. it doesn't need to be iserted into a db, so flat files or
binaries will probably be the end result. what would be the most
efficient way of acheiving this with c++? the data is arriving via the
internet on a broadband connection 3mbps and the hardware is an
unraided uata-133 with a 1.4ghz athlon and 512mb of ram. if any more
information is needed to base your input on, please let me know. any
help would be greatly appreciated.

many thanks,
lorn

1500000*32 bytes ~= 45.7 MB/hour. This is a slow stream even for
slowest PC that you can find . I think you can use any method to
store this to disk. I guess it would be efficient enough to buffer the
data and append it to the disk file only once in a while.

 

[Lorn]

ok, thanks for your reply.

assuming a much more aggresive stream. 500-1000MB/hr., is a write
buffer still the most efficient method?

thanks

 

[Cristian Iorga]

ok, thanks for your reply.

assuming a much more aggresive stream. 500-1000MB/hr., is a write
buffer still the most efficient method?

thanks

Bigger the buffer, the less you would write to disk. But, on power
failure you could loose that buffer. So it's up to you to come with a
number that seems feet to your needs.

Also, you could chose to use a big buffer and flush it to disk ever X
seconds / minutes or when it's full. Maybe this would be efficient for
all the possibilities.

 

[msalters]

Lorn schreef:

ok, thanks for your reply.

assuming a much more aggresive stream. 500-1000MB/hr., is a write
buffer still the most efficient method?

Still no technical challenge, still no C++ subject
(what efficiency? power?battery? CPU? IO bytes/sec?)