Insert data in a file

[The Doctor]

Hey People,

I am writing an application, which has to insert and remove data from a
file. I have a quite big file (~500 mb), and I want to insert and remove
data from it. How do I do this without having to move huge blocks of data
manually? That would really cost a lot of time.

Grtz, The Doctor

 

[sean_in_raleigh]

Hey People,
I am writing an application, which has to insert and remove data from a
file. I have a quite big file (~500 mb), and I want to insert and remove
data from it. How do I do this without having to move huge blocks of data
manually? That would really cost a lot of time.

This is a question about handling files on your operating
system, not about C++, so it's going to depend on your
OS.

On the popular OSes, a file is just a stream of bytes, so
there's no way to do what you want natively. You'll
have to do something in the file data itself that indicates
whether that block is valid.

Sean

 

[Juha Nieminen]

I am writing an application, which has to insert and remove data from a
file. I have a quite big file (~500 mb), and I want to insert and remove
data from it. How do I do this without having to move huge blocks of data
manually? That would really cost a lot of time.

Your question is not very accurate. Where exactly do you want to
insert the data? Are you inserting it at the end of the file, or are you
also inserting it in the middle of it? Where are you removing the data from?

Even if you resorted to system-specific libraries, AFAIK no system
supports adding/removing data from the middle of a file without moving
all the data which comes after that location. (In fact, AFAIK no system
offers any function whatsoever to add or remove data from the middle of
a file, so you'll have to do it "manually".)

If what you want is to *replace* existing bytes in a file, without
modifying its size nor the location of the unmodified parts, then that's
a completely different issue, and can be implemented with standard C/C++
functions. (Basically you open the file for reading and writing, seek to
the position you want, and write the data there.)

 

[James Kanze]

If you want to move data inside the file to other location,
then you have to do it, nobody else is doing that for you
(except maybe DMA can do it faster? never tried...).

I would map the file in memory (partially at least) and worked
in that interface, but this depends on OS and is not on topic
here. In portable C/C++ you would have to open the file in
binary mode, seek around in the file, read some parts into a
buffer, seek to somewhere else, write them out there, etc.

Even that doesn't really work. You can't insert data into a
file, nor remove it. Period. You can only overwrite it.

Note that C++ stream abstraction is probably just nuisance
here, you might want to work directly on C level (FILE*
interface).

There's no difference with regards to what you can do with FILE*
or with an fstream; the same constraints hold. For that matter,
the fact that you can't insert or remove data also holds at the
system level (int or HANDLE as a file descriptor) under Unix or
Windows.

 

[James Kanze]

Your question is not very accurate. Where exactly do you want
to insert the data? Are you inserting it at the end of the
file, or are you also inserting it in the middle of it? Where
are you removing the data from?

Even if you resorted to system-specific libraries, AFAIK no
system supports adding/removing data from the middle of a file
without moving all the data which comes after that location.
(In fact, AFAIK no system offers any function whatsoever to
add or remove data from the middle of a file, so you'll have
to do it "manually".)

Sure they do. Not all, but most (although they do introduce
restrictions---things like fixed length records). In fact, the
only two I've seen that don't support it are Unix and Windows
(and it's predecessors).

If what you want is to *replace* existing bytes in a file,
without modifying its size nor the location of the unmodified
parts, then that's a completely different issue, and can be
implemented with standard C/C++ functions. (Basically you open
the file for reading and writing, seek to the position you
want, and write the data there.)

Being sure to open it in binary mode (otherwise, you can't seek
to a random location).

 

[vlado]

InsertLib

It is possible to insert into or delete data from file using InsertLib (http://insertlib.com). It is very fast and simple way how to perform these non-standard operations on common file systems. I have been using this c++ library many times.