open() function quesiton

[blackdog]

I use open() function to open a existing file write data into it. And
the file is huge, approximately 200MB. Will it slow down the process if
it is a huge file? I believe it will slow down the
process, but would like to have confirmation from others.

Tks.

 

[Brian Wakem]

I use open() function to open a existing file write data into it. And
the file is huge, approximately 200MB. Will it slow down the process if
it is a huge file? I believe it will slow down the
process, but would like to have confirmation from others.

Tks.

Do you mean you wish to append to the file?

In which case you will pay a time penalty of a disk seek to the EOF, which
might take a few milliseconds. But the size of the file would be
irrelevant I think. It may depend on the filesystem itself though.

 

[blackdog]

Do you mean you wish to append to the file?

In which case you will pay a time penalty of a disk seek to the EOF, which
might take a few milliseconds. But the size of the file would be
irrelevant I think. It may depend on the filesystem itself though.

yes, I use open() function and append new data at the end of the file.
So the file will
keep growing.

 

[xhoster]

yes, I use open() function and append new data at the end of the file.
So the file will
keep growing.

It probably will not slow down (until you run low on disk) but try it
and see.

$ perl -l
use Time::HiRes qw(time);
use Statistics::Regression;
my $r=new Statistics::Regression(2,'',[qw/constant slope/]);
unlink "fooo";
foreach (1..1000) {
my $t=time;
open my $fh, ">>fooo" or die $!;
print $fh "x"x1_000_000;
close $fh or die $!;
$r->include(time()-$t,[1,$_]);
};
$r->print
__END__
****************************************************************
Regression 'with no name'
****************************************************************
Theta[0='constant']= 0.0336
Theta[1='slope']= 0.0000
R^2= 0.000, N= 1000
****************************************************************

So it takes 0.0336 seconds, regardless of how big the file is.

 

[Tintin]

I use open() function to open a existing file write data into it. And
the file is huge, approximately 200MB. Will it slow down the process if
it is a huge file? I believe it will slow down the
process, but would like to have confirmation from others.

200MB might have been considered huge a few years ago, but these days that's
pretty small.

Are you concerned that your script might take a few hundreds or thousands of
a second longer to process?

 

[blackdog]

200MB might have been considered huge a few years ago, but these days that's
pretty small.

Are you concerned that your script might take a few hundreds or thousands of
a second longer to process?

The perl script will open the data file every 2 minutes, in this case,
I guess it will
slow down the process. Do you think so. By the way, the data file keep
growing.

tks.

 

[Brian Wakem]

The perl script will open the data file every 2 minutes, in this case,
I guess it will
slow down the process. Do you think so. By the way, the data file keep
growing.

A single disk seek is needed to get to EOF. It does not matter how big the
file is.

 

[xhoster]

A single disk seek is needed to get to EOF. It does not matter how big
the file is.

That is not strictly true. The OS/FS may need to execute seeks on your
behalf to find out the location of the FS block which has the EOF. Now, if
it is the same file each time, that info may very well be cached, so those
extra disk seeks (probably not more than 5 anyways, depending on OS, FS,
and file size) may not be needed every time.

But I agree with you in principle, this overhead just isn't worth worrying
about.

Xho