Need help with proper file locking of flatfile data

[Randy]

Hello,

I seem to be having some difficulties locking a simple counter flatfile
database. To start off, here is the code I am presently using:

#!/usr/bin/perl
use Fcntl qwDEFAULT :flock);

open (LOCKERFILE, "<$activeuser/locker.txt") or die "Can't open: $!";
flock (LOCKERFILE, LOCK_EX) or die "Can't lock: $!";

open (GETCOUNT, "<$activeuser/counter.txt") or die "Can't open file: $!";
flock (GETCOUNT, LOCK_SH) or die "Can't lock: $!";
$currentcount=<GETCOUNT>;
($date,$count,$users)=split(/\|/,$currentcount);
close(GETCOUNT);

$count = $count + 1;

open (WRITECOUNT, ">$activeuser/counter.txt") or die "Can't open file: $!";
flock (WRITECOUNT, LOCK_SH) or die "Can't lock: $!";
print WRITECOUNT "$date|$count|$users";
close(WRITECOUNT);

close(LOCKERFILE);
exit;

I use the locker.txt file to start a primary lock before the reading and
writing begins, suggested to me by a user from this newsgroup long ago. The
script seems to work most of the time, but every now and then I check the
data and its completely gone ... only leaving the data separators | | | So I
guess it doesnt really work well after all.

I would be interested in knowing if this script is infact flawed in my
approach or if the data loss is being caused by something else. I have
really done lots of reading on locking flatfile databases and there seems to
be alot of opinions, but few hard facts on how to do it correctly.

This script performs a very simple function but is very important to me to
make it work right without losing my data. I would very much value feedback
on this. Many thanks in advance.

Robert

 

[John W. Krahn]

I seem to be having some difficulties locking a simple counter flatfile
database.

That is covered by the FAQ:

perldoc -q "I still don't get locking. I just want to increment the number in
the file. How can I do this"


John

 

[Randy]

That is covered by the FAQ:

perldoc -q "I still don't get locking. I just want to increment the number in
the file. How can I do this"

I have checked perldoc many times. perldoc has only limited information on
locking, and it's method is not adequate for high traffic flatfile database
locking. I have learned that a sentinel locking file is essential, which
perldoc does not cover.

Robert

 

[Mark Clements]

Hello,

I seem to be having some difficulties locking a simple counter
flatfile database. To start off, here is the code I am presently
using:

#!/usr/bin/perl
use Fcntl qwDEFAULT :flock);

You need to have

use strict;
use warnings;

Read the posting guidelines for why.

open (LOCKERFILE, "<$activeuser/locker.txt") or die "Can't open: $!";
flock (LOCKERFILE, LOCK_EX) or die "Can't lock: $!";

You are trying to get an exclusive lock on a file opened read-only.

I get:

bob 548 $ cat testflock.pl
#!/usr/local/bin/perl

use strict;
use warnings;

use Fcntl qwDEFAULT :flock);
my $filename = shift;
open IN,"<$filename" or die $!;
flock IN,LOCK_EX or die $!;

bob 549 $ perl testflock.pl testlock
Bad file number at testflock.pl line 9.

bob 550 $ ll testlock
-rw-r--r-- 1 mark cvs 0 May 25 07:13 testlock

The following comments are sort of academic because you are using an
external lockfile, but...

open (GETCOUNT, "<$activeuser/counter.txt") or die "Can't open file:
$!"; flock (GETCOUNT, LOCK_SH) or die "Can't lock: $!";
$currentcount=<GETCOUNT>;
($date,$count,$users)=split(/\|/,$currentcount);
close(GETCOUNT);

You have a race condition here: you have opened the file read-only with
a shared lock, and then you close it. It is possible that another
process will come and read the file in between this close and the next
open.

$count = $count + 1;

open (WRITECOUNT, ">$activeuser/counter.txt") or die "Can't open
file: $!"; flock (WRITECOUNT, LOCK_SH) or die "Can't lock: $!";
print WRITECOUNT "$date|$count|$users";
close(WRITECOUNT);

close(LOCKERFILE);
exit;

What happens if you duplicate the code in the perldoc that you were
pointed to? Are you running over NFS?

Mark

 

[Anno Siegel]

I have checked perldoc many times. perldoc has only limited information on
locking, and it's method is not adequate for high traffic flatfile database
locking. I have learned that a sentinel locking file is essential, which
perldoc does not cover.

Who told you so? It's a myth, and not even a plausible one. Even if
it were true, applying it at this stage is premature optimization.

Sentinel locking has nothig to do with traffic. If anything, it is
less efficient than straight file locking. You use it basically when
you want to protect a resource with flock semantics that isn't a (single,
local) file.

One example, as indicated, is locking of multiple files (the sentinel
can be one of the files or an extra one). Another is the database a
hash might be tied to. While the database technically is a file, Perl
doesn't treat it as such. In particular, you don't routinely get a
handle to the database file, so you can't lock it.

This is the kind of problem you solve with a lock on a sentinel file.
If that is not your case (it doesn't seem to be), just go with the
FAQ solution.

Anno

 

[xhoster]

Hello,

I seem to be having some difficulties locking a simple counter flatfile
database. To start off, here is the code I am presently using:

#!/usr/bin/perl
use Fcntl qwDEFAULT :flock);

open (LOCKERFILE, "<$activeuser/locker.txt") or die "Can't open: $!";
flock (LOCKERFILE, LOCK_EX) or die "Can't lock: $!";

open (GETCOUNT, "<$activeuser/counter.txt") or die "Can't open file: $!";
flock (GETCOUNT, LOCK_SH) or die "Can't lock: $!";
useless

$currentcount=<GETCOUNT>;
($date,$count,$users)=split(/\|/,$currentcount);
close(GETCOUNT);

$count = $count + 1;

open (WRITECOUNT, ">$activeuser/counter.txt") or die "Can't open file:
$!";
flock (WRITECOUNT, LOCK_SH) or die "Can't lock: $!";
useless

print WRITECOUNT "$date|$count|$users";
close(WRITECOUNT);

or die $!

close(LOCKERFILE);

or die $!

exit;

There is no point in flocking the counter.txt file, because access to it
is already controlled by the locker file. And if it weren't for that, it
would still be useless because you get shared rather than exlcusive locks.
And even if it werent' for that it would still be useless because there
is a race condition.

I use the locker.txt file to start a primary lock before the reading and
writing begins, suggested to me by a user from this newsgroup long ago.
The script seems to work most of the time, but every now and then I check
the data and its completely gone ... only leaving the data separators | |
| So I guess it doesnt really work well after all.

I would be interested in knowing if this script is infact flawed in my
approach or if the data loss is being caused by something else.

At a first glance, I don't see anything in the code that would plausibly
cause this problem without dying or triggering a warning. (If your script
was dying or triggering a warning sometimes, would you know?)

Maybe there is some rogue program that is accessing counter.txt without
going through the locking process. (Change the names of both files, and
see if the problem goes away). Or maybe your OS/FS has silently broken
locks.

I have
really done lots of reading on locking flatfile databases and there seems
to be alot of opinions, but few hard facts on how to do it correctly.

This script performs a very simple function but is very important to me
to make it work right without losing my data. I would very much value
feedback on this. Many thanks in advance.

Xho