My auction: Multiple people accessing the same text data file at the same time... Ouch!

[John Smith]

I created a program for a local on-line auction used in a charity
fundraiser.
In all, there were approximately 150 items, 700 users (people who
registered), and approximately 2000 bids during the one week that it was
live.

I used a tab-delimited text file for keeping track of bids.
Although the file contains 5 fields: (the item number, bid amount, date of
bid, username of bidder, and the number of bids on the item), let's keep
things simple and assume only 3 fields (the item number, bid amount, and the
username of the bidder).

The file would look something like this:
1 5.25 joe
2 6.15 sue
3 2.50 john
4 3.75 bob
Each field is separated by a tab. The actual file had 150 lines (1 line for
each item).

When someone submitted a bid, the program did 2 things:
a) Read the file and close it
b) Re-write the file with the new information and close it.

It worked great, until the last 10 minutes of the auction when the file got
corrupted. It was fortunate that I also kept a log of every bid so I didn't
lose anything.

What happened of course is that everyone waited until the last minute to
make their bid. There were approximately 500 bids in the last 15 minutes.

At first, I thought that if someone submitted a bid from the website, it
would go to the web server and my perl script would do what it had to and
then terminate. If someone else was bidding at the same time, the server
would have to wait until the perl script had finished the first request, and
then the server would call on the perl script again to do the second
request.

What I have to assume now is that many threads of the perl script could be
running at the same time. What I mean is that although the perl script
hadn't finished its first request, a second thread could be started, etc,
etc,... So basically multiple threads of the perl script could be running
simultaneously, and accessing that text data file at the same time.

I don't know if there's a way to limit the number of threads that the perl
script can run simultaneously (to just one). Or perhaps if it was possible
to set the file properties to disallow multiple access.

So now I'm thinking that perhaps this type of programming (using simple text
files as data files) has many disadvantages compared to using real data
files. Unfortunately, I'm not at that stage of linking Microsoft Data Base
files or whatever data files, I haven't graduated to that level yet.

Does anyone know a simple way to tackle this? Any ideas at all?

Thanks for all
G.Doucet

The following are parts of the routines that read, and re-write the data
file.
# Variables
# $item = number of the item
# $bida = bid amount
# $code = username of bidder
# $bidsfil = 'bids.txt' <--- the actual data file described above
# @bidsfile = content of entire file
# $bidsline = individual line (record) of file
# $bidsfield = individual field of record

if(open(FILE,"$bidsfil")) # Open the file for
reading
{ #
@bidsfile=<FILE>; # Read the entire file
close(FILE); # Close the file
} #




if(open(FILE,">$bidsfil")) # Open the file for
writing
{ #
$i=0; # set counter to zero
foreach $bidsline (@bidsfile) # Loop through every line
in the file
{ #
$i++; # increment counter
if($i==$item) # check if we're at the
correct item
{ #
@bidsfield=(); # push the contents of the
line into fields
push(@bidsfield,split(/\t/,$bidsline)); # get the current bid
$cbid = $bidsfield[1]; #
#
if($bida>$cbid) # if the bid is higher...
{ #
$bidsline="$i\t$bida\t$code\t\n"; # ...re-create the line
with new bid amount
} #
#
} #
print FILE $bidsline; # print line to the file
} #
close(FILE); # Close the file
} #

 

[William Herrera]

I created a program for a local on-line auction used in a charity
fundraiser.
In all, there were approximately 150 items, 700 users (people who
registered), and approximately 2000 bids during the one week that it was
live.

I used a tab-delimited text file for keeping track of bids.
Although the file contains 5 fields: (the item number, bid amount, date of
bid, username of bidder, and the number of bids on the item), let's keep
things simple and assume only 3 fields (the item number, bid amount, and the
username of the bidder).

The file would look something like this:
1 5.25 joe
2 6.15 sue
3 2.50 john
4 3.75 bob
Each field is separated by a tab. The actual file had 150 lines (1 line for
each item).

When someone submitted a bid, the program did 2 things:
a) Read the file and close it
b) Re-write the file with the new information and close it.

It worked great, until the last 10 minutes of the auction when the file got
corrupted. It was fortunate that I also kept a log of every bid so I didn't
lose anything.

You failed to lock the file. You need to use flock() to lock as shared when the
file(s) are read and as exclusive when the files are written. This means that
500 people trying to bid at the same time will not happen--last second bidders
will not have their bids made, most likely.

Large bidding sites actually use a timed transaction model--they get
submissions which are forwarded to the engine that updates the bidding file, so
that only one program actually updates the files on disk.

What happened of course is that everyone waited until the last minute to
make their bid. There were approximately 500 bids in the last 15 minutes.

At first, I thought that if someone submitted a bid from the website, it
would go to the web server and my perl script would do what it had to and
then terminate. If someone else was bidding at the same time, the server
would have to wait until the perl script had finished the first request, and
then the server would call on the perl script again to do the second
request.

What I have to assume now is that many threads of the perl script could be
running at the same time. What I mean is that although the perl script
hadn't finished its first request, a second thread could be started, etc,
etc,... So basically multiple threads of the perl script could be running
simultaneously, and accessing that text data file at the same time.

I don't know if there's a way to limit the number of threads that the perl
script can run simultaneously (to just one). Or perhaps if it was possible
to set the file properties to disallow multiple access.

So now I'm thinking that perhaps this type of programming (using simple text
files as data files) has many disadvantages compared to using real data
files. Unfortunately, I'm not at that stage of linking Microsoft Data Base
files or whatever data files, I haven't graduated to that level yet.

This is not a file type problem. It is a file locking problem. Direct database
file updates would have at least as many problems as text files here. A
separate transaction database server program is likely what would work best.

Of course, since you probably want to encourage people to bid earlier, maybe
you should just use text files and flock()? Of course, some higher bidders at
the end would be locked out. Another method is to announce in advance that the
bidding will go, say, one minute with no higher bids before the bidding closes,
and program the end of bidding time accordingly.