Multidimensional array

[Paul E. Schoen]

But I tried that with no joy. So I added the following at the beginning
of
my script:

print "Param1=$ARGV[0]\n";
print "Param2=$ARGV[1]\n";

Using

perl EventProcessor.pl Full_Name="Paul E. Schoen" [email protected]

I got

Param1=Full_Name=Paul E. Schoen
[email protected]
List form of pipe open not implemented Line 46

46: open(MAIL, "|-", $mailprog, $recipient)

So, IMHO this is possibly a bug (or a feature) of the CGI getvars()
function.

Err... no, this is because the list form of pipe open isn't implemented
on Win32 (I'm not entirely sure why... ISTM it should be possible using
the same quoting code as list-exec). It is completely unconnected with
CGI.pm, and would still have happened if you hadn't been using it.

Thanks for the explanation about that error. But I'm talking about the CGI
module incorrectly reading command line parameters with quotes to take care
of spaces.

Paul

 

[Paul E. Schoen]

CGI.pm is working just fine, or would be if your script didn't exit due
to that error before it got anywhere.

Yes, it seems fine now. I don't know why I was getting the incorrect
parameter. Probably cranial flatus

Paul

 

[Paul E. Schoen]

Quoth "Paul E. Schoen" <[email protected]>:

DON'T DO THAT. You have a race condition between the check and the
create. Just create the table beforehand (either with perl or with the
sqlite command-line tool).

I've changed it to this, which works. It didn't work previously because the
Connect created a blank file. But I agree that it's best to create the table
outside the script.

BTW, I need different paths for the two servers. I can do it this way, but
for other places where I need to do something different depending on where
the script is located, is there some sort of conditional compile so I don't
get errors on unused code as I did when I used goto to skip over it?

my $dbfile = "../SCGBG/events.db"; #for Dreamhost
#my $dbfile="/home/jail.root/home/pstech/www/SCGBG/events.db"; #for
SmartNet

print fErrors "Connecting to $dbfile\n";

my $db = DBI->connect( # connect to your database, create if
needed
"dbi:SQLite:dbname=$dbfile", # DSN: dbi, driver, database file
"", # no user
"", # no password
{ RaiseError => 1, AutoCommit => 1 } # complain if something goes wrong
) or die $DBI::errstr;

#newly created file has size zero
if ((-e $dbfile) and not (-s $dbfile)) { #check for zero size
print fErrors "Creating table\n";
$db->do("CREATE TABLE tEvents (dt DATETIME KEY, tl TEXT, de TEXT)");
}
$db->do("PRAGMA foreign_keys = OFF");
$db->do("INSERT INTO tEvents (dt,tl,de) VALUES ('$E_DT', '$E_Title',
'$E_Descr')");

print fErrors "Generating query\n";
my $all = $db->selectall_arrayref("SELECT * FROM tEvents ORDER BY dt");

print fErrors "Printing events to $errorfile\n";
foreach my $row (@$all) {
($E_DT, $E_Title, $E_Descr) = @$row;
my @dt = split('T',$E_DT);
my $d = $dt[0];
my $t = $dt[1];
print fErrors "$E_Title\n$d $t\n$E_Descr\n";
}

Thanks,

Paul

 

[Paul E. Schoen]

How can you detect which server your code is on?

Can you test the hostname?

my $dbfile = qx/hostname/ =~ /dreamhost/i
? '../SCGBG/events.db'
: '/home/jail.root/home/pstech/www/SCGBG/events.db';


I suggest always using single quotes unless you _need_ one of
the two extra things that double quotes gives you (backslash
escapes or variable interpolation).

Thanks for the tip. Much of this is starting to make sense now. I could try
your suggestion about testing for the hostname, but if it only involves
changing some comments it may not be worth it. I also thought about putting
the file paths and other host-specific items in an "include" or "uses" file
(module) which could be customized for each installation.

But this was mostly for the case where I run the script on my local machine
where the environment is much different. For instance, the problem where I
got an error on:

open(MAIL, "|-", $mailprog, $recipient)

I tried using a goto to jump over the offending code, but it still produced
a compile error even though the interpreter would never see it. Maybe I
could set up a real localhost with the Apache server and make it very
similar to the remote host where I have set up my web page and CGI scripts.
But that's not an easy task, and so far it's been OK using the live server
for testing.

Would it be possible (and advisable) to make my own Perl module with code
that is proven to work on the server but not locally, such as the emailing
procedure from open(MAIL... to close(MAIL)? Perhaps make it a subroutine and
optionally call it from my main script? That way I could have a full
featured sendmail function in the modules I include on the servers, and a
dummy function in the one for testing on the local machine?

Just throwing some ideas. I can probably make something work, but my method
may not be the "right" way to do it. I want to avoid bad habits before they
become a problem.

Thanks,

Paul

 

[Paul E. Schoen]

So why aren't you doing that?

It's just a matter of convenience while I am developing this system. If I
want to change the structure of the table I just delete the $dbfile and run
the new script. And I'm not familiar with the SQlite command line tool. This
way, also, the script is self-documenting by having the database structure
in the table create code.

No, there isn't (well, you can fake up conditional compile with eval,
but it's rarely a good idea). You don't need to, though: an ordinary
runtime 'if' will do just fine. You can use the Sys::Hostname module to
get the current hostname to key off.


You're still not getting it. Say you start with a nonexistent file, and
then two requests come in very close together. They will (or, at least,
they might) execute like this:

FIRST REQUEST SECOND REQUEST

-s $dbfile # false
-s $dbfile # also false
$db->do("CREATE TABLE");
$db->do("CREATE TABLE");

Now you've got the second request trying to create a table which the
first request has already created. This will fail, and give you a fatal
error (since you asked for RaiseError => 1). You need to either put a
lock around the test-and-create, or find a form of create that will do
nothing quietly if the table already exists (and forget about the test),
or create the table beforehand and just have the script assume it's
already there.

This is probably due to my lack of experience with networked applications
where multiple users may run the script simultaneously. At least, that is
how I picture your scenario, and I can understand your concern if there was
any likelihood of heavy usage. But I would expect that this script would run
only several times a week, so there is little chance of such a conflict. But
I appreciate the heads up for good programming practice.

I have thought about possible problems if two requests came in at about the
same time, after the database and table have already been created. If the
second Connect request came while the database was already connected, I
would assume that it would simply ignore the request or perhaps connect with
a different user identity. But the problem might be when the first
Disconnect request was made, while the second user was still accessing the
database. If each database connection was uniquely identified, then both
processes might run concurrently. But I really don't know, and I think the
chances are minimal.

I have implemented a form of lock on the script at the start where I put the
10 second delay. I was going to put this lock around the entire script, but
it would fail if there were an error and the HTMLdie or other error exit
occurred. I don't know if there is anything like an "OnExit" event that
could be handled by unlocking the script. I am creating and deleting a
"lock.txt" file and checking for existence and displaying a "Busy, try again
later" message if a request is received while the first request was being
processed (mostly sleeping).

I appreciate the advice, and I'd like to learn all the correct ways to
program for this environment, but I think these concerns are not critical in
this situation. Please correct me if there is any major issue with what I am
doing. Thanks!

Paul

 

[Mart van de Wege]

my $dbfile = "../SCGBG/events.db"; #for Dreamhost
#my $dbfile="/home/jail.root/home/pstech/www/SCGBG/events.db"; #for
SmartNet

print fErrors "Connecting to $dbfile\n";

my $db = DBI->connect( # connect to your database, create if
needed
"dbi:SQLite:dbname=$dbfile", # DSN: dbi, driver, database file
"", # no user
"", # no password
{ RaiseError => 1, AutoCommit => 1 } # complain if something goes wrong
) or die $DBI::errstr;

#newly created file has size zero
if ((-e $dbfile) and not (-s $dbfile)) { #check for zero size
print fErrors "Creating table\n";
$db->do("CREATE TABLE tEvents (dt DATETIME KEY, tl TEXT, de TEXT)");
}

As Ben pointed out, you still have a race.

Think on what you are testing for. You want to know if the file is new,
in which case you need to create a table, right? Now, what happens if
you create a table in an existing SQLite db file?

The easy solution is to *not* test for file existence or size, but to
wrap the create statement in an eval{} block (Nota bene: an eval BLOCK!
see perldoc -f eval), and then test the error code after the block. If
that is the SQLite error for 'table already exists', then continue,
otherwise die using the original errorstring.

This, BTW, is a common error in DB programming, and not Perl specific. I
see it all the time when people want to know if a row exists before
inserting, they do a 'SELECT' beforehand. Unnecessary; just INSERT and
catch the error.

Mart

 

[Paul E. Schoen]

[snip: there is a race]

This is probably due to my lack of experience with networked applications
where multiple users may run the script simultaneously.

It isn't the networking aspect that causes a race.

It is probably due to your lack of experience with _multitasking_.

I have written real-time programs in Delphi where some processes run as
threads. But most of my code runs in the main application thread and is
event driven, where events may be user input or timer ticks. I have
encountered race conditions where I must wait for completion of a process
before continuing, and I have often used loops containing a delay and loop
count timeout, where the delay continues to process messages.

Since Perl is mostly an interpreted language, I would expect the
instructions to be executed sequentially and any functions that use IO would
have a built-in delay to allow completion before proceeding. I realize that
on a server there may be many IO routines running simultaneously in their
own time slices with cooperative or preemptive multi-tasking, but I would
assume that each application would run in its own memory space and all
variables and processes would be private to that user's application.

So I can't understand how a second request could collide with the first, and
the only danger may be multiple user threads accessing the same file. Since
the file in question is a database, I would assume it has some form of
allowing multiple users to have access at the same time. The race condition
described by Ben seems to be only at such a time as the database file has
been deleted in order to make changes to the structure, and that should be a
rare occurrence once I have this running properly.

You seek an "END block".

See the "BEGIN, UNITCHECK, CHECK, INIT and END" section in:

perldoc perlmod

OK. That's pretty cool. So much to learn

I think the best way to avoid collisions for the database is to use the
methods in "perldoc DB_File".

DB_File::Lock
An extremely lightweight DB_File wrapper that simply flocks a lockfile
before tie-ing the database and drops the lock after the untie. Allows one
to use the same lockfile for multiple databases to avoid deadlock problems,
if desired. Use for databases where updates *are* reads are quick and simple
flock locking semantics are enough.

BTW I think I found a wrong word. "are" should be "or".

Also, a friend suggested I use Delphi for PHP.

http://www.embarcadero.com/products/delphi-for-php

It's $300, but he has a copy I can use. However, he also sent me the
following comparisons, and even though I still haven't developed a love for
Perl, PHP sounds like an abomination (or is that ObamaNation?). Sorry,
couldn't resist

http://www.thesitewizard.com/archive/phpvscgi.shtml (Pro-Perl)
http://www.tnx.nl/php.html (Anti-Perl)

Actually, he recommends .NET, because of a large support community and third
party tools. But the same can be said for Perl. I am confident that I can
learn enough to be adequate for doing what I need and not dangerous, but
YOMV.

Thanks again,

Paul

 

[Jürgen Exner]

I have written real-time programs in Delphi where some processes run as
threads. But most of my code runs in the main application thread and is
event driven, where events may be user input or timer ticks. I have
encountered race conditions where I must wait for completion of a process
before continuing, and I have often used loops containing a delay and loop
count timeout, where the delay continues to process messages.

A race condition is something different than that.

Since Perl is mostly an interpreted language, I would expect the
instructions to be executed sequentially and any functions that use IO would
have a built-in delay to allow completion before proceeding.

True. But that applies only within a single instance of a given program
(and even then only if you don't fork or use threads)..

I realize that
on a server there may be many IO routines running simultaneously in their
own time slices with cooperative or preemptive multi-tasking, but I would
assume that each application would run in its own memory space and all
variables and processes would be private to that user's application.

Absolutely. However, if 20 users decide to make the same HTTP request at
the same time, then the web server _will_ (unless configured
differently) launch 20 instances of the corresponding CGI-script within
a fraction of a second to satisfy those 20 requests.

So I can't understand how a second request could collide with the first, and
the only danger may be multiple user threads accessing the same file. Since
the file in question is a database, I would assume it has some form of
allowing multiple users to have access at the same time.

That depends on the DB and the action you want to perform. Some are save
for concurrent access, some are not.

jue

 

[Paul E. Schoen]

http://padre.perlide.org/download.html (Free, partial debug, syntax
check)

(Note that I don't use it myself, but it has a good reputation.)

It seems that Padre requires installation of something called Strawberry
Perl. I'd rather not install yet another version of Perl, and I wonder why
that IDE needs a special version. I also found:

http://www.epic-ide.org/ (Free, OpenSource, includes debug,
etc.)
http://www.eclipse.org/
http://www.activestate.com/komodo-ide ($300 full featured)
http://www.activestate.com/komodo-edit (free, editor only, Perl,
Javascript, HTML, etc.)
http://packages.python.org/spyder/ (? not Perl but Python?)
http://www.darserman.com/Perl/Oasis/ (Editor and code browser, no
debug?)

I like the idea of an IDE, such as provided for Delphi. It would be very
good to have one that also handles HTML and JavaScript, especially if
controls, such as date pickers and calendars and specialized text entry for
date, time, URLs, and email addresses, would be available as drag and drop
objects for HTML forms. I don't know if that would be useful in Perl. If I
need a much more complex and highly featured CGI utility, maybe I'll look
into it.

I am now using PerlEdit http://www.indigostar.com/perledit.php on a 30 day
trial, and I like it, but for a full-featured version it's $49. Not bad, but
maybe Komodo is the way to go. And $300 for a full featured IDE including
JavaScript and other languages is not bad. If it saves me a few hours it
will pay for itself if I use it for commercial purposes. And it might be
tax-deductible if I use it for the Sierra Club website. You guys may also be
able to deduct your time helping me on this project!

Thanks,

Paul

 

[Paul E. Schoen]

I don't quite follow what you are talking about here, but if you mean
busy-waiting that's nearly always a bad idea.

Well, for instance, I send a command to a device which I have connected by
means of a USB port, which may be configured as a CDC Serial COM port or a
Custom device. In either case, there is an expected delay from the time of
sending the command to the time the device responds. And the response may be
a sequence of characters, or a continuous stream of characters which will
continue until a command to stop has been sent. And there may be times that
the command cannot be sent, or the response cannot be received, because the
USB connection is broken for various reasons.

So, I send the command, and then loop while waiting for a response. Other
processes may continue because in my delay loop I ProcessMessages(). I check
the status of the receive buffer and if nothing, or the wrong response, is
received at the end of the timeout I post an error message and break out of
that thread. This is in a single-user environment where the application
resides on a single computer connected to a single hardware device, although
other devices such as printers may be connected, and other applications may
be running. But the main communication is a separate thread which runs
continuously to process input and output.

What does *that* have to do with it? (Anyway, Perl isn't interpreted.
It's semi-compiled, in the same way Java (usually) is. Unlike Java
there's no (supported) way to save the bytecode out to a separate file,
and the language supports (and requires) eval, but the compilation step
is still important.) If you were to say 'Perl is an *imperative*
language' I might see your point.

I'm not sure what you mean. Many lines of the script might be precompiled
into machine language code to speed execution, but something like eval()
must either step through the code line by line or perform another
compilation every time it is encountered unless the code in the eval() has
not changed. I'm not planning to use eval(), and I don't intend to run any
data received from outside the script. Whether it runs as an interpreter on
the text of the script, or as snippets of machine code, it still processes
the script in a sequential manner (unless threads have been started, which
will not be the case unless they are generated by other instructions.)

Memory yes. Files no. We're talking about two consecutively-running
processes modifying the same file, so locking is an issue.

I'm not sure how two consecutively running processes can access the same
file unless they were started within whatever time is involved to create a
file. Otherwise, the existence of the file will notify the sender that the
script is busy.

No, you're not understanding how this works. Any decent web server is
handling many requests at the same time (whether it does so through
multi-processing, multi-threading or an event loop is up to the server
in question: Apache is usually multi-process on Unix systems). When a
request comes in that is handled by a CGI script, it fires off the CGI
but continues to process other requests while it waits for it to finish.
Thus you could well get two (or more) copies of the CGI running at the
same time.

Of course, that is true. And I have simulated that by having two instances
of the HTML form that issues the request in two browser windows, and I sent
them as close as possible to the same time. The first request was processed
normally, but the second request, (which of course used its own memory copy
of the script), encountered the existence of the lock file and sent back a

It does. One of the things it does *not* protect you against is checking
the size of the file (that is, going outside of the database interface)
and then assuming it won't change before you create the table. Read the
docs for the libraries you are using.

The existence of the file is checked first, and that should block an attempt
to read the file size.

Rare or no, you still need to handle it properly. If this is not
something you expect to happen often, it would be better done manually
without bothering the main script with it.

It's only a convenience for now. Doing it manually seems to be more trouble
than just deleting the file and running the script. I guess it depends on
how you define "better".

Err... *what*? What does DB_File have to do with *any* of this? You're
using DBD::SQLite (or, at least, you were in the last bit of code I
saw).

Yes, at one time I planned to use DB_File, but then I found that DBI was
better, and SQLite seemed easiest. I found the locking mechanism in a
document on Perl DBI and MySQL. But it seems to be in the context of
threads, although I think it could be used in the main code which may be
considered a thread.

The problems you are having with security have nothing to do with Perl.
They would be the same regardless of what language you were using. The
most important is that you are simply *not taking this seriously
enough*, and assuming you can learn enough to safely publish a program
on the Internet in a couple of evenings.

Are you still using full names as passwords?

Only temporarily. I plan to check the Email as well as a Password which will
be delivered by the HTML form using a Password type input control. But I
might call it something else so that anyone snooping the CGI vars won't see
"Password" as a variable name. I have even thought of other measures to
increase security, such as adding a generated string based on something like
the now() Date, which should match for the sender as well as the perl
script, as long as the user and server are in the same time zone and their
clocks are reasonably synched. I could also check other information passed
in the HTTP request.

This obsession with security is interesting and important, but I think it is
being overplayed in this case. I think I have mitigated the threat of email
flooding and server tie-up with a flood of requests. Only one in every ten
second time interval will actually be processed, and all others will result
in "busy" messages being returned to the offender (or to wherever s/he has
forged). There is no sensitive data, and in fact we'd be happy for a hacker
to get information about healthy activities with the Sierra Club as an
alternative to goofing off on a computer. If these requests continued for an
hour, there would just be 360 bogus events listed in an HTML file, or (when
I implement the Delete function), one or more events might be deleted. I can
just download copies every day or so as backup. And the delete request would
involve a dialog of some sort which may even request reentry of a password,
or a special password only for deleting.

Please tell me where I'm wrong. Your security concerns are valid in the
context of a truly public web page that may attract many users, and/or when
sensitive information, such as CC numbers, SSNs, and missile launch codes
are being accessed. This is a simple utility which is intended for easy and
well-organized entry of events, outings, and news items, to be displayed on
the website of a local group with several thousand members but only a dozen
or so who will be authorized to post new items. This should be pretty boring
to a hacker when there are so many other ripe opportunities for genuine
mayhem or useful information.

I have added more of an outline to my Perl script which defines what I
intend to do:

#!/usr/bin/perl -T
#
# EventProcessor.pl-- A program to:
# (1) Receive data from an HTML form EventSubmit.htm
# (2) Bounce multiple requests (10 second sleep)
# (3) Process the following data:
# (a) Full_Name
# (b) Email (sender's email address)
# (c) Password (6 or more characters)
# (d) Request_Code (Add or Replace or Delete)
# (e) Event_Title (text only)
# (f) Event_URL (optional web address to be displayed)
# (g) Event_DT (Date and Time in yyyy-mm-ddThh:mm:ss format)
# (h) Event_Description (Text with optional HTML tags)
# (4) Check for authorized user Email and Full_Name and Password
# (5) Check other data for reasonable content
# (6) Perform database request (SCGBG_Events.htm)
# (a) Add record to database, sorted by DateTime (non-primary key)
# (b) Replace record with matching DateTime and Title
# (c) Delete record with matching DateTime and Title
# (7) Send Event_ data back to sender (Failure or success)
# (8) Mail form data to my email address as notification (Failure or
success)
#
# From a script written in 1997 by James Marshall, (e-mail address removed)
# For the latest, see http://www.jmarshall.com/easy/cgi/
#
# Modified in September 2010 by Paul E. Schoen
# This version for use on Dreamhost

use warnings;
use strict;
use CGI 'Vars';
use DBI;
use Fcntl;
use DateTime;

Thanks,

Paul