Multidimensional array

[Dr.Ruud]

my $E_Date = $in{'Event_Date'}; #Date in yyyymmdd format
my $E_Time = $in{'Event_Time'}; #Time in hh:mm format
my $E_Title = $in{'Event_Title'};
my $E_Descr = $in{'Event_Description'}; #Description may have newlines

Why copy these to new variables?

for my $row ( @$all ) {
( $E_DT, $E_Title, $E_Descr ) = @$row;
my @dt = split "T", $E_DT;
my $d = $dt[ 0 ];
my $t = $dt[ 1 ];

No need to set up @dt first, you could just do

my ( $d, $t ) = split "T", $E_DT;

Or forget about $d ad $t, and just use the @dt.

print "$E_Title\n$d $t\n$E_Descr\n";

printf "%s\n%s %s\n%s\n", $E_Title, @dt, $E_Descr;

 

[Dr.Ruud]

Uh, in what way is that better?

It was in the context of getting rid of the separate $d and $t.

(So I was tempted to write the @dt as @dt[0,1].

local ($", $,, $\) = (" ", "\n", "\n");
print $E_Title, "@dt", $E_Descr;

Or, you know, use a heredoc.

Yes, many alternatives available.

 

[Steve]

$db->do("INSERT INTO tEvents (dt,tl,de) VALUES ('$E_DT', '$E_Title',
'$E_Descr')");

You may be setting yourself up for an SQL Injection Attack since it
appears you are working with unchecked user input.

This is a *very* serious security hole.

If you do not know what an SQL Injection Attack is, you need to google
the term and do some reading.

It is relatively easy to solve the problem, especially with DBI, but you
*must* discover and implement the solution yourself so you will
(hopefully) understand what it is you are doing why you are doing it.


hth,

\s

 

[ccc31807]

In the context of this thread obviously we are discussing CGI
development.

So you were not working on the live server application, were you?

I did. I explicitely said live server.

No he isn't. He is interpreting my words exactly as I intended them and
exactly as is customary in the relevant industry.
If you are not familiar with those terms then that is your problem, not
mine or Sherm's.

I have several stand alone machines where I have Apache and a DB
server running. I have a small (4 machines) home network on which I
have Apache and a DB server running. At work, I have both an internal
network with Apache and a DB server, and an extranet with the same
thing. I also have a personal server that runs around 60 production
sites.

I develop CGI scripts on all of these machines. Typically, I will
develop on a standalone machine, and then upload the directory tree to
a server and configure httpd.conf to serve it, using the local code as
the 'source' and the remote code as a 'backup.' However, from time to
time I find myself developing directly on production servers for one
reason or another.

I don't do contract work for web sites (and do very little in any
case) but I do a fair amount of database interfaces via CGI at work,
and maintain a web/database server for a group. I got my start as a
web/database/system administrator (on BSD) in 1999.

Your statement that "it is quite stupid to do any development work on
a live web server" struck me as reasonable in general but as an
overstatement. If you do 'development work' you HAVE to use a 'live
web server.' That doesn't mean that you screw around with production
code, although sometimes you will, either by necessity or by
convenience or sometimes by sheer stupidity.

Words have meanings, and we need to pay attention to the clear and
plain meanings of words. If we want to communicate with others, we
can't insist on idiosyncratic usage, but need to use words in their
generally understood meaning. If I'm guilty of anything, it's a
reluctance to use jargon and a preference for common usage as opposed
to technical usage.

CC.

 

[Jim Gibson]

???
You did read the section "DEBUGGING" in "perldoc CGI", didn't you?

As far as I can tell, Paul is not using the CGI module.

 

[Uri Guttman]

c> Your statement that "it is quite stupid to do any development work on
c> a live web server" struck me as reasonable in general but as an
c> overstatement. If you do 'development work' you HAVE to use a 'live
c> web server.' That doesn't mean that you screw around with production
c> code, although sometimes you will, either by necessity or by
c> convenience or sometimes by sheer stupidity.

c> Words have meanings, and we need to pay attention to the clear and
c> plain meanings of words. If we want to communicate with others, we
c> can't insist on idiosyncratic usage, but need to use words in their
c> generally understood meaning. If I'm guilty of anything, it's a
c> reluctance to use jargon and a preference for common usage as opposed
c> to technical usage.

as other said, technical jargon is real and should be used to
communicate with technical people. try talking natural english to a
lawyer and watch you lose all your lawsuits. live means production, not
just running. ALL servers are live by your silly definition. live means
exposed to the real ugly dirty nasty cracking masses out there. this
could be internal to your shop or on the net, it is still live. live
means it does actual production work and it can't be broken or played
with.

uri

 

[Paul E. Schoen]

Despite being shown how it can be a drop-in replacement for
what he is using with only 2 lines of code.

If Paul feels free to ignore the advice given here, then it will likely
become the case that folks will feel free to ignore his posts
altogether...

You reap what you sow.

I'm not intentionally ignoring your advice, but I am trying to achieve a
working level of proficiency and understanding, which is mostly having
something that does most of what I aim to do. Today I tried using command
line parameters as well as using "< textfile" to simulate the HTTP request
from the HTML form. But I could not get it to work on my local machine or on
the Smart.net host.

Then I added a debug file that would be written to at several key places in
the script. This helped me to see that the DBI module seemed to be lacking
on the remote host, as well as the DateTime module. So I had one problem
running the script on my local machine, and another on the remote host. I
tried to contact the Smart.net administrator, but no reply yet.

I had an email reminder that my free 30 day Dreamhost account was about to
expire and it was set for autorenewal on my credit card which was about $35
for a year. So I uploaded the local website to that host, which required a
few changes to the EventProcessor.pl script, and now it seems to be working
perfectly. I had some security issues but I used chmod 711 for the cgi-bin
and now I don't get the directory of files as I had previously.

The Dreamhost forum was very helpful in finding the problem with the
location of sendmail, and the reply to the OP also showed a security
problem: http://discussion.dreamhost.com/thread-126568.html. I did not
realize that user-provided data could be used for exploit by spammers, but
the fix provided seems simple enough.

Thanks for being patient and providing much needed help.

Paul

 

[Jürgen Exner]

Your statement that "it is quite stupid to do any development work on
a live web server" struck me as reasonable in general but as an
overstatement. If you do 'development work' you HAVE to use a 'live
web server.'

For eventual deployment obviously yes, for development most certainly
not.

That doesn't mean that you screw around with production
code,

So now you are saying you shouldn't develop on a live server? Maybe you
can make up your mind?

although sometimes you will, either by necessity or by
convenience or sometimes by sheer stupidity.

Not arguing those exceptions.

Words have meanings, and we need to pay attention to the clear and
plain meanings of words. If we want to communicate with others, we
can't insist on idiosyncratic usage, but need to use words in their
generally understood meaning.

Right. Then why don't you do that? "Live server" has a well established
and generally understood meaning in the relevant industry.
Or are you looking for scissors, needle, and thread, too, when your boss
asks you to move the "button" 2 cm to the right ?

If I'm guilty of anything, it's a
reluctance to use jargon and a preference for common usage as opposed
to technical usage.

It has nothing to do with jargon, but with established industry-wide
terminology.
If you don't get the difference between a live server and an alive
server, then that is certainly not my problem.

*PLONK*

jue

 

[Peter J. Holzer]

Absolutely. Developing on a test server has significant advantages over
developing on a live server:
- If you make a mistake you are taking down only the test server, not
the live server with maybe hundreds of thousands of customers
- It is far less likely that crackers can break into the system due to
security holes in the unfinished code, because the system is not open to
the public
- Even if they do manage to break in the rewards are meager: they only
get access to mockup data instead of PII, financial, confidential and
other critical customer data
- You can try your code without risking to majorly screw up your live
customer data
- You can try unusual and rare data combinations without messing up your
live data, e.g. have the fictitious visiting business man in Bangkok
enter his name in Hangul and his current location in full lenght Thai
(http://en.wikipedia.org/wiki/Bangkok#Full_name).

Still, it has some serious drawbacks:
- the test team will be upset because you are changing their test
environment constantly. Therefore there is no way to create a reliable
repro scenario

It's still better if your test team is upset than if your
customers/users are upset because you are changing their work
environment constantly .

If a constantly changing (due to development) testing environment is a
problem, you probably want to add another layer, not subtract one. So
you would have a development system, a test system and a production
system.

- management will be upset because the vast majority of bugs found will
be of the kind "Server Error 500: Syntax error in line 42", which could
have been found much much cheaper by running "perl -c" instead of having
a human actually call the site in a browser.

How would a test server prevent me from invoking "perl -c"? Especially
compared to a production server which often has a much more restrictive
environment?

- everyone will be upset because they are blocked from doing anything
until you fix that syntax error. And even worse, nobody knows if you are
actively working on a fix, are taking an urgent leak, or have gone home
for a 2 week vacation.

Again, that's an advantage of the test server, not a disadvantage. If
your test server is down because of a syntax error "everyone" is just
the test team (and possibly your co-developers). If that happens on your
production system, "everyone" is all your users.


For me the most important drawback of a test server is that it isn't
identical to the live server:

* often you don't have the same data, only mockups.
* real users use the system differently than testers
* the load is (usually) smaller
* hard- and software may be different

So it may be difficult to reproduce problems on the test system.

hp

 

[Martijn Lievaart]

If a constantly changing (due to development) testing environment is a
problem, you probably want to add another layer, not subtract one. So
you would have a development system, a test system and a production
system.

But then the people doing testing will change the test server! That's not
acceptable to the people doing acceptance. So lets another layer and add
an acceptance server. Let's call it Development-Test-Acceptance-
Production (DTAP). What a novel concept!

M4

 

[Paul E. Schoen]

Replace the 2 lines that I showed you, and your working program
(with the hokey library) should continue to work (with CGI.pm).

I did that, and it works very well. I did not really understand the package
concept, but it is really very much like the USES statement in Delphi. But I
didn't realize that the CGI package had the same function as the code in my
script for handling the vars.

With CGI.pm you would test it from the command line thus:

someprog name=Paul email='(e-mail address removed)'

Simple!

I was able to get that to work on my local machine, but there is a Full_Name
variable that has spaces in it, and I did not find any way to use it in
command line parameters. It. is important because I am using that as a form
of password. But I could change it for testing

^^^
^^^

Past tense?

I am quite certain that you are being overly optimistic there.

The code you showed some time ago was vulnerable to a Denial Of
Service attack. Have you mitigated that one somehow?

I'm not sure how it is vulnerable to a DOS attack, or if that is likely to
occur. This form will be accessed by only a few Sierra Club members, and the
correct Full_Name must be provided in order to proceed. Otherwise it returns
"Unauthorized User". If the correct name is given, then the event
information is verified for reasonable values (in the Submission Form's
JavaScript), before it will make the HTTP request to the CGI script.

Of course anyone could look at the HTML source and find the URL of the perl
script so it could be accessed from another site. Perhaps the perl script
with the passwords in the cgi-bin could be examined and exploited in some
way, but it would just send emails to me indicating new events have been
posted, and the email includes the website URL where it originated (which
should be my hosts). The Email provided is only for further identification
of the senders.

You will be cracked. I see it coming...

Perl has a feature called "taint checking" that can help mitigate
that risk:

perldoc perlsec

I am concerned about security and I'd like to know if this is really likely
to be subject to a malicious and debilitating attack, and find ways to
mitigate the damage or protect it. I'll look into the "taint checking"

The fix is ass backwards.

It specifies "bad" characters.

The proper approach is to specify "good" characters.

That is, you don't try and idenfify what might be harmful (what if you
miss one?) you instead specify what you will allow.

Since you are doing something with mail, you have surely searched
the Perl FAQ for that term

perldoc -q mail

and have seen the answer to

How do I check a valid mail address?

I trust.


If not, then use the FAQ Luke!

I can see that there is a huge amount of information to assimilate and this
may be more of a challenge than I want to tackle. But I'm learning a lot and
it's actually fun when I make progress. Sometimes I just don't know what to
look for or where to find it. I generally check the FAQ and do a web search,
but sometimes the resources are not very good, as was the case with this
simple mailer that I used and modified.

Now I have been working on the HTML submission form and JavaScript to make
sure the data is reasonable. I suppose it's best to do that at that level so
that the events will be consistently formatted. I added a simple date picker
that works pretty well, and I also found a simple calendar, but a fancier
version was just too complex. And of course that is OT for this NG, but it
helps to see the big picture from the input provided by the user to the
final usage in the website.

I appreciate the help, and I understand your frustration with my bumbling
attempts at getting this working.

Paul

 

[Mart van de Wege]

I am concerned about security and I'd like to know if this is really
likely to be subject to a malicious and debilitating attack, and find
ways to mitigate the damage or protect it. I'll look into the "taint
checking"

Let me don my professional hat here (I'm a security guy): yes, any code
that handles outside data is subject to attack.

Perl's taint mode is brilliant because it forces you to validate all
external inputs before processing. If you don't do any checking on
external inputs, your code will die.

Of course, taint mode will not stop you from writing stupid input
checks, but at least it will force you to think about input validation.

Mart

 

[John Mason Jr]

I did that, and it works very well. I did not really understand the
package concept, but it is really very much like the USES statement in
Delphi. But I didn't realize that the CGI package had the same function
as the code in my script for handling the vars.


I was able to get that to work on my local machine, but there is a
Full_Name variable that has spaces in it, and I did not find any way to
use it in command line parameters. It. is important because I am using
that as a form of password. But I could change it for testing



I'm not sure how it is vulnerable to a DOS attack, or if that is likely
to occur. This form will be accessed by only a few Sierra Club members,
and the correct Full_Name must be provided in order to proceed.
Otherwise it returns "Unauthorized User". If the correct name is given,
then the event information is verified for reasonable values (in the
Submission Form's JavaScript), before it will make the HTTP request to
the CGI script.

Of course anyone could look at the HTML source and find the URL of the
perl script so it could be accessed from another site. Perhaps the perl
script with the passwords in the cgi-bin could be examined and exploited
in some way, but it would just send emails to me indicating new events
have been posted, and the email includes the website URL where it
originated (which should be my hosts). The Email provided is only for
further identification of the senders.



I am concerned about security and I'd like to know if this is really
likely to be subject to a malicious and debilitating attack, and find
ways to mitigate the damage or protect it. I'll look into the "taint
checking"

Another good resource for web app security
http://www.owasp.org/index.php/Main_Page

and to get an idea of the problems you can face

http://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project





John

 

[Jürgen Exner]

It's still better if your test team is upset than if your
customers/users are upset because you are changing their work
environment constantly .

If a constantly changing (due to development) testing environment is a
problem, you probably want to add another layer, not subtract one. So
you would have a development system, a test system and a production
system.

Congratulations, you just won 100 points.

How would a test server prevent me from invoking "perl -c"? Especially
compared to a production server which often has a much more restrictive
environment?

Nowhere did I say it prevents you from doing so. But _if_ you are
developing on the test server then the damage is already done the moment
you hit "Save" in your favourite editor, i.e. _before_ you even get a
chance to run perl -c.
Although certainly better than developing on the live server, test
servers are still not a good development environment. That's why you
have a development environment on your own local computer where you can
screw around and experiment to your hearts desire without affecting
anyone else when things go wrong.

Again, that's an advantage of the test server, not a disadvantage. If
your test server is down because of a syntax error "everyone" is just
the test team (and possibly your co-developers). If that happens on your
production system, "everyone" is all your users.

Congratulations, you just won another 100 points. Keep up the good work
and you may reach 500 soon.

For me the most important drawback of a test server is that it isn't
identical to the live server:

Unfortunately that is quite true. While you can try to make them as
similar as possible more often than not this is impossible.

* often you don't have the same data, only mockups.

Which on the other hand can be a good thing, too. It not only protects
potentially highly sensitive real data from theft and damage (where
there is no real data it cannot be stolen or damaged) but also allows
you to test data configurations that you couldn't use in the live
service.

* real users use the system differently than testers
* the load is (usually) smaller

True and true. But can be mitigated to some extend by carefully
designing the test plan and running load simulations. Still, you are
right, it is not the real deal.

* hard- and software may be different

This is IMO the main problem. Even if you do invest in a large set of
identical hardware(*) there will always be some differences. That's one
reason why you want to keep the software suite running on a server as
minimal as possible. The smaller it is the easier it is to keep it
synchronized between different machines.
Unfortunately this goes only that far, e.g. the test team needs their
own test tools on their test servers. Or production wants to run their
own service monitoring tools on live servers. Which in turn you could
run on the test servers, too, but then you would have to simulate the
same load on the monitoring tools as in production, etc, etc, etc.
So yeah, at some point you will have to decide how similar is similar
enough.

*: and in the business world there are even contracts which guarantee
that identical hardware will be available for x years to come for
replacements and expansion of the system. Very expensive!

So it may be difficult to reproduce problems on the test system.

Oh so true....

jue

 

[Paul E. Schoen]

^ ^
^ ^

Some shell quoting there. Waddya know?




That is a problem with your lack of understanding of your very
own shell, not a Perl or CGI.pm problem.

Simply learn how to quote command line arguments in whatever shell
you are using.


Full_Name="Paul Schoen"
or even:
"Full_Name=Paul Schoen"

These produced an error of "List form of pipe not implemented Line 42.

Using single quotes 'Paul Schoen' causes the script to reply

Unauthorized user: 'Paul

So it is on some sort of internal network rather that the
World Wide Web then?

It if it accessable on the WWW, then you do not really know
who will be accessing it.

It can be accessed by pretty much anybody in the World.

This is set up as a www application, but it is intended for only authorized
users.

You must validate it *again* on the server.

It is trivally easy to circumvent browser-side (Javascript) validation.

Yes. The browser side validation is only to assure consistent entry of data
by the user.

It may not be a browser that is making the HTTP request.

Pretty much any program can make an HTTP request, programs known
as "browsers" are examples of only one such program.

OK. I meant "in normal use".

And if they call that Perl program 10,000 times a second, what happens?

Your server gets so slow at to be unusable.

Voila! You have been Denied the Service of that web server.

I had put a sleep(nSec) line in the perl script at one time. I did that so
the script would delay from the time it created or connected to the database
file before trying to access the table. Perhaps I could use that to limit
the rate at which the program would respond to HTTP requests. And I could
also set a variable at the start of the script which would bounce any
additional requests until the first has finished processing. Would that
work? I haven't read much of the documentation about security issues. Yet.

at a rate of 10,000 times a second...

With the sleep() and the ignore request system this could be made
manageable, and perhaps the script can identify multiple requests from the
same source and blacklist it. But again, I need to read (and understand) the
documentation.

Also, perhaps I can make the directory where the HTML form is located
accessible only with a password. That will make it harder for a hacker to
see the HTML and the URL for the CGI script. I can also rename the Perl
script to a random string of characters, and abandon the one I'm now using
(which has been exposed already on this NG).

If it is accessable on the WWW, then absolutely yes!




You must also make sure the data is reasonable by checking
it again *on the server*.

Absolutely. The JavaScript is just for the legitimate user.

You hang in there and we'll hang in there.

It ain't rocket science, (it's about one degree less complicated though

There is a lot to learn, and the Perl language is just difficult for me to
grok. That's why I have used lots of comments in my script that seem
superfluous to those who are well versed in the language. I've only been
seriously coding in Perl for a little less than two weeks, and much of that
time was spent on problems with the server setup and basic understanding of
server side scripting and issues such as non-unix newlines.

I'm doing this project as a volunteer, and it would probably be enough for
me to simply receive emails from our members and then just format the
information in HTML and upload it to the server, without using CGI at all. I
could make a local script that would save me the work of formatting the HTML
using a text editor or an HTML optimized editor such as OpenOffice. The
former takes more time but the HTML is easier to read. The latter is fast
and gives quick WYSIWYG, but the HTML is horrible and bloated.

I think I'm close to being able to make this "live". Thanks.

Paul

 

[Scott Bryce]

These produced an error of "List form of pipe not implemented Line 42.

Using single quotes 'Paul Schoen' causes the script to reply

Unauthorized user: 'Paul

What happens if you use:

Full_Name=Paul%20Schoen

 

[Uri Guttman]

PES> These produced an error of "List form of pipe not implemented Line 42.

PES> Using single quotes 'Paul Schoen' causes the script to reply

PES> Unauthorized user: 'Paul

learn some shell stuff too. single quotes don't to ANYTHING on winblows
shells. unix shells allow both single or double quotes. did you actually
try the double quoted first version? it should work fine on
winblows. the second form is suspect IMO but i am not sure.

uri

 

[Paul E. Schoen]

What happens if you use:

Full_Name=Paul%20Schoen

List form of pipe not implemented.

I had tried that before with similar results. I'm using the command line in
Windoes Vista Business.

"Thanks for playing"

Paul

 

[Paul E. Schoen]

PES> These produced an error of "List form of pipe not implemented Line
42.

PES> Using single quotes 'Paul Schoen' causes the script to reply

PES> Unauthorized user: 'Paul

learn some shell stuff too. single quotes don't to ANYTHING on winblows
shells. unix shells allow both single or double quotes. did you actually
try the double quoted first version? it should work fine on
winblows. the second form is suspect IMO but i am not sure.

Yes, I tried that first. My actual full command line is:

perl EventProcessor.pl Full_Name="Paul E. Schoen"

I also tried using

set /p myname= Paul E. Schoen
perl EventProcessor.pl Full_Name=%myname%

but it returned

Unauthorized user: %myname%

On http://www.perlmonks.org/?node_id=605706 it says that each application
does its own command line parsing so this appears to be a problem with the
CGI module.

I found this at http://www.stonehenge.com/merlyn/UnixReview/col57.html:

If we want a space within one of the values, we need to use shell quoting
rules:

perl my-script 'arg1a arg1b' arg2

This passes two arguments now, not three. We get the same result with:

perl my-script arg1a\ arg1b arg2

But I tried that with no joy. So I added the following at the beginning of
my script:

print "Param1=$ARGV[0]\n";
print "Param2=$ARGV[1]\n";

Using

perl EventProcessor.pl Full_Name="Paul E. Schoen" [email protected]

I got

Param1=Full_Name=Paul E. Schoen
[email protected]
List form of pipe open not implemented Line 46

46: open(MAIL, "|-", $mailprog, $recipient)

So, IMHO this is possibly a bug (or a feature) of the CGI getvars()
function.

I can use JavaScript to replace spaces with underline characters.

Paul

 

[Paul E. Schoen]

Another good resource for web app security
http://www.owasp.org/index.php/Main_Page

and to get an idea of the problems you can face

http://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project

I looked at those and I only understand some of it. It seems like most of
the threats involve passing executable script to the CGI program, but I'm
not doing that and everything is interpreted as text in the print
statements. But perhaps if one of the strings contained executable HTML or
script code, it might be interpreted as an executable on the sender's
browser or whatever is used to display returned data on STDOUT. And that
would only be if s/he got past the password verification. If the Full_Name
contains code, then my script does print that back to the sender. But I can
filter that variable to allow only alphanumeric characters.

Also I think my idea of adding a sleep(n) in the script would limit the rate
at which it would accept new requests, especially if it returned something
like:

Server busy. Please try again later.

I think this would just send thousands of busy messages to the sender if a
DOS attack was attempted. In my application I doubt there would be much
chance of two legitimate users trying to send data at the same time, and
waiting perhaps 10 seconds would not be a serious problem. Those are some
ideas, anyway.

I'll have a look at the "Taint" mode. I used it as follows:

perl -T EventProcessor.pl Full_Name="Paul E. Schoen" [email protected]

and it gave this message:

Insecure $ENV{PATH} ... Line 46

46: open(MAIL, "|-", $mailprog, $recipient)

I'll have to look into why that is so. Thanks again!

Paul