crisis Perl

[Andrew DeFaria]

This is an academic environment. So what.
The people are pretty good,

From your description they are not!

particularly the IT dept (all hardware guys except for me). Most of
our requirements come from the academic side, faculty, deans,
provosts, vice chancellors, and the like, and they commonly don't know
the right questions to ask or sometimes even what they want!

Huh? This is the case in 90% of all business. Most are not technically
oriented. This is otherwise known as "Tell me something new"!.

It's a good job, little stress, lots of time off, interesting work,
the only drawback is the pay but there are always tradeoffs.

You came in here complaining about stress, having to code in an
emergency and all and now it's a low stress job. You're sounding
unbelievable.

I don't think this is intentional, but more of not understanding the
process.

You don't try to tell your manager how to do his job - don't let him
tell you how to do yours!

I quite routinely get my work out real quick, and my clients have come
to expect results well within the hour of making the request.

As you know, not every task is the same.

I've automated so much of what I do that much of it is mindlessly
pushing buttons.

Sounds to me like it's time to move on, move on, *move on!!!*

From their point of view, if I take a long time with something, it's
obviously my fault

Invalid assumption and you know it. Now convey it to them.

-- but this isn't is bad as it sounds because I'm the only guy who
knows how to do what I do (yes, we are one deep) and the praises I get
sometimes are unearned.

I'm now sure what this has to do with anything.

A story:

Oh geeze. Let me pull up the covers...

We generate several thousand contracts five times a year, using a
script I wrote. I got a panic call a couple of hours ago because the
data file had the contract amounts all as zeros. So I started at the
bottom of the stack, checking the inputs of the functions. Yep. All
zeros. Worked my way up the stack to the very top. Zeros all the way
up. Then, I looked at the input file and wouldn't you know it -- the
script expects an eleven column input file but the actual input file
contained ten columns.

Well that's stupid... Oh sorry. Go on...

The reason we were getting zeros was that the relevant column had been
shifted one to the left. Once we revised the input file to the correct
format, all was fine. Yeah, this is a routine event in the World of
Work, but it's kind of interesting that we get to deal with people
issues as well as logical and language issues, "If you would match the
input file to the documentation you would have seen that you were
using the wrong input file, and saved both yourself and me a panic
attack and a wasted morning."

Boring and irrelevant story!!! I'm going to bed now...

 

[Peter J. Holzer]

A story: We generate several thousand contracts five times a year,
using a script I wrote. I got a panic call a couple of hours ago
because the data file had the contract amounts all as zeros. [...]
The reason we were getting
zeros was that the relevant column had been shifted one to the left.

[...]
This is a nice example to emphasize the "write good code in the first
place" point.
Your code probably looked something like this:

while(<STDIN>){
my @line =3D split("/;/");
generate_contract($line[0], $line[3], $line[9]);
}

By only investing 15 seconds more to write e.g.

while(<STDIN>){
my @line =3D split("/;/");
die("Invalid input line, columns !=3D 10") unless $#line =3D=3D 9;
generate_contract($line[0], $line[3], $line[9]);
}

you would immediately have seen what the problem is and spared
yourself from a panic attack.

Even a simple "use warnings" would probably have spewed out a lot of
"uninitialized value" warnings, which might have alerted the user that
something was wrong (even if he doesn't understand what is wrong).

If course you have the benefit of already knowing the unknown-at-the-time
failure mode. What if two columns were swapped rather than one being
missing?

Depends. If a numeric column is swapped for a text column, you get
warnings if you try to use the text strings in a calculation. But if you
don't calculate anything, or you swap two numeric columns, perl can't
help you. You need to add plausibility checks yourself. As Jürgen wrote,
CSV files often have a header line, and you can check that. Other checks
might need domain specific knowledge - for example, if a column contains
fractions in percent, you can check that all the values are between 0
and 100. If I'm processing EU trade data, does the file contain data for
each of the 27 countries? And so on.

hp

 

[jl_post]

As to writing good code to begin with, it's easier said than done.
When it's almost midnight, and you've been at work for about 14 hours,
and you have people (not just your manager, but his boss, the big
boss, and the guy in charge of the project) calling you every 15
minutes, and the people you had promised things mad at you because you
were tasked with doing a job THAT THEY KNEW ABOUT FOUR WEEKS AGO BUT
DIDN'T GIVE YOU A HEADS UP (!!!) ... well, pretty code takes a back
seat.

Okay, I know people have already beaten this thread to death, but I
still wanted to add my input.

I've discovered writing any code, including crisis code, is vastly
improved when the lines "use strict;" and "use warnings;" are used at
the top of the script. You might be saying, "I don't have enough time
to put those in!" but hear me out:

It really doesn't take any more time to program with those lines
than without, and they are HUGE timesavers. They basically point out
obscure errors that would take hours to find without them, and they
end up forcing you to write cleaner code to begin with (which doesn't
take any longer than writing "un-clean" code).

I once had a co-worker who was learning Perl. He was writing a
script, and later told me that his script wouldn't compile, but
eventually figured out how to fix it. I praised him for fixing the
bug and asked how he fixed his script, and he replied, "I removed the
'use strict;' line and the script ran just fine." I told my co-worker
that that didn't actually remove the real problem and offered to look
over his code. Sure enough, there was a simple little error that was
easily fixed (that he didn't have enough experience to identify).

Why am I telling you this story? Because a lot of people
mistakenly think that programming with "use strict;" and "use
warnings;" takes lots longer than programming without them, but in
practice, I've found just the opposite is true (they tell you exactly
where a potential error is). All you have to do is learn to program
using them, and then you'll see your development times plummet (or at
least significantly drop). And if you don't regularly program with
them, I suggest learning to as soon as possible, and not waiting for
the next crisis to learn!

As an added bonus, the "use warnings;" line won't actually change
the behavior of your program, but will spit out warnings when it
thinks something's wrong with your script. You might consider this a
bonus because if you discover weeks later that, due to the warnings
your script is giving, your input is in the wrong format, you may have
an easier time convincing your manager that the script needs to be
cleaned up. Otherwise, the script will be silent about your incorrect
input assumptions -- giving the managers the false impression that
everything is okay. (I've seen this a lot myself.)

I also recommend adding "use strict;" and "use warnings;" to any
script that doesn't already have it. I know it's easier said than
done when dealing with "crisis code" that you didn't write yourself,
but the effort you take to convert the script (such as declaring
variables) will be worth it. I've had to do it several times myself,
and I can tell you that it gets easier the more times you do it.

And if you contest that you don't have the time to convert your
script to run with "use strict;" then I seriously recommend that AT
THE VERY LEAST you still put in "use warnings;". Like I said before,
"use warnings;" won't change the behavior of your script -- but IT
WILL report many simple errors, like misspelled variable names (which
are often difficult to track down), uninitialized variables, and input
in the wrong format (like trying to add two non-numbers together).

Also, be aware that you can still use "strict" and "warnings" in
blocks of code that you add in yourself. If you find yourself adding
a new block of code (such as in a loop or a condition), you can still
put "use strict;" and "use warnings;" in that block to take advantage
of their special powers of error-finding. (If you use any outside
variables in that block, however, you'll have to declare those
variables (with "my") where they're first used in order to prevent
"use strict;" from complaining. But that should be fairly easy to
do.)

Also, I highly recommend that you use "or die "[error message goes
here]: $!\n";" (or something similar) after every open(), opendir(),
and chdir() statement you use. Now, several people I've worked with
shun them saying "I don't want the program to die, I want it to
continue." This is usually just a cop-out, but if their reasons are
legitimate, then use "or warn" instead of "or die". The program will
still continue, but it will give a nice warning when something doesn't
happen that it expected to happen.

Another excuse I hear is, "I don't have time to type out 'or die
"blah blah blah: $!\n";' hundreds of time in my code" That's a lousy
excuse, but if they insist, then have them use just " or die;" (or AT
LEAST " or warn;"). Those seven extra keystrokes should be easy
enough for anyone to type that it won't increase development times
(presuming they CAN type).

I mostly grab data from one place, massage it, and send it to another
place. The format of the data I get, and/or the format of the final
form, changes several times a year. It's not that the code breaks a
lot, but that the specifications change. Since we can't 'fix' the
specifications, we have to 'fix' the code, and guess who gets blamed
if the code isn't 'fixed.'

I do a lot of "data massaging" myself, and I can say that it's a
life-saver to be able to code with "use strict;" and "use warnings;".
Otherwise, a simple misspelled variable name will create a bug that
can take hours to track down, and a parsed out value that I expect to
be a number but isn't can go undetected for years (literally!).

And, like you, I have also had cases where the specifications
change. Especially in those cases, I will take care to DOCUMENT IN
THE FORM OF COMMENTS what the input is supposed to look like,
including GIVING EXAMPLES of the input. The examples are a
lifesaver! That's because when the specifications DO change, then
I'll have a record of what the OLD specifications looked like, and
have an easier time massaging the new input into the old data
structures.

Here's an example of how I'd document examples:

# Look for a line that looks like:
# MGAP 30N50W ...INCREASING PRESSURE...
# and store the latitude and longitude in the %location hash:
if (m/^(\w{4})\s+(\d+[NS])(\d+[EW])\b/)
{
$location{$1}{latitude} = $2; # $2 looks like "30N"
$location{$1}{longitude} = $3; # $3 looks like "50W"
}

That way, when the next maintainer encounters the code, they won't
have to decipher the regular expression to figure out what it was
meant to do. They may have to verify that the regular expression does
as advertized, but in my experience this is MUCH simpler to do than
deciphering the expression from scratch, as deciphering the regular
expression (with no intent known as to what it's supposed to be
looking for) makes it almost impossible to figure out if it contains a
bug -- because there is no way to verify if its actual behavior is
what the original programmer meant it to be.

In other words, placing comments of expected input gives the
maintainer (which I suspect will probably be you) an easier time of
following the input along the code, and of being able to tell if a
possible bug lies in one area of code (versus another).

One more thing:

Once I read a post by a poster named "A. Sinan Unur" that
recommended declaring variables IN AS SMALL AS SCOPE AS POSSIBLE.
That is, instead of declaring all your variables at the top of your
script, declare then inside the smallest block needed. So if a
variable isn't needed OUTSIDE a loop nor BETWEEN its iterations, go
ahead and declare it INSIDE the loop brackets.

So instead of code like this (which I've seen before and and had to
clean up):

$level = 0;
$station = "";

 

[cartercc]

1.  If you don't already, ALWAYS program with "use strict;" and "use
warnings;" and learn to fix the errors they give out.
Agree.

2.  If you're fixing a script someone else has written that doesn't
use "strict" and "warnings", ALWAYS add "use warnings;" and strongly
consider adding "use strict;".

Agree. I tend to test the code with the -w switch rather than use
warnings because users get confused about whatever warning messages
may remain in the script, but yes, always run with warnings enabled.

3.  ALWAYS handle open(), opendir(), and chdir() statements.  I
recommend you use "or die "[error message]: $!\n"", but using "or die
$!;", "or warn;" or any other method of handling a failure is also
acceptable.

Agree. Sometimes I get lazy and omit this, but I've made this a habit
so I have to deliberately omit it.

4.  When massaging input, ALWAYS give examples of the expected input.
The actual input will eventually change, but having examples of
previous expected input will let the future maintainer know which new
input goes into which existing data structures.

This is where we break down. It takes time to document, and time is
sometimes a luxury I don't have. I generally like to include a README
file along with the script, because it's usually several things (e.g.,
a database query, a distribution list) that wouldn't go in the code. I
almost always include a script header and a function header (required
arguments, return values, and side effects), but honestly, most of the
time it's just a blank header. In particular, POD is time consuming
and only used by me. This is my biggest failure.

5.  Get in the habit of declaring your variables in as small as scope
as possible.  This will lead to less-polluted namespaces, less
variables to keep track of at any one time, and fewer chances of like-
named variables writing over each other's values.

Agree with a couple of caveats. If I use a standard variable
throughout, like $counter, or $sum, I'll declare it at the top simply
to avoid the overhead of a multitude of 'my' declarations. If I'm
writing a CGI script or a module, I'll declare all the variables at
the top as a form of documentation or pseudocode or like fields in OO
languages, such as:

my $nametitle;
my $namefirst;
my $namemiddle;
my $namelast;
my $namesuffix;
my $addressstreet;
my $addresscity;
my $addresszip;
my $phonehome;
my $phonecell;
my $phonework;

.... and so on. That way, I can see at a glance my logical fields, even
though I may use some of them only within subs.

   I hope this helps.  I've been in your situation before, and I've
found that following all of these suggestions helps immensely.

Agree. I think that the biggest problem is a lack of support or a lack
of understanding on the part of the immediate IT upstream. We have
about 12 staff, one software guy (me), a couple of management types
(including my immediate supervisor), and all the rest are hardware
guys who if they have had any programming experience at all have used
a drag and drop IDE like VS and think that all there is to it is
dropping widgets on a canvas and setting their properties. It's
sometimes frustrating to be told to do the obvious by someone who
doesn't have the knowledge or experience to understand why the obvious
won't work.

Thanks, CC.

 

[Juha Laiho]

cc> How do you deal with a manager who tells you to leave a script
cc> alone, when you know good and well that it's such poorly written
cc> code that it will be extremely hard to maintain, and perhaps
cc> will be buggy as well? Getting another job isn't an option, and
cc> firing the manager isn't an option, either.

Educate the manager. Keeping shoddy code in production is a gamble:
you're gambling that the cost of fixing the code *now* is higher than
the cost of fixing it when it breaks or when there's a crisis.

Further, making modifications to a clean code base is cheaper than
making the same modifications to a degraded code base.

The common term for this is "design debt"; see articles at
http://jamesshore.com/Articles/Business/Software Profitability Newsletter/Design Debt.html
http://www.ri.fi/web/en/technology-and-research/design-debt

As far as the code does what it is intended to do now, it is, stirctly
speaking, not a technical issue (which, I guess, would make it belong to
your domain), but instead it is an economic issue (and as such belongs
to your managers domain). What you could do would be to try and see
whether your manager understands the "debt" aspect of the situation
his decision is creating, and how much of this debt he is willing
to carry (and also pay the running interest for -- in more laborious
changes to the code).

 

[Juha Laiho]

cc> How do you deal with a manager who tells you to leave a script
cc> alone, when you know good and well that it's such poorly written
cc> code that it will be extremely hard to maintain, and perhaps
cc> will be buggy as well? Getting another job isn't an option, and
cc> firing the manager isn't an option, either.

Educate the manager. Keeping shoddy code in production is a gamble:
you're gambling that the cost of fixing the code *now* is higher than
the cost of fixing it when it breaks or when there's a crisis.

Further, making modifications to a clean code base is cheaper than
making the same modifications to a degraded code base.

The common term for this is "design debt"; see articles at
http://jamesshore.com/Articles/Business/Software Profitability Newsletter/Design Debt.html
http://www.ri.fi/web/en/technology-and-research/design-debt

As far as the code does what it is intended to do now, it is, stirctly
speaking, not a technical issue (which, I guess, would make it belong to
your domain), but instead it is an economic issue (and as such belongs
to your managers domain). What you could do would be to try and see
whether your manager understands the "debt" aspect of the situation
his decision is creating, and how much of this debt he is willing
to carry (and also pay the running interest for -- in more laborious
changes to the code).

 

[sln]

Further, making modifications to a clean code base is cheaper than
making the same modifications to a degraded code base.

The common term for this is "design debt"; see articles at
http://jamesshore.com/Articles/Business/Software Profitability Newsletter/Design Debt.html
http://www.ri.fi/web/en/technology-and-research/design-debt

As far as the code does what it is intended to do now, it is, stirctly
speaking, not a technical issue (which, I guess, would make it belong to
your domain), but instead it is an economic issue (and as such belongs
to your managers domain). What you could do would be to try and see
whether your manager understands the "debt" aspect of the situation
his decision is creating, and how much of this debt he is willing
to carry (and also pay the running interest for -- in more laborious
changes to the code).

^^^^^^^
This is not a good argument. The best you can do is what was intended.
Nobody can get schizo about it and program place holders for what if's
pathologically.

Most economical companies don't hire/have working program managers, ie:
somebody who wan't to program, let alone someone who will lay out details
for the real workers.

As a result, technical knowledge usually works out to be:
programmer > manager > director > vice > president > CEO > Bill Gates

Unfortunately, the idea/blame runs in the opposite direction.
Shit does roll downhill.

Lastly, the paygrade runs in the opposite direction as well, where
it is cheaper to toss the programmer, and justify starting all over again.

If you work in such a place (I have in many). Shut your mouth, never admit
a problem, modify thier ****-up fast. If you have a programmer that blames
everything on you, beat the crap out of them, see if that helps next time.

sln

 

[Juha Laiho]

(e-mail address removed) said:

^^^^^^^
This is not a good argument. The best you can do is what was intended.
Nobody can get schizo about it and program place holders for what if's
pathologically.

No, what I wrote about was not about putting in placeholders for
future enhancements. I wrote about having the code written in such
a shoddy way that each change starts to require enormous amounts
of testing, because you just cannot be sure what all is affected
by the change -- or having code that requires you to write layer
upon layer of special cases to cover for earlier bad design
decisions. Or code where "reuse" was made using cut&paste, instead
of modular, composable pieces -- where, to consistently make a single
change, you end up making the same change many times over.

 

[corky]

Over the past four years, I've written a fair amount of Perl. Some of
it has been 'crisis Perl'. This results in scripts that solve a
problem but are thrown together in a hurry with inefficient, untested,
and confusing code. When the crisis resolved, I wanted to go back, do
real testing, and rewrite the script, but have been told on almost
every occasion to leave it alone. The fact that it seemed to worked
was good enough, and most of this code found its way into production.

In the past month or so, I've had to look at four scripts I have
written this way. One script was over two years old, and the newest
was several months old. Needless to say, dealing with confusing,
uncommented, and inefficient code is a problem! It would have been
much easier to clean up the code when it was written than to rewrite
it months after the fact.

I actually knew better than to not clean up the code, but it was
easier at the time not to pick a fight with my managers. This isn't an
excuse, but an explanation.

How do you deal with a manager who tells you to leave a script alone,
when you know good and well that it's such poorly written code that it
will be extremely hard to maintain, and perhaps will be buggy as well?
Getting another job isn't an option, and firing the manager isn't an
option, either.

CC

Why am I not surprised that there was a crisis in the first place, with this
management in charge.
My life experience is that when people insist on screwing themselves, let
them do it. If you must, perhaps send an email explaining your idea, so you
have some CYA later. Otherwise, let it go and move on to the next thing.

 

[News123]

Hi,

No, what I wrote about was not about putting in placeholders for
future enhancements. I wrote about having the code written in such
a shoddy way that each change starts to require enormous amounts
of testing, because you just cannot be sure what all is affected
by the change -- or having code that requires you to write layer
upon layer of special cases to cover for earlier bad design
decisions. Or code where "reuse" was made using cut&paste, instead
of modular, composable pieces -- where, to consistently make a single
change, you end up making the same change many times over.

Welcome in the real world

A pragmatic compromise might be:
- Do not modify the code, as your boss told you so, as least don't
modify it in anticipation.
- Whenever you have to add a feature or fix a bug, refactor at least
parts of the code at the same time. If your manager asks then why you
needed so long to fix it you can tell him, that the code was badly
structured and it was very tough to apply the fix.


bye



N

 

[News123]

Hi,

No, what I wrote about was not about putting in placeholders for
future enhancements. I wrote about having the code written in such
a shoddy way that each change starts to require enormous amounts
of testing, because you just cannot be sure what all is affected
by the change -- or having code that requires you to write layer
upon layer of special cases to cover for earlier bad design
decisions. Or code where "reuse" was made using cut&paste, instead
of modular, composable pieces -- where, to consistently make a single
change, you end up making the same change many times over.

Welcome in the real world

A pragmatic compromise might be:
- Do not modify the code, as your boss told you so, as least don't
modify it in anticipation.
- Whenever you have to add a feature or fix a bug, refactor at least
parts of the code at the same time. If your manager asks then why you
needed so long to fix it you can tell him, that the code was badly
structured and it was very tough to apply the fix.


bye



N