a defense of ad hoc software development

[ccc31807]

On page 229 of Paul Graham's 'ANSI Common Lisp' he writes:

<quote>
If programming were an entirely mechanical process -- a matter of
simply translating specifications into code -- it would be reasonable
to do everything in a single step. But programming is never like that.
No matter how precise the specifications, programming always involves
a certain amount of exploration -- usually a lot more than anyone had
anticipated.
It might seem that if the specifications wee good, programming
would simply be a matter of translating them into code. This is a
widespread misconception. Programming necessarily involves
exploration, because specifications are necessarily vague. If they
weren't vague, they wouldn't be specifications.
In other fields, it may be desirable for specifications to be as
precise as possible. If you're asking for a piece of metal to be cut
into a certain shape, it's probably best to say exactly what you want.
But this rule does not extend to software, because programs and
specifications are made out of the same thing: text. You can't write
specifications that say exactly what you want. If the specification
were that precise, then they would be the program.
</quote>

In a footnote, Graham writes:

<quote>
The difference between specifications and programs is a
difference in degree, not a difference in kind. Once we realize this,
it seems strange to require that one write specifications for a
program before beginning to implement it. If the program has to be
written in a low-level language, then it would be reasonable to
require that it be described in high-level terms first. But as the
programming language becomes more abstract, the need for
specifications begins to evaporate. Or rather, the implementation and
the specifications can become the same thing.
If the high-level language is going to be re-implemented in a
lower-level language, it starts to look even more like specifications.
What Section 13l.7 is saying, in other words, is that the
specifications for C programs could be written in Lisp.
</quote>

In my SwE classes, we spent a lot of time looking at process and
processes, including MIL STD 498 and IEEE Std 830-1984, etc. My
professors were both ex-military, one who spent his career in the USAF
developing software and writing Ada, and both were firmly in the
'heavy' camp (as opposed to the lightweight/agile camp).

In my own job, which requires writing software for lay users, all I
ever get is abbreviated English language requests, and I have learned
better than to ask users for their requirements because they have no
idea what requirements are. (As a joke, I have sent a couple a copy of
IEEE Std 830-1984 and told them that I needed something like that, but
the joke went over like a lead balloon -- not funny at all.) Of
necessity I have learned to expect to write a number of versions of
the same script before the user accepts it.

I understand that Graham isn't talking about requirements, and to many
people specifications and requirements amount to the same thing. I
also understand the necessity for planning. However, the Graham quote
seems to me a reasonable articulation for ad hoc development. (It's
something I wanted to say to jue in particular but couldn't find the
words.)

Comments?

CC

 

[Marc Girod]

Comments?

Paul Graham is a genious. He speaks Gold.
I loved his book /On LISP/.
Planning is useful, but plans are worthless.

Marc

 

[Richard McBeef]

ccc31807 wrote:
[snip]

Comments?

Yeah, Paul Graham is a total b.s. artist.
Besides a prolific author of internet screeds
he wrote some successful software a long time ago.
Oh, and I do not deny he is a lisp expert.
But, yeah, PG is full of shit and his bazillion
bullshit essays all have the same annoying
common theme "I'm the best! Be like me!".

 

[fortunatus]

In my own job, which requires writing software for lay users, all I
ever get is abbreviated English language requests, and I have learned
better than to ask users for their requirements because they have no
idea what requirements are.

Nobody expects non-programmers to be able to generate programmers's
requirements.

Programmers generate requirements for themselves based on less formal
Q&A with the customer.

No matter what, a programmer works to specific requirements that are
supposed to match the user's expectations and/or needs. The
requirements might be in the programmer's head, and they might be
developed by evolutionary process throughout the code writing, but
they exist. If there is more than one programmer on a team then there
is a need to codify requirements into documents to some degree.

I think it is not feasible to say that requriements can be so detailed
as to be directly executable. In teamwork it is useful to communicate
when there remains a large degree of ambiguity, and allow team members
to contribute to resolutions.

So you can use Lisp to capture/express requirements, but it won't be a
complete solution to the problem before you break the work up among
team members. Yet at that stage it would normally be named a
requirements document.

My point is that the work needs to be broken up and distributed among
team members before ambiguity is resolved. At that point there should
be some requirements written down for team members to start working
from. Therefore the requirements, at that stage, even if captured in
Lisp or other executable format, /must/ have too many holes to really
solve the whole problem, or else the team is not dividing the workload
effectively.

 

[Kenneth Tilton]

ccc31807 wrote:
[snip]

Comments?

Yeah, Paul Graham is a total b.s. artist.
Besides a prolific author of internet screeds
he wrote some successful software a long time ago.
Oh, and I do not deny he is a lisp expert.
But, yeah, PG is full of shit and his bazillion
bullshit essays all have the same annoying
common theme "I'm the best! Be like me!".

No, he's pretty good and a very nice guy, too. No b.s., unless by b.s.
you mean anything you disagree with. That is my definition, too!

kt

 

[Barry Margolin]

ccc31807 wrote:
[snip]

Comments?

Yeah, Paul Graham is a total b.s. artist.
Besides a prolific author of internet screeds
he wrote some successful software a long time ago.
Oh, and I do not deny he is a lisp expert.
But, yeah, PG is full of shit and his bazillion
bullshit essays all have the same annoying
common theme "I'm the best! Be like me!".

That may be true, but what he says in this particular article is
correct. Read the book "Coders at Work", and you'll learn that most of
the great programmers think similarly. Programming simply cannot be
mechanized. Writing specifications precise enough to be validated
mechanically *is* programming.

 

[Pascal J. Bourguignon]

On page 229 of Paul Graham's 'ANSI Common Lisp' he writes:

<quote>
[...]
Comments?

Indeed. See also:
http://www.developerdotstar.com/mag/articles/reeves_design_main.html
http://www.developerdotstar.com/mag/articles/PDF/DevDotStar_Reeves_CodeAsDesign.pdf

 

[Marc Girod]

Nobody expects non-programmers to be able to generate programmers's
requirements.

Unfortunately, many people do.
This is bullshit from the start and
results in more bullshit. How this
bullshit is dealt with is an important
part of everyday life nowadays.

Programmers generate requirements for themselves based on less formal
Q&A with the customer.

Formality is another interesting way
to glide astray from reality.
Formality doesn't bind to precision,
or to correctness.

Formal semantics are a myth, precisely
one which fights the reality of code
semantics.
What makes code semantics special, and
thus more interesting, is that code is
meant to be run, and not only to be read.

That is, contrarily to any other kind
of representation, code is not semantics
about something else: it is embodied,
operational, semantics.

Now, the problem is one of communications
(whether Q&A with customers or else.)

The
requirements might be in the programmer's head, and they might be
developed by evolutionary process throughout the code writing, but
they exist.

Until there is a program, their existence
doesn't imply 'essence', i.e may not be
consistent of even feasible (no proof of
convergence).

My point is that the work needs to be broken up and distributed among
team members before ambiguity is resolved.

And even then.
Ambiguities get resolved while merging the
contributions (aka integration).

Marc

 

[Nick Keighley]

Nobody expects non-programmers to be able to generate programmers's
requirements.

but it isn't unreasonable to expect them to be able to articulate
their own requirements. Do they want a word processor or an ABS?

Programmers generate requirements for themselves based on less formal
Q&A with the customer.

No matter what, a programmer works to specific requirements that are
supposed to match the user's expectations and/or needs.  The
requirements might be in the programmer's head, and they might be
developed by evolutionary process throughout the code writing, but
they exist.  If there is more than one programmer on a team then there
is a need to codify requirements into documents to some degree.

I think it is not feasible to say that requriements can be so detailed
as to be directly executable.

various people have though (wrongly I think). The TDD people sometimes
claim the test is an executable requirement.

In teamwork it is useful to communicate
when there remains a large degree of ambiguity, and allow team members
to contribute to resolutions.

So you can use Lisp to capture/express requirements,

no. requirements are a communication with an end user. Only the most
sophisticated of end users could be epected to read lisp.

 

[Tamas K Papp]

but it isn't unreasonable to expect them to be able to articulate their
own requirements. Do they want a word processor or an ABS?

Reasonableness is not the right concept here.

There is a market for programmers, and some employers/customers want
programmers who can write software without being given super-specific
formal requirements. Programmers who can cater to this market by
making an effort to understand the problem domain may enjoy an
advantage. The market will decide. If you don't like the market
outcome, you can complain that it is "unreasonable", but no one will
care.

Tamas

 

[ccc31807]

Nobody expects non-programmers to be able to generate programmers's
requirements.

I don't know the difference between 'requirements' and 'specification'
but I assume that the requirements state what the software will do,
and the specification states how the software will do it.

People like to poke fun at UML use cases and use case diagrams, me
included. However, use cases can describe in the language of the
problem domain the functional requirements (if not the non-functional
requirements as well), e.g.
- customer presents the goods to the cashier
- cashier determines the price of the goods
- cashier calculates the total price including tax
- cashier informs customer of the total price
- customer tenders the price in cash, check, or plastic
- cashier accepts customer's tender
- cashier gives customer a receipt for the goods.

Programmers generate requirements for themselves based on less formal
Q&A with the customer.

It may be more accurate to say that the programmer generates the
design based on (1) functional requirements furnished by the client,
and (2) an explicit or tacit specification based on those
requirements.

requirements might be in the programmer's head, and they might be
developed by evolutionary process throughout the code writing, but
they exist.

I've been taken to task (on c.l.p.m.) for saying that many times I
just start writing code without doing any analysis or design.
Obviously, for a larger project, analysis and design is essential. I
think that Graham's point is that analysis, design, and implementation
can be integrated with more efficiency and less work. Obviously, you
can't build a sizable application this way, but just as obviously any
sizable application necessarily contains problems that can only be
solved by experimentation.

CC.

 

[Tim Bradshaw]

Writing specifications precise enough to be validated
mechanically *is* programming.

I was going to say "and there are these programs called compilers which
turn them into executable programs" but that's not quite right. A
specification could just give a way of testing whether something meets
it, without giving a way for that thing to be constructed, while a
program has to give a way to construct that thing. So perhaps a
specification is not programming: it's just something which is as hard
as programming.

 

[MarkHaniford]

That may be true, but what he says in this particular article is
correct.  Read the book "Coders at Work", and you'll learn that most of
the great programmers think similarly.  Programming simply cannot be
mechanized.  Writing specifications precise enough to be validated
mechanically *is* programming.

Programming is already mechanized. The problem is with having
"business analysts" spending 10x as much time "writing out the specs"
to be mechanized than a "normal" programmer. Or on another note, your
CFO could spend hours a day cleaning toilets at the office too.

 

[Tim X]

I've been taken to task (on c.l.p.m.) for saying that many times I
just start writing code without doing any analysis or design.
Obviously, for a larger project, analysis and design is essential. I
think that Graham's point is that analysis, design, and implementation
can be integrated with more efficiency and less work. Obviously, you
can't build a sizable application this way, but just as obviously any
sizable application necessarily contains problems that can only be
solved by experimentation.

Analysis is critical because that is how, you as the programmer, get to
understand the domain problem you will be working in. Design is also
important, but it should not be carved in stone.

I think PGs main point is that it is often during the implementation
that you gain that deeper understanding of the problem that provides the
insight for better solutions and you want to avoid any specification
that is so detailed that you cannot adopt those better solutions because
they would not fit with the specification.

I've worked on projects in the past where the specification was so
detailed, it included all the names of procedures/functions, the
arguments they would accept and the values they would return. As a
developer on that project, all you were expected to do was write those
functions and procedures and at the end, it would all fall together. It
was an uninspiring, horrible boring project and it was pretty much a
failure.

I think what PG is talking about is close to what the Agile (God I hate
that term - bloody weasel word. Lets face it, who the hell doesn't want
to be 'agile'!) camp has been proposing. Use user stories to get an
understanding of what is required and then use test driven development
to not only verify the code and protect against regressions after
refactoring, but also to verify you as the programmer understand the
customer/owner's requirements. This is why tests should be developed in
conjunction with the product owner or representatitve/expert from the
client side.

Requirements and specifications are all about communication. They are
there to ensure that everyone knows what the widget is and what the
widget needs to be able to do. The old waterfall methodology tried to
make it like dam and bridge building - eveyrthing would be specified
down to the last bolt and nail. Management would know exactly how much
it would cost and exactly how long it would take to build (at least,
they felt reassured they did). This approach
essentially failed. It failed because it was unable to take into account
that programming isn't like dam and bridge building. We don't have the
hundreds of years experience or the relatively well understood laws,
such as the laws of physics, that engineering in the physical world has.
Programming has more art than science. The solution often comes with
understanding and imagination, which is acquired as you attempt to solve
the problem and will seldom become obvious through analysis alone.

A major point that PG was trying to make is that your development
environment can assist in this process by making it easier to experiment
and explore the problem domain. He argues that lisp is one of the best
languages for doing this and it is why he believes a single lisp
developer can be more productive than, lets say, a developer using C.
This theme is also carried through in his other writings. For example,
he argues that the success of viaweb was because it used lisp and they
were able to add features faster than their competition. In fact, he
says that their competition often thought he had spies in their team
because they would release a new feature and viaweb would have
incorporated it by the end of that day.

Tim

 

[MarkHaniford]

Analysis is critical because that is how, you as the programmer, get to
understand the domain problem you will be working in. Design is also
important, but it should not be carved in stone.

I think PGs main point is that it is often during the implementation
that you gain that deeper understanding of the problem that provides the
insight for better solutions and you want to avoid any specification
that is so detailed that you cannot adopt those better solutions because
they would not fit with the specification.

I've worked on projects in the past where the specification was so
detailed, it included all the names of procedures/functions, the
arguments they would accept and the values they would return. As a
developer on that project, all you were expected to do was write those
functions and procedures and at the end, it would all fall together. It
was an uninspiring, horrible boring project and it was pretty much a
failure.

I think what PG is talking about is close to what the Agile (God I hate
that term - bloody weasel word. Lets face it, who the hell doesn't want
to be 'agile'!) camp has been proposing. Use user stories to get an
understanding of what is required and then use test driven development
to not only verify the code and protect against regressions after
refactoring, but also to verify you as the programmer understand the
customer/owner's requirements. This is why tests should be developed in
conjunction with the product owner or representatitve/expert from the
client side.

Requirements and specifications are all about communication. They are
there to ensure that everyone knows what the widget is and what the
widget needs to be able to do. The old waterfall methodology tried to
make it like dam and bridge building - eveyrthing would be specified
down to the last bolt and nail. Management would know exactly how much
it would cost and exactly how long it would take to build (at least,
they felt reassured they did). This approach
essentially failed. It failed because it was unable to take into account
that programming isn't like dam and bridge building. We don't have the
hundreds of years experience or the relatively well understood laws,
such as the laws of physics, that engineering in the physical world has.
Programming has more art than science. The solution often comes with
understanding and imagination, which is acquired as you attempt to solve
the problem and will seldom become obvious through analysis alone.

A major point that PG was trying to make is that your development
environment can assist in this process by making it easier to experiment
and explore the problem domain. He argues that lisp is one of the best
languages for doing this and it is why he believes a single lisp
developer can be more productive than, lets say, a developer using C.
This theme is also carried through in his other writings. For example,
he argues that the success of viaweb was because it used lisp and they
were able to add features faster than their competition. In fact, he
says that their competition often thought he had spies in their team
because they would release a new feature and viaweb would have
incorporated it by the end of that day.

Tim

Tim, I might reply to your response in detail later on, but I think
you made some good points. "Agile" has become a religion of its own,
and the whole mission of sitting down with your "customers" and
rapidly banging out mockups has been lost. It's the rapid-feedback
that gave death to waterfall (communication).


Not responding to you Tim, but on another note, nobody in their right
mind would hire anybody that writes Paul Graham type code. It's the
classic, "what happens when Billy gets hit by a bus".

 

[Robert Maas, http://tinyurl.com/uh3t]

No matter how precise the specifications, programming always involves
a certain amount of exploration -- usually a lot more than anyone had
anticipated.

There is one exception where very little experimentation may be
needed, namely implementing formal protocol specifications, such as
RFC822. It would be reasonable for a high-level API to be written
directly from the specs to implement each little piece of RFC822 as
a callable routine. Thus there might be a function/method to create
an empty RFC822 object, another for each type of header line to add
to the object, another to verify that all the required lines are
present, and then one to pass the completed object to some
receiver. Likewise there might be a parser for RFC822 which returns
an object showing each of the fields that was present, and then
accessors for each field separately.

The rest of the application would be as you quote, experimental
development, but this one module which does the RFC822 API
implementation would be from the specs rather than very
experimental.

In my SwE classes, we spent a lot of time looking at process and
processes, including MIL STD 498 and IEEE Std 830-1984, etc. My
professors were both ex-military, one who spent his career in the
USAF developing software and writing Ada, and both were firmly in
the 'heavy' camp (as opposed to the lightweight/agile camp).

I'm less familiar with military specifications, but I would guess
these are like InterNet specs, where you might code an API module
for each set of specs, coding directly from the specs, then write
the rest of the application in a more experimental style.

Of necessity I have learned to expect to write a number of
versions of the same script before the user accepts it.

It is sometimes desireable to quickly write non-functional UI which
shows the user how it "looks", and get their approval of that,
before starting the real code under the UI.

 

[Robert Maas, http://tinyurl.com/uh3t]

There is a market for programmers,

If by "programmer" you mean "somebody who is capable of designing
new software and implementing them", and if by "market" you mean
somebody wiling to hire, I don't believe you. I'm a "programmer"
per that defintion, and I challenge you to find even one person or
company willing to hire for what I do.

and some employers/customers want programmers who can write
software without being given super-specific formal requirements.

I would prefer to brainstorm with the employer to work out our
common idea of what they want that I believe is
possible/reasonable to implement.

Programmers who can cater to this market by making an effort to
understand the problem domain may enjoy an advantage.

How deeply must I understand the problem domain?? Can you give an
example of a currently open requirement, the problem domain it
involves, and some idea how deep I would need to understand it
before I even talk to the employer, and how much the employer can
explain to me to bring me up to the needed level of understanding?

 

[Robert Maas, http://tinyurl.com/uh3t]

"Agile" has become a religion of its own, and the whole mission
of sitting down with your "customers" and rapidly banging out
mockups has been lost. It's the rapid-feedback that gave death
to waterfall (communication).

Are you saying that software designer and customer/employer should
talk to the point where they think they agree what needs to be
implemented, then software designer quickly makes a mockup, and
shows it to the customer/employer to "try" to see whether it's what
c/e wanted or not, and if not then another round of mockup, until
c/e is satisfied, then do same for next requirment, etc. until the
whole application is mocked-up to the satisfaction of the c/e, then
the s.d. can integrate everything into an application? If so, I
agree, and I just wish I could find some c/e to hire me so that I
can try it with a real live c/e for the first time ever.

Note that for Web-server applications, PHP may be the best language
for rapid prototyping/mockingup, better even then Lisp because it
better integrates the HTML with the algorithms, with not much loss
of programming power compared to Lisp. Thus the time saved in not
needing to do any work integrating HTML with algorithms more than
makes up for the time spent figuring out how to do an algorithm in
PHP which was trivial in Lisp. On the other hand, for any really
serious new algorithms, I'd prefer to develop in Lisp, and not even
bother to interface to HTML until after the algorithm is working as
wanted by the c/e.

 

[Pascal J. Bourguignon]

There is one exception where very little experimentation may be
needed, namely implementing formal protocol specifications, such as
RFC822.

In theory. But in practice, you still need to bump against other
implementations to learn how to treat the cases that are not covered
by the protocol. There are RFC that are specifically describing the
"best practices", which means that there are a lot of implementations
implementing "worst practices" or "good-enough practices".

It also depend on the specific protocol (and its specification) some
protocol specifications may lead to more variability and more "fun".

 

[Tamas K Papp]

If by "programmer" you mean "somebody who is capable of designing new
software and implementing them", and if by "market" you mean somebody
wiling to hire, I don't believe you. I'm a "programmer" per that

That's your problem. Clearly there are employed programmers. And you
are a very special case, so your job market experience is not indicative
of anything.

defintion, and I challenge you to find even one person or company
willing to hire for what I do.

Whatever. I am not a job search agency. (BTW, these "I challenge you"-
type utterances must be a clear winner in job interviews).

How deeply must I understand the problem domain?? Can you give an
example of a currently open requirement, the problem domain it involves,
and some idea how deep I would need to understand it before I even talk
to the employer, and how much the employer can explain to me to bring me
up to the needed level of understanding?

These things are project specific, and are usually decided between you
and your employer/customer. Eg if you are writing a library for
numerical applications, most employers would prefer that you have a
clear understanding of floating-point arithmetic (eg on the level of
"What Every Scientist Should Know ...").

But the more you know, the less your customer will have to explain.
And the flipside is that if you don't know anything, sometimes
introducing you to the problem domain is prohibitively costly, so
people become their own programmers (which happens in science very
frequently).

Tamas