Herald: Python surpasses Perl in popularity!

[Steven D'Aprano]

Xah Lee wrote: ....

your remark is a bit overzealous. After all, we all know that site is
websearh based. Although it not some kinda scientific report, but it
does give some good indication of language popularity, however you
define that.

[... rant omitted ...]
I wondered how long you would be able to resist making your vituperative
remarks. Please stop this rubbish.

Oh please Steve. Did you read Xah's post or stop after the second
paragraph? It was amazingly *non* vituperative, and I don't just mean
"for Xah".

I only counted two uses of the f-word and one of the a-word, none of
which I personally would object to (we're grown-ups, naughty words
shouldn't shock us). The only things which were even *close* to a
vituperative rant were a short comment about Brian Harvey being unhappy
with Scheme 6, and that Sun Microsystems use inferior versions of shell
tools. Maybe half a dozen lines expressing strong opinions, out of a 300
line post.

 

[Jorgen Grahn]

When you've seen some of the shell scripts I've seen, it's not hard to
imagine that logo might be a better choice of programming language.

I have seen them, too. But I have also written and maintained
medium-sized shell scripts which are very successful.

(I could get away with using Bash in these cases. It has functions,
local variables and so on. Writing portable Bourne shell is not as
much fun.)

Not to mention how useful interactive shell scripting is. I use it
more and more: pipelines which combine xargs, while, for, sort and
perl oneliners. This is surely a kind of programming, too.

/Jorgen

 

[Paul Boddie]

Oh please Steve. Did you read Xah's post or stop after the second
paragraph? It was amazingly *non* vituperative, and I don't just mean
"for Xah".

Agreed, although I had to look "vituperative" up first. Is the mere
presence of Xah Lee's name now shocking in itself?

I only counted two uses of the f-word and one of the a-word, none of
which I personally would object to (we're grown-ups, naughty words
shouldn't shock us). The only things which were even *close* to a
vituperative rant were a short comment about Brian Harvey being unhappy
with Scheme 6, and that Sun Microsystems use inferior versions of shell
tools. Maybe half a dozen lines expressing strong opinions, out of a 300
line post.

And I'd definitely want to express frustration with the latter
situation if I had to use a Sun operating system without the GNU suite
of programs installed. I haven't checked, but perhaps even the latest
release of OpenSolaris still has the /usr/ucb, /usr/ccs and /usr/xpg5
directories (or whatever they're called), although these probably no
longer provide the default tools.

Paul

 

[Tam Ha]

(I could get away with using Bash in these cases. It has functions,
local variables and so on. Writing portable Bourne shell is not as
much fun.)

Can you explain this? Bourne is always more portable than Bash.
That's why you'll find experienced shell programmers writing everything
that doesn't absolutely require a bash feature in /bin/sh. Boot scripts,
install scripts, etc. should never be written in bash and if where you
find one using bash you can be sure a Linux-only newbie has written it.
For one there are too many versions of bash, for two it is not installed
by default on every Unix/Linux OS, for three it has poor backwards
(and forwards) compatibility. It is also found at different places on
the path.

As to the "study" that started this thread, clearly 100% pure garbage out.

Tam

 

[Stephane CHAZELAS]

2008-11-29, 16:23(+00), Tam Ha:

Can you explain this? Bourne is always more portable than Bash.
That's why you'll find experienced shell programmers writing everything
that doesn't absolutely require a bash feature in /bin/sh. Boot scripts,
install scripts, etc. should never be written in bash and if where you
find one using bash you can be sure a Linux-only newbie has written it.
For one there are too many versions of bash, for two it is not installed
by default on every Unix/Linux OS, for three it has poor backwards
(and forwards) compatibility. It is also found at different places on
the path.

[...]

There's a common confusion in this in the nature of /bin/sh.
There's no standard (neither POSIX nor Unix) that specifies that
/bin/sh should be any variant of the Bourne shell. Over the
years and on the different flavours of Unix, /bin/sh has been a
Thomson shell, Mashey shell, Bourne shell and all its variants,
Almquist shell, bash, ksh, zsh...

For a very long time, on a wide range of Unices, it has been the
Bourne shell (modified slightly in different ways by the
different Unix vendors), so that's probably why it is still
nowadays confused with the Bourne shell even though it is still
so only on very few Unices (mainly only Solaris, Tru64 and some
SCO OSes).

Nowadays, what the POSIX and Unix standards say is that a conformant Unix
should have somewhere (and the location is not specified but on
most Unices except those quoted above is generally /bin) a
command called "sh" that is an interpreter of the POSIX shell
language it defines.

Shells that do implement an interpreter for that language
(though most shells of course have extensions which the
specification doesn't prohibit, such as many C compilers (like
gcc) have extensions to the standard C language as published by
the ISO) include (with lesser or greater accuracy) bash (2 or
above), the AT&T Korn shell (88<x> or 93<x> versions) and its
variants, the public domain Korn shell and its derivatives such
as posh, mksh or OpenBSD sh, modern variants of the Almquist
shell such as the NetBSD sh or dash.

That shell language is based on a subset of the ksh88 language
and the Bourne shell is not a conformant implementation. Amongst
the feature of the POSIX/Unix sh that are not in the Bourne
shell are the $(...) form of command substitution, arithmetic
expansions as in $((...)), the processing of IFS that differs,
redirections of compound commands that no longer invoke a
subshell... That language is mostly backward compatible with the
Bourne syntax though there are a few small differences.

To sum up, /bin/sh is no longer the Bourne shell. It is only so
on very old systems and on systems that have chosen to keep the
Bourne shell as /bin/sh for backward compatibility (and for
those, the standard sh is in another place such as
/usr/xpg4/bin on Solaris). Free software Unices have never had
the Bourne shell as their /bin/sh as the source code of the
Bourne shell has not been released until recently (except for
some early versions of BSD and they were then breaching the
law).

 

[Xah Lee]

Great to see quality post from real expert once in a while. Thanks!

Xah
∑ http://xahlee.org/

☄

 

[Tam Ha]

There's a common confusion in this in the nature of /bin/sh.
There's no standard (neither POSIX nor Unix) that specifies that
/bin/sh should be any variant of the Bourne shell.

Sure there is, POSIX. Or rather their Austin Group. And while they done
an extremely poor job of it the POSIX shell is still the Bourne shell,
/bin/sh as it's always been known, with a few enhancements such as tilde
expansion and such.

Over the years and on the different flavours of Unix, /bin/sh
has been a Thomson shell, Mashey shell, Bourne shell and all its
variants, Almquist shell, bash, ksh, zsh...

Zsh? Seriously? Where? The fact that various flavors of Linux have
used various poorly designed "compatiblity mode" hacks to circumvent
Bourne/POSIX regardless of compatibility consequences does not change
the fact that /bin/sh is still there in all distributions, is still the
cross-platform compatibility winner by a county mile, and can still be
relied on for shell scripts.

should have somewhere (and the location is not specified but on
most Unices except those quoted above is generally /bin) a
command called "sh" that is an interpreter of the POSIX shell
language it defines.

Right, that's /bin/sh. It's common name is the Bourne shell even though
it is not the code that Mr Bourne wrote years ago, and even though
GNU/POSIX and ATT/David Korn want to remake it into a "POSIX shell"
regardless of the compatibility consequences.

That shell language is based on a subset of the ksh88 language
and the Bourne shell is not a conformant implementation.

Only if you limit your use of the term "Bourne shell" to something
that the rest of us don't recognize as a Bourne shell. The fact that
POSIX has been known to jerry-rig ksh features (most of which werre
derived from tcsh by the way) into /bin/sh (and sort, and a few other
APIs they really should not be changing) only reflects the fact that
A) David Korn (still with ATT) has been pushing his vision, and B)
systems administrators (the ones who suffer most from cross-platform
incompatibilities) are not well represented within POSIX in general,
and not well represented within its Austin Group in particular.

Tam

 

[Stephane CHAZELAS]

2008-11-30, 06:11(+00), Tam Ha:

Sure there is, POSIX. Or rather their Austin Group. And while they done
an extremely poor job of it the POSIX shell is still the Bourne shell,
/bin/sh as it's always been known, with a few enhancements such as tilde
expansion and such.

No, saying that /bin/sh is the Bourne shell is as wrong as
saying /bin/csh the Thomson shell. It's at best a very
inaccurate approximation but it's true it's a very commonly made
mistake. All you can say is that "sh" is meant to be a POSIX
compliant shell (which the Bourne shell is not) or that /bin/sh
(nowadays) is a Bourne-like shell. If you want to write a shell
script for the /bin/sh of any modern Unix, you have to write it
in the language that is the intersection of that of the Bourne
shell and the POSIX shell syntax.

As I said, the POSIX shell specification is based on a subset of
ksh88, has incompatibilities with the Bourne shell. The Bourne
shell, as can still be found on some systems either in some
non-standard place (/bin on Solaris, /usr/old/bin on HPUX) or
named differently (see
http://www.in-ulm.de/~mascheck/various/shells/ for more details)
is not conforming to that standard.

On most commercial Unices, the "sh" utility is implemented by
one version of the other of ksh88 (possibly slightly modified).
That's the case at least of Solaris (where it is in
/usr/xpg4/bin/sh, and you can see that /usr/xpg4/bin is before
/bin in the output of "getconf PATH"), HPUX since at least
10.10, AIX since at least 4.3.2, IRIX...

See on Tru64 http://nixdoc.net/man-pages/Tru64/sh.1.html:

sh(1)> [Tru64 UNIX] Subsequent shells spawned from the initial
sh(1)> shell depend on the value in the environment variable
sh(1)> BIN_SH. If this variable is set to xpg4, the POSIX shell
sh(1)> is started. If this variable is set to svr4, an SVR4
sh(1)> compliant version of the shell is started. If this
sh(1)> variable is unset, the Bourne shell is started. If this
sh(1)> variable is set to any other value, an error is
sh(1)> reported and the results are unpredictable. See the
sh(1)> EXAMPLES section for information on setting this
sh(1)> variable.

The POSIX one being ksh. On Linux and all the GNU OSes, it's
generally bash. On BSDs, it initially was the Almquist shell, a
public domain rewriting of the System V shell (the Bourne shell)
with a few modifications, and nowadays FreeBSD and NetBSD have
adapted that ash so as to become POSIX compliant, and OpenBSD is
using pdksh. MacOS/X has been using zsh and now bash

Zsh? Seriously? Where?

Some versions of MacOSX/Darwin.

used various poorly designed "compatiblity mode" hacks to circumvent
Bourne/POSIX regardless of compatibility consequences does not change
the fact that /bin/sh is still there in all distributions, is still the
cross-platform compatibility winner by a county mile, and can still be
relied on for shell scripts.

Could you please clarify what you mean? Yes, most systems have a
/bin/sh (for those that have a /bin directory), but as I said
/bin/sh can be any of a Thomson, Mashey (only in museums
nowadays), Almquist, Bourne, Korn, bash...

Nowadays, they are most likely to be Bourne-like.

If you want to write a portable script, today, the best advice
to give is to write it in a syntax conforming to POSIX (note
that of course that applies to the shell but also to the
standard utilities). Then you'll be guaranteed to find a "sh" on
any compliant Unix (most at least try to be nowadays) that will
interpret it. It is a much better situations than a few decades
ago where every vendor had its own variant of the Bourne shell
and the utilities.

Right, that's /bin/sh.

Not always. As I said, on a few systems, the POSIX sh is not in
/bin. That's the case of at least Solaris, where /bin/sh is a
Bourne shell and therefore not a POSIX compliant sh.

It's common name is the Bourne shell even though
it is not the code that Mr Bourne wrote years ago, and even though
GNU/POSIX and ATT/David Korn want to remake it into a "POSIX shell"
regardless of the compatibility consequences.

Saying that would be denying the reality.

Only if you limit your use of the term "Bourne shell" to something
that the rest of us don't recognize as a Bourne shell. The fact that
POSIX has been known to jerry-rig ksh features (most of which werre
derived from tcsh by the way) into /bin/sh (and sort, and a few other
APIs they really should not be changing) only reflects the fact that
A) David Korn (still with ATT) has been pushing his vision, and B)
systems administrators (the ones who suffer most from cross-platform
incompatibilities) are not well represented within POSIX in general,
and not well represented within its Austin Group in particular.

[...]

A few tcsh features have made it to bash and even more to zsh
but mostly only on the interactive usage side not the syntax
but hardly any to ksh, ksh has added a few features from csh
such as tilde expansion, brace expansion (though differently),
FIGNORE.

The reality is that nowadays the Bourne shell has almost
disappeared, but it's syntax is still used in scripts, which is
most cases, modern shs can still interpret.

I would strongly recommend the reading of
http://www.in-ulm.de/~mascheck/bourne/ and
http://www.in-ulm.de/~mascheck/various/shells/ which is a very
good reference on sh and the Bourne shell.

 

[Stephane Chazelas]

2008-11-30, 06:11(+00), Tam Ha:

Sure there is, POSIX.

[...]

And on this. First, POSIX has no juridiction on defining the
Bourne shell, let alone its location as the Bourne shell is
completely outside POSIX, it's a legacy shell and as been for
years. Then, POSIX makes it clear that the location of its sh
utility (again, which the Bourne shell is not) is unspecified,
only that the lookup of "sh" via $PATH in a conformant
environment should resolve to a conformant "sh".

And if you need some examples to be convinced, see the different
choices of those 4 different POSIX conformant Unices:

Solaris (7, 8, 9 at least):
/bin/sh: legacy Bourne/SVr4 shell
/usr/xpg4/bin/sh: a POSIX compliant shell (actually ksh88)
Solaris choice was to keep /bin/sh as the Bourne shell for
backward compatibility (as changing it for a POSIX shell
introduces a slight chance of breaking some existing scripts
as there are a few corner-case areas where the POSIX shells
are not backward compatible with the Bourne shell), and make
/bin a non standard place. How to get a conformant environment
(typically one where /usr/xpg4/bin/sh is before /bin) is
described in standards(5). The default one is not, which makes
Solaris a pain when porting scripts.

HPUX (10.10 and newer at least):
/usr/bin/sh (and /bin/sh if there's a symlink /bin ->
usr/bin): a POSIX compliant shell (based on ksh88)

/usr/old/bin/sh: the Bourne shell (though I think nowadays,
it's part of an optional package).

Tru64:
/bin/sh can behave either as a Bourne shell or a POSIX shell
(ksh88) depending on the environment

most Linux based systems and most other Unices:
/bin/sh is a POSIX compliant shell (bash generally for Linux)
and there's no Bourne shell (either there has never been or it
has been removed/replaced by a POSIX shell).

 

[Sven Mascheck]

The Bourne shell, as can still be found on some systems either in some
non-standard place (/bin on Solaris, /usr/old/bin on HPUX) or named
differently [...]

What do you mean with "non-standard place" here?

 

[Stephane CHAZELAS]

2008-12-1, 01:10(+01), Sven Mascheck:

The Bourne shell, as can still be found on some systems either in some
non-standard place (/bin on Solaris, /usr/old/bin on HPUX) or named
differently [...]

What do you mean with "non-standard place" here?

It's true it was vague and misleading,

/bin is not the standard place to look for "sh" as far as the
"POSIX" standard is concerned. That doesn't mean that standard
commands (POSIX or not) cannot be found in /bin. But /bin/sh has
been made a non-standard place for "sh". It's the "legacy"
place. Those commands that have not undergone non-backward
compatible changes with POSIX can still be found in /bin, the
others are to be found in /usr/xpg<x>/bin (for the standard
versions).

 

[Casper H.S. Dik]

It's true it was vague and misleading,

/bin is not the standard place to look for "sh" as far as the
"POSIX" standard is concerned. That doesn't mean that standard
commands (POSIX or not) cannot be found in /bin. But /bin/sh has
been made a non-standard place for "sh". It's the "legacy"
place. Those commands that have not undergone non-backward
compatible changes with POSIX can still be found in /bin, the
others are to be found in /usr/xpg<x>/bin (for the standard
versions).

The proper place is "`getconf PATH`". POSIX doesn't list any
pathnames.

Of course, the problem is then "where did getconf come from?".

Casper

 

[Andre Majorel]

2008-11-30, 06:11(+00), Tam Ha:

Sure there is, POSIX.

[...]

And on this. First, POSIX has no juridiction on defining the
Bourne shell, let alone its location as the Bourne shell is
completely outside POSIX, it's a legacy shell and as been for
years. Then, POSIX makes it clear that the location of its sh
utility (again, which the Bourne shell is not) is unspecified,
only that the lookup of "sh" via $PATH in a conformant
environment should resolve to a conformant "sh".

And if you need some examples to be convinced, see the different
choices of those 4 different POSIX conformant Unices:

Solaris (7, 8, 9 at least):
/bin/sh: legacy Bourne/SVr4 shell
/usr/xpg4/bin/sh: a POSIX compliant shell (actually ksh88)
Solaris choice was to keep /bin/sh as the Bourne shell for
backward compatibility (as changing it for a POSIX shell
introduces a slight chance of breaking some existing scripts
as there are a few corner-case areas where the POSIX shells
are not backward compatible with the Bourne shell), and make
/bin a non standard place. How to get a conformant environment
(typically one where /usr/xpg4/bin/sh is before /bin) is
described in standards(5). The default one is not, which makes
Solaris a pain when porting scripts.

HPUX (10.10 and newer at least):
/usr/bin/sh (and /bin/sh if there's a symlink /bin ->
usr/bin): a POSIX compliant shell (based on ksh88)

/usr/old/bin/sh: the Bourne shell (though I think nowadays,
it's part of an optional package).

Tru64:
/bin/sh can behave either as a Bourne shell or a POSIX shell
(ksh88) depending on the environment

How does it decide ? argv[0] ? isatty (STDIN_FILENO) ?

most Linux based systems and most other Unices:
/bin/sh is a POSIX compliant shell (bash generally for Linux)
and there's no Bourne shell (either there has never been or it
has been removed/replaced by a POSIX shell).

Two other Unixen I know of that have a non-POSIX sh are SCO Open
Server and UnixWare. I'm not sure they even provide a
POSIX-compliant alternative like Solaris does.

On the other hand, their utilities (awk, grep, sed et al.) seem
to be POSIX-compliant, at least to the extent that I've always
managed. Unlike Solaris where I've often had to use the ones in
/usr/xpg4/bin.

Many people assume "standard" equates "portable". As far as
shell programming is concerned, the safest way to get portability
is to ignore the standard and code for the Bourne shell.

 

[Stephane Chazelas]

2008-12-01, 08:51(+00), Casper H.S Dik:

The proper place is "`getconf PATH`". POSIX doesn't list any
pathnames.

Of course, the problem is then "where did getconf come from?".

[...]

Sorry, I was specifically speaking of Solaris 7, 8, 9. "getconf
PATH" is not enough, you have to pick the correct getconf, and
that getconf be called in the relevant environment. For
instance, in later versions of Solaris, I beleive you get
/usr/xpg6/bin or /usr/xpg4/bin first depending on the
environment.

Also, PATH is not necessarily the only thing needed to get you
the conformant utilities. See BIN_SH on Tru64, POSIXLY_CORRECT
on GNU...

I beleive the only thing POSIX requires is that how to get into
the proper environment be documented (but it is unspecified).
On Solaris, I beleive it's in the standards(5) man page.

 

[Stephane Chazelas]

2008-12-1, 10:16(+00), Andre Majorel:
[...]

Tru64:
/bin/sh can behave either as a Bourne shell or a POSIX shell
(ksh88) depending on the environment

How does it decide ? argv[0] ? isatty (STDIN_FILENO) ?

That was answered in another article with a quote of the sh man
page on Tru64: via an environment variable: BIN_SH.

Two other Unixen I know of that have a non-POSIX sh are SCO Open
Server and UnixWare. I'm not sure they even provide a
POSIX-compliant alternative like Solaris does.

Again, see http://www.in-ulm.de/~mascheck/various/shells/

[...]

Many people assume "standard" equates "portable". As far as
shell programming is concerned, the safest way to get portability
is to ignore the standard and code for the Bourne shell.

Yes, it's enough and relatively safe in most cases. It's the
only option if you want to be portable to systems over 15 years
old.

But nowadays, especially since the specifications have been made
public, I find that it is sufficiently widespread to make POSIX a
safer bet. At least having a specification gives you some sort
of guarantee: if you write your script by the rules that are
clearly (most of the time) specified, then it should work on any
compliant system. If it doesn't work, then it's not your fault
(though it's true in many case, you'll still be the one who'll
have to do something about it).

 

[Jorgen Grahn]

["Followup-To:" header set to comp.unix.shell.]

Can you explain this?

Sorry for the late answer. No, it's actually what I said:

I could get away with using bash in these cases, so I did it. It was
an in-house application, it had already unknown dependencies to bash,
and to Linux versions of other utilities. If I had asked for time to
modify it to make sure it was portable Bourne shell, people would have
laughed at me.

Bourne is always more portable than Bash.
That's why you'll find experienced shell programmers writing everything
that doesn't absolutely require a bash feature in /bin/sh. Boot scripts,
install scripts, etc. should never be written in bash and if where you
find one using bash you can be sure a Linux-only newbie has written it.

Sure, nothing controversial about that. I don't argue that people
should use bash features in any random script they write. I just noted
that if you decide to use it, it's a pretty useful language. Probably
more useful than Python in my case, where most of the work was about
starting and managing external commands and pipelines.

For one there are too many versions of bash, for two it is not installed
by default on every Unix/Linux OS, for three it has poor backwards
(and forwards) compatibility.

Worse compatibility than Perl or Python? The Bourne shell timescale
is probably impressive, but often you aren't interested in decades.

It is also found at different places on
the path.

Surely that applies to almost any interpreter, like perl and python.
It's a problem (on the non-free Unixes at least) but if you let it be
the deciding factor, you could use no scripting language except
/bin/sh and awk.

/Jorgen