Pythonic indentation (or: beating a dead horse)

[James Britt]

You are proposing a Rube Goldberg contraption whereby the scanner would
require some degree of grammatical awareness. This is a complex solution.
The minor flaw is that you don't like "end" statements. Is the minor flaw
of having repeated end statements really worth the level of complexity the
solution would require?

Complexity never goes away (I imagine there is some Conservation of
Complexity defined someplace), so you have to decide where you want it.

If you push it down into the parser, you may gain something at the
code-writing level, but odds are that when a new feature is proposed, or
a bug discovered, adding or fixing things will be much, much harder.

A magic whitespace addition could be a Pyrrhic victory.


--
James Britt

www.jamesbritt.com - Playing with Better Toys
www.ruby-doc.org - Ruby Help & Documentation
www.rubystuff.com - The Ruby Store for Ruby Stuff
www.neurogami.com - Smart application development

 

[Ryan Davis]

And I guess I just can't shut up about this: your solution would
require a
scanner with some degree of grammatical awareness for every
expression which
contains an indent block. When you start talking about a solution
like
that, I really think you're going down a rabbit hole.

To be fair, if you looked at ruby's lexer and parser you'd know that
we're already down a rabbit hole. Your statement completely ignores
the complexity already inherent in the language.

Don't chalk me up as a proponent of this idea as I've written more
than my fair share of python and hate the language (and especially the
libraries).

 

[Tony Arcieri]

[Note: parts of this message were removed to make it a legal post.]

To be fair, if you looked at ruby's lexer and parser you'd know that we're
already down a rabbit hole.

Oh yes, things like Ruby's lexer/parser feedback involved in interpolated
strings makes my eyes bleed. That's sure begging for a PEG.

That said, as messy as lexer/parser feedback is we're talking about
something worse here: including grammatical knowledge in the scanner which
cannot be expressed as a CFG at all. As bad as Ruby already is, the
solution is even messier than anything which already exists.

It would involve some ad hoc grammatical analysis by the lexer for the
purposes of having a pushdown deeper in the stack terminate one which is
higher up by sneakily injecting "end" tokens. Aiee!

 

[Yossef Mendelssohn]

(Side rant: interesting that so many Rubyists go ga-ga over magic
indentation in Yaml and Haml, but find the idea repulsive in Ruby.)

YAML and Haml are fundamentally different from Ruby, or do you
consider markup to be programming?
It's true that Haml has the same any-Ruby-is-allowed rules as ERB, but
you naturally restrict yourself because you're working with a template
as opposed to an actual program. (Or at least you should.)

And for the record, I get kind of annoyed with the significant
whitespace in Haml when it comes to Ruby code, but I love how easy and
clean HTML becomes.

 

[Tony Arcieri]

[Note: parts of this message were removed to make it a legal post.]

It would involve some ad hoc grammatical analysis by the lexer for the
purposes of having a pushdown deeper in the stack terminate one which is
higher up by sneakily injecting "end" tokens. Aiee!

Perhaps a minimalist example of the "expression problem" is in order.

Take this example (in normal Ruby):

result = 2 * [1,2,3].inject(0) do |a, b|
a + b
end
puts result

prints "12"

Here the 2 * takes precedence over the method invocation (with a block).
Furthermore, nothing precludes us from sticking additional expressions at
the end of the block:

result = 2 * [1,2,3].inject(0) do |a, b|
a + b
end + 1
puts result

Here the 2 * [method_with_block] is lower precedence than the plus operand,
so we get a parse tree ala:

(+ (* 2 [method_with_block]) 1)

And that doesn't even include setting "result"!

But through funky lexer tricks, we want indent blocks to terminate the
entire expression stack somehow. We want [method_with_block] when it hits a
dedent to pop all the way up the pushdown stack. It also means that a
method with a block must always be the rightmost operand in any pushdown.

That is not an easy problem to solve, and afaict cannot be done in a context
free grammar. It also limits some of the expressivity that Ruby normally
provides. Quite frequently I do something like:

str = [a, b, c].map do |x|
thing1 = do_something_to x
thing2 = do_something_else_to x
"#{thing1}:#{thing2}"
end.join(",")

An ident-sensitive grammar would preclude that sort of thing.

But hey, if anyone's still about trying to make an indent-sensitive Ruby,
here's a nice simple example based on the one above to cut your teeth on:

result = 2 * [1,2,3].inject(0) do |a, b|
a + b
puts result

Parse that. I dare ya.

 

[Steven Arnold]

Quite frequently I do something like:

str = [a, b, c].map do |x|
thing1 = do_something_to x
thing2 = do_something_else_to x
"#{thing1}:#{thing2}"
end.join(",")

Me too. It's interesting how seemingly small changes can generate a
lot of unexpected corner cases and unforeseen consequences. Of
course, one could argue that using indentation does not preclude using
explicit blocks in the same file. I doubt, for example, that anyone
wishing to remove the 'end' keyword would wish to disallow something
like:

array_of_strings_that_are_numbers.sort! {|a,b| a.to_i <=> b.to_i}

...even though the {} brackets are the same as do..end.

If explicit blocks were allowed in the same file where blocks were, by
default, determined by indentation, then you could have your cake and
eat it too in the example above.

Nevertheless, I think your gut feel is correct......this would be a
tricky thing to do right.

steven

 

[Tony Arcieri]

[Note: parts of this message were removed to make it a legal post.]

Okay. But just to recap what's gone before: you've said all along that
Pythonic indentation in Ruby was impossible. It's impossible, because
it's incompatible with blocks. It's impossible, because everything's
an expression. It's impossible, because Guido said so. And I've asked
you over and over for an example demonstrating its impossibility.
You've had time to think it over, you were under no constraints, and
this is what you've come up with.

My claims were it's impossible with a Pythonic lexer and a backing context
free grammar. I certainly didn't claim that if you throw enough regexes at
the problem it won't go away.

I found the old preprocessor script from awhile ago that I mentioned
earlier in this thread. I've pastebinned it at
http://pastebin.com/m5fee1e92.
It's very short, and pretty simple. I have no doubt that it would need
a lot of work to be robust. (In particular, I think things need to be
added to the enumeration at line 68.)

But Tony, it blasted through your "impossible" example without
skipping a beat, working perfectly on the first try.

Well great! I'm not really sure how that script works, but cool. However,
what you have there is a marked departure from how Python actually works.
But if you're happy with it, great. Go for it and see how popular you can
make it.

 

[Tony Arcieri]

[Note: parts of this message were removed to make it a legal post.]

Okay. But just to recap what's gone before: you've said all along that
Pythonic indentation in Ruby was impossible. It's impossible, because
it's incompatible with blocks. It's impossible, because everything's
an expression. It's impossible, because Guido said so. And I've asked
you over and over for an example demonstrating its impossibility.
You've had time to think it over, you were under no constraints, and
this is what you've come up with.

And sorry to be frank about this: I've been cordial and fact oriented in
this discussion, trying to explain my opinion as best as possible. Not only
are you strawmanning me here, you're being a right ****. Perhaps you could
try to express your ideas without making it personal?

 

[Joshua Ballanco]

As I've said repeatedly, I don't follow your reasons for claiming that
this is impossible. Can you come up with a better demonstration?

First, let me say thank you for the code. This community thrives on
code, not on bickering. More code is always a good thing. That said...

cat tough.rb

# "end" can make things clearer, sometimes...

result = case ARGV[0]
when /^\d*$/
"That's an integer!"
when /^[A-Za-z]*$/
"That's a word!"
else
"That's something that's not an integer or a word..."
end if ARGV[0]

puts result

 

[Jörg W Mittag]

The Ant XML example is a bit ironic. Ant's creator regrets using
XML. He believes that he should have used a scripting language
instead. Take a look at Rake and you'll see a DSL that does what Ant
does and more.

JDD actually not only regrets using XML for Ant, he eclusively uses
Rake now to build his Java projects. At least that's what I read,
anyway.

jwm

 

[Reid Thompson]

Well great! I'm not really sure how that script works, but cool.
However,
what you have there is a marked departure from how Python actually
works.
But if you're happy with it, great. Go for it and see how popular you
can
make it.

It doesn't really. Or at least only simplistically; by re-writing the
input code, attempting to place 'end' in the proper place based on
indentation. It breaks silently if you happen to misplace your
whitespace incorrectly, placing an 'end' in the wrong place.

 

[Roger Pack]

n = 2 * -[1,2,3,4,5].inject(0) do |a, b|:
c = a * 2
a * c + b
result = 8 * -if n > 0:
-n += 10000
else:
n + 500000
puts result

Once again, worked perfectly on its first time through the
preprocessor.

It would appear that "end's" actually help with readability in a few
cases

I'm glad to see you have some code. Now put it on github, make a gem of
it, and advocate it

Here are a few other thoughts on it. (Keeping things friendly and
civil...)

1) maybe it could become more of a generic preprocessor? (i.e. you give
it arbitrary rules, like "see this, change it to this")?

2) maybe it could be even smarter about its extrapolation, i.e. allow

4.times {|n|:
puts n

[accomodate for braces--one fewer keystroke!]

or even

4.times |n|:
puts n

[put in its own do's and end's if not there]

or who even needs :'s?

4.times |n|
puts n

[whenever it sees an ending | without preceding do or {, count it as an
indent based block]

That would be interesting, and much more whitespacey, though slightly
less ruby-y .

To clarify Tony's point from above, you can't really do case statements
using only indentation and :'s. Other constructs would work though
(except for the ambiguities mentioned earlier might prevent anythin but
a preprocessor being able to, for ruby's case).

-=r

 

[J Haas]

My claims were it's impossible with a Pythonic lexer and a backing context
free grammar. I certainly didn't claim that if you throw enough regexes at
the problem it won't go away.

Tony, while I may be a newb to Ruby, I assure you that I'm no stranger
to mailing lists and Usenet, and I'm aware of these things called
"archives". Perhaps you are too. If not, allow me to demonstrate:

These are both incompatible with a Python-style indentation sensitive
syntax. You can have the Pythonic indent syntax or a purely expression
based grammar with multi-line blocks. You can't have both.

And yet here I've _demonstrated_ having both, while repeatedly asking
you for examples to exhibit the "incompatibility", and repeatedly
shooting down every one you've so proudly produced. In the quoted text
above, please note the absence of words like "lexer" and "backing
context free grammar". In fact, you were quite explicit about Pythonic-
style indentation _syntax_ being the problem. You stood behind that
claim several times. Now tell me, Tony, was that claim correct, or was
it wrong?

Now getting back to your amended claim, that it's impossible with a
Python lexer and a backing context free grammar, well, let's not
forget that Ruby-style blocks themselves are impossible. Yes,
completely impossible! Can't be done! You just can't have Ruby-style
blocks, there's no way to make them work... with the Commodore BASIC
interpreter, a ham sandwich, and half a ton of elephant dung. I
realize that this is a bold claim to make, but I'm gonna stand by it
without fear.

As I've said before, this isn't Python. And whether or not my proposal
would be possible in a Python parser is of absolutely no interest to
me. Clearly it _is_ possible in Ruby's parser, it's even possible in
Ruby itself.

Well great! I'm not really sure how that script works, but cool. However,
what you have there is a marked departure from how Python actually works.

Oh noes!! You mean my Ruby script is a marked departure from Python?
Heaven forfend!

And sorry to be frank about this:

No, you're not. Not even a tiny bit.

I've been cordial and fact oriented in
this discussion,

Actually, Tony, I've found you to be arrogant and condescending
throughout. I've merely responded in kind.

Not only are you strawmanning me here, you're being a right c--t.

I'm strawmanning? *I'm* strawmanning?? Thanks for the tip, Mr. Your-
Proposal-Is-Impossible-But-Only-If-I-Set-Ridiculous-Conditions-Such-As-
Requiring-Python's-Parser-For-Some-Reason. And have you forgotten
this?

And a bit of supplemental information: I conducted a poll of what Rubyists'
favorite features are in the language. Blocks were #1 by a wide margin.

Clearly, this one deserves at least honorable mention on the all-time
list of most flagrant strawmen. I think you're projecting.

And by the way, I've heard that in other English-speaking countries,
the profanity you used is not considered as shockingly offensive as it
is in the United States. Perhaps you're from one of those countries
and are unaware that in America it's among the most vile of curse
words. I know that you, with your emphasis on cordiality and fact-
orientation, will appreciate me bringing this to your attention.

That being said I will plead guilty to being a terrible spokesman for
my ideas (and this extends well beyond the current discussion) because
I lack tact, empathy, and sensitivity... in short, I'm an asshole (and
not, as you suggested, someplace a couple of inches anterior.) This
has bitten me more times than I can say, but in the end, I gotta be me.

 

[J Haas]

It doesn't really.  Or at least only simplistically; by re-writing the
input code, attempting to place 'end' in the proper place based on
indentation.  It breaks silently if you happen to misplace your
whitespace incorrectly, placing an 'end' in the wrong place.

As I said when I posted it:

It's very short, and pretty simple. I have no doubt that it
would need a lot of work to be robust.

In its present form, this is not something I want to do in production
code. It does however serve at least one valuable purpose: it
demonstrates that the code snippet that Tony posted, that he
specifically contrived as an example of a piece of Ruby code that
would be utterly impossible to parse without 'end' statements, was in
fact quite easily parsable.

 

[Roger Pack]

Tony, while I may be a newb to Ruby, I assure you that I'm no stranger

to mailing lists and Usenet, and I'm aware of these things called
"archives". Perhaps you are too. If not, allow me to demonstrate:

Tony's point was that certain constructs, like case statements, won't be
transformable into indentation only blocks. Does that make sense?
If you feel frustrated then maybe take a walk (to cool down) before
posting
Take care.
-=r

 

[Tony Arcieri]

[Note: parts of this message were removed to make it a legal post.]

That being said I will plead guilty to being a terrible spokesman for
my ideas (and this extends well beyond the current discussion) because
I lack tact, empathy, and sensitivity... in short, I'm an asshole (and
not, as you suggested, someplace a couple of inches anterior.) This
has bitten me more times than I can say, but in the end, I gotta be me.

Well, everything else aside: if you can't present your ideas or engage in
discussion without being an asshole, you're going to have a hard time
getting people to listen to you. I've been trying to give constructive
criticism, but when you come back at me with personal attacks I'm left to
wonder why I should even bother.

 

[Michael Bruschkewitz]

And I specifically addressed this in my OP. A lot of engineers who
haven't done work in Python think that the idea of giving syntactical
significance to whitespace is a ridiculous idea. I used to be one of
them. Now I know better. Don't knock it until you've tried it.

I've used many languages in team projects, for example Assembler, Fortran,
Modula, C, C++, Pascal, Ruby, Batch, bash, TCL.
The only thing common to these projects: Nobody seems to be able to use
formatting consistently on different platforms.

Aside from the fact that this wouldn't solve the problem that I'd
still have to _write_ the damned things, this is possibly the
kludgiest solution imaginable.

So just use a macro which replaces CTRL-BS by end.

The easiest way to solve _my_ problem would be for me to use one of
the preprocessing scripts that impose Python-like indentation syntax
on Ruby.

You will additionally need preprocessing for importing code.

But that wouldn't solve the larger problem, which is that
Ruby could be better than it is.

Then it would not be your problem anymore, but the problems of tons of other
Rubyists. Which, at current time, probably have absolutely no problem using
end or "}".
(BTW: For at least 10 years, I've never read or heard about this suggestion
for Ruby.)

Are you really sure you are no FUD-guy paid by MS which tries to
spoil other scripting languages in favor of F#?

 

[James Britt]

It doesn't really. Or at least only simplistically; by re-writing the
input code, attempting to place 'end' in the proper place based on
indentation. It breaks silently if you happen to misplace your
whitespace incorrectly, placing an 'end' in the wrong place.

This is an interesting aspect of indentation-sensitive syntax.

I've had Yaml files that, technically, were valid Yaml, but I had
mistakenly added some extra indentation. I then had to puzzle over some
odd errors when running my program.

It seems like an easy enough mistake to make, with the unfortunate risk
of creating proper Yaml, so you don't get any errors from the parser.

I've wondered if this sort thing is a problem in Python; perhaps it is
less likely there that any accidental indent or out-dent will still give
you well-formed code.



--
James Britt

www.jamesbritt.com - Playing with Better Toys
www.ruby-doc.org - Ruby Help & Documentation
www.rubystuff.com - The Ruby Store for Ruby Stuff
www.neurogami.com - Smart application development

 

[James Britt]

Tony, while I may be a newb to Ruby, I assure you that I'm no stranger
to mailing lists and Usenet, and I'm aware of these things called
"archives". Perhaps you are too. If not, allow me to demonstrate:

# rampant snideness elided

However justified you may feel, you're now presenting yourself really
poorly here.

You may disagree that Tony was being civil (he certainly struck me as
civil), but replying to perceived offense with even more rudeness is
tacky. Worse, it seems a good way to kill what could continue to be a
good thread.

Tacking on self-deprecating comments doesn't change that, either.



--
James Britt

www.jamesbritt.com - Playing with Better Toys
www.ruby-doc.org - Ruby Help & Documentation
www.rubystuff.com - The Ruby Store for Ruby Stuff
www.neurogami.com - Smart application development

 

[Reid Thompson]

I've wondered if this sort thing is a problem in Python; perhaps it is
less likely there that any accidental indent or out-dent will still give
you well-formed code.

I've only minimally looked at python, but I'm pretty sure that it will throw an
error when indentation is fouled.