If you are unhappy with the direction of Ruby 1.8.7+, respond

[James Coglan]

[Note: parts of this message were removed to make it a legal post.]

Iteration order is undefined, even for hash literals.
This is on my machine:

$ ruby -v -e '{2 => 0, 1 => 0}.each_pair { |k,v| p k }'
ruby 1.8.6 (2007-09-24 patchlevel 111) [x86_64-linux]
1
2

Your code happened to work by chance.

To work with 1.8.7, you need to create an empty hash
and add keys one by one rather than putting them all in a hash
literal (see the patch).

Do I understand that the code relies on insertion order now?
Then it's still working by chance in case it works at all.

Stefan

+1

Mr. Coglan seems not to understand how a hash-table is implemented.
One cannot predict the iteration order because he cannot know in which
bucket each element has been stored. Furthermore, it makes no sense
at all to be concerned about iteration-order for a hash-table.

If one is concerned about iteration-order, he should use an
association-list.

I do know how hash tables work, I was responding to the question of whether
any code was broken by 1.8.7. I've conceded elsewhere in this thread that no
iteration order was ever guaranteed for 1.8, and that the question of
whether this counts as a true backward incompatibility is a grey area at
best.

I am using an ordered hash implementation, but was initialising it using
hash literals, which obviously does not work if iteration order is different
from source order. I've since fixed the bug by adding keys explicitly one by
one.

This begs the question of whether 1.9 should guarantee insertion order for
hashes. Since hashes are not inherently ordered beings, wouldn't it make
more sense to provide an OrderedHash subclass that explicitly provides that
behaviour? At least then it would be clearer if you were using a 1.9 feature
that won't work on 1.8.

 

[William James]

Granted. As I've said elsewhere, this is just my example of a
significant change in 1.8.7. I accept that no iteration order was
ever explicitly stated for 1.8, and a change like this is something
of a grey area in terms of whether it would be considered a backward
incompatibility.

No, it's not a gray area. Anyone who has a basic understanding of
hash-tables doesn't even think about the iteration order.

 

[James Coglan]

[Note: parts of this message were removed to make it a legal post.]

No, it's not a gray area. Anyone who has a basic understanding of
hash-tables doesn't even think about the iteration order.

Not true. Some languages (Ruby 1.9, most JavaScript implementations) iterate
over hashtables in insertion order. This is often done (as it is in Ruby) by
storing the table entries as a linked list so that every entry has pointers
to its neighbours. Though a basic hashtable implementation will not
guarantee iteration order, there is potential confusion if an implementation
*appears* to guarantee it. I referred to this is a grey area because 1.8
never made any such promises, it just *looked* like it did in some use cases
until 1.8.7.

Anyway, this has gone a little off-topic. As previously stated, I would
point out that http://github.com/jcoglan/packr/tree/3.1 still has a
regex-related bug under 1.8.7 that I've not found a fix for and that does
not appear in any other 1.8 or 1.9 release. If anyone has had regex issues
on 1.8.7 I'd really love some help fixing my bug!

 

[Mohit Sindhwani]

Engine Yard is willing to step up as the official maintainer of
ruby1.8.6. We can provide the right amount of human resources as well
as server and QA resources with automated CI running the rubyspecs.

I feel that it is very important to keep a stable ruby 1.8.6
lineage for some time to come.

Matz has stated that he is willing to pass the torch on
maintenance of ruby1.8.6 and we would like to step up to the plate for
the whole ruby community. Please respond here if you think Engine yard
would make good maintainers of the ruby1.8.6 lineage

Thanks

Yes... (and thanks!)

 

[Rick DeNatale]

[Note: parts of this message were removed to make it a legal post.]

Hi,

At Fri, 13 Feb 2009 09:16:39 +0900,
James Coglan wrote in [ruby-talk:327995]:

I realise that there is no explicit iteration order specified for hashes in
1.8. All I know is that this patch fixed by 1.8.7 bugs:

http://github.com/jcoglan/packr/commit/9ece59428bbd1bfd497b45a2ae4de2d163a24849

Basically involves inserting keys in an explicit order rather than using
hash literals. This project uses a bunch of custom collection classes but
they're basically wrappers for arrays and hashes that merge array and hash
functionality into a single class. The constructors would iterate over the
hashes provided and insert values, and this insertion order changed in
1.8.7.

Hash in 1.8 doesn't define any kind of "order". It's
definitely a bug in your code as you use the word "order".

Yes, and this particular kind of change, making something which was
previously undefined now be defined, seems to be a benign change, since the
defined behavior is within the 'definition' of the previous behavior.

It's the changing of defined behavior while implying compatibility which
causes problems, particulary when the change is significant enough, or there
are enough such changes. And when this threshold is crossed is, admittedly,
subjective.

 

[M. Edward (Ed) Borasky]

I have no problems with the 1.9 release. I feel like the reasoning has
been well explained and I understand it.

Essentially, a lot was promised for 2.0. About half of it has been
built. We could have kept waiting for the other half, or release what
was done. It's not all of 2.0 though, so it's instead called 1.9.

I suppose once all the smoke clears on the current discussion, dare I
ask about the road map for the 1.9.x --> 2.0 transition, and perhaps
even more important, the *process* for it?

<ducking>

 

[Rick DeNatale]

[Note: parts of this message were removed to make it a legal post.]

No, it's not a gray area. Anyone who has a basic understanding of
hash-tables doesn't even think about the iteration order.

Let me try to calm the waters on this particular part of the conversation.

A couple of years ago, there was a long discussion here, when some, led by
Hal Fulton IIRC, advocated defining the order for enumerating Ruby hashes.

At that time, I argued vehemently to your point that hash tables are
inherently unordered, and therefore it was inconceivable to order them.

I was wrong.

First, although Ruby Hash is implemented using a hash table, they aren't
hash tables, they are Ruby objects which for better or worse are named
simply Hash. Perhaps if they had been named something like Dictionary, or
Map, a name which doesn't betray the implementation, this all would have
been less controversial.

Sometime after that discussion, Matz, who I know has much more than a basic
understanding of hash-tables, decided that, yes,
defining the iteration order of Ruby hashes to be by insertion order, and
that this was easily done with minimal impact to performance/space
requirements by linking buckets in the internal hash table used to IMPLEMENT
Hash, and made this change in Ruby 1.9, which has been backported to Ruby
1.8.7.

And as for whether or not changing something which was previously undefined
to something which is defined introduces a backwards compatibility issue, I
think it's pretty clear that it does not.

 

[M. Edward (Ed) Borasky]

What about 1.8.6.1?

Someone suggested having 1.8.8 being 1.8.6 syntax and semantics with
only bug fixes, security fixes and efficiency enhancements. That appeals
to me ... that way, "1.8.7" is a development release and "1.8.8" is
stable. But I would think EngineYard would want to "brand" their ruby
1.8.6 implementation in a manner similar to Ruby Enterprise Edition.

BTW, Perl had no trouble going from 5.8.x to 5.10.x, and I understand
there is a 5.12.x coming. So I don't see a problem with Ruby doing
similar things in the third digit. 1.8.10 works for me, as does 1.9.10.

 

[Rick DeNatale]

[Note: parts of this message were removed to make it a legal post.]

This begs the question of whether 1.9 should guarantee insertion order for
hashes. Since hashes are not inherently ordered beings, wouldn't it make
more sense to provide an OrderedHash subclass that explicitly provides that
behaviour? At least then it would be clearer if you were using a 1.9
feature
that won't work on 1.8.

For Ruby 1.9, the issue is settled. Ruby 1.9 is a new and different Ruby
than Ruby 1.8.6.

To make an analogy from ancient history, there is a Rubicon between Ruby
1.8.6 and Ruby 1.9. Because of the consequences of crossing the Rubicon,
one should make a conscious decision to do so.

Ruby 1.8.7 represents the middle of the river.

 

[David A. Black]

Hi --

Sometime after that discussion, Matz, who I know has much more than a basic
understanding of hash-tables, decided that, yes,
defining the iteration order of Ruby hashes to be by insertion order, and
that this was easily done with minimal impact to performance/space
requirements by linking buckets in the internal hash table used to IMPLEMENT
Hash, and made this change in Ruby 1.9, which has been backported to Ruby
1.8.7.

Hashes aren't ordered in 1.8.7, though.


David

--
David A. Black / Ruby Power and Light, LLC
Ruby/Rails consulting & training: http://www.rubypal.com
Coming in 2009: The Well-Grounded Rubyist (http://manning.com/black2)

http://www.wishsight.com => Independent, social wishlist management!

 

[James Coglan]

[Note: parts of this message were removed to make it a legal post.]

Hi --

On Sat, 14 Feb 2009, Rick DeNatale wrote:

Sometime after that discussion, Matz, who I know has much more than a

Hashes aren't ordered in 1.8.7, though.

Seconded -- 1.8.7 is precisely where my code that was relying on insertion
order *does not work*. Actually let me be more specific. My code was relying
on keys being inserted in the order they appear in a hash literal, and on
subsequent iteration occuring in the same order. My own code was maintaining
key order explicitly using an array of keys (
http://github.com/jcoglan/packr/blob/3.1/lib/packr/collection.rb), but when
using a hash literal to initialize an instance, the keys are not inserted in
literal appearance order in 1.8.7.

So it may be that 1.8.7 does use insertion order for iteration, but does not
use literal appearance order when creating a new hash.

 

[David A. Black]

Hi --

Seconded -- 1.8.7 is precisely where my code that was relying on insertion
order *does not work*. Actually let me be more specific. My code was relying
on keys being inserted in the order they appear in a hash literal, and on
subsequent iteration occuring in the same order.

That doesn't happen in any 1.8 version, though. (Sorry; I feel like
something's going right past me, and I'll stop after this iteration



David

--
David A. Black / Ruby Power and Light, LLC
Ruby/Rails consulting & training: http://www.rubypal.com
Coming in 2009: The Well-Grounded Rubyist (http://manning.com/black2)

http://www.wishsight.com => Independent, social wishlist management!

 

[James Coglan]

[Note: parts of this message were removed to make it a legal post.]

That doesn't happen in any 1.8 version, though. (Sorry; I feel like
something's going right past me, and I'll stop after this iteration

Regardless of whether they're *supposed* to, the Rubies I've used before
1.8.7 actually do iterate in insertion order. Could be machine/OS specific,
a fluke, whatever. The point is that 1.8.7 definitely does not iterate in
the order that keys appear in a literal, at least on my machine. If this is
supposed to now be defined behaviour I'll investigate further and file a bug
if necessary.

 

[Tanaka Akira]

Regardless of whether they're *supposed* to, the Rubies I've used before
1.8.7 actually do iterate in insertion order. Could be machine/OS specific,
a fluke, whatever. The point is that 1.8.7 definitely does not iterate in
the order that keys appear in a literal, at least on my machine. If this is
supposed to now be defined behaviour I'll investigate further and file a bug
if necessary.

Would you show us an example which the insertion order is
preserved on 1.8.6 and not preserved on 1.8.7?

I tested as follows. The order is changed since 1.9.
1.8.7 is same as 1.8.6 and older.

% all-ruby -e 'h = {2=>"b", 1=>"a"}; p h'
ruby-1.4.6(2000-08-16) {1=>"a", 2=>"b"}
ruby-1.6.0(2000-09-19) {1=>"a", 2=>"b"}
ruby-1.6.1(2000-09-27) {1=>"a", 2=>"b"}
ruby-1.6.2(2000-12-25) {1=>"a", 2=>"b"}
ruby-1.6.3(2001-03-19) {1=>"a", 2=>"b"}
ruby-1.6.4(2001-06-04) {1=>"a", 2=>"b"}
ruby-1.6.5(2001-09-19) {1=>"a", 2=>"b"}
ruby-1.6.6(2001-12-26) {1=>"a", 2=>"b"}
ruby-1.6.7(2002-03-01) {1=>"a", 2=>"b"}
ruby-1.6.8(2002-12-24) {1=>"a", 2=>"b"}
ruby-1.8.0(2003-08-04) {1=>"a", 2=>"b"}
ruby-1.8.1(2003-12-25) {1=>"a", 2=>"b"}
ruby-1.8.2(2004-12-25) {1=>"a", 2=>"b"}
ruby-1.8.3(2005-09-21) {1=>"a", 2=>"b"}
ruby-1.8.4(2005-12-24) {1=>"a", 2=>"b"}
ruby-1.8.5(2006-08-25) {1=>"a", 2=>"b"}
ruby-1.8.5p52(2007-06-07) {1=>"a", 2=>"b"}
ruby-1.8.5p113(2007-09-23) {1=>"a", 2=>"b"}
ruby-1.8.5p115(2008-03-03) {1=>"a", 2=>"b"}
ruby-1.8.5p231(2008-06-20) {1=>"a", 2=>"b"}
ruby-1.8.5head(2008-06-20p231) {1=>"a", 2=>"b"}
ruby-1.8.6p0(2007-03-13) {1=>"a", 2=>"b"}
ruby-1.8.6p36(2007-06-07) {1=>"a", 2=>"b"}
ruby-1.8.6p110(2007-09-23) {1=>"a", 2=>"b"}
ruby-1.8.6p111(2007-09-24) {1=>"a", 2=>"b"}
ruby-1.8.6p114(2008-03-03) {1=>"a", 2=>"b"}
ruby-1.8.6p230(2008-06-20) {1=>"a", 2=>"b"}
ruby-1.8.6p287(2008-08-11) {1=>"a", 2=>"b"}
ruby-1.8.6head(2009-02-13p330) {1=>"a", 2=>"b"}
ruby-1.8.7p0(2008-05-31) {1=>"a", 2=>"b"}
ruby-1.8.7p17(2008-06-09) {1=>"a", 2=>"b"}
ruby-1.8.7p22(2008-06-20) {1=>"a", 2=>"b"}
ruby-1.8.7p72(2008-08-11) {1=>"a", 2=>"b"}
ruby-1.8.7head(2009-02-13p116) {1=>"a", 2=>"b"}
ruby-1.8(2009-02-05r22080) {1=>"a", 2=>"b"}
ruby-1.9.1p0(2009-01-30r21907) {2=>"b", 1=>"a"}
ruby-1.9(2009-02-11r22240) {2=>"b", 1=>"a"}

 

[Stefan Lang]

Regardless of whether they're *supposed* to, the Rubies I've used before
1.8.7 actually do iterate in insertion order.

NO THEY DO NOT. It is the nature of a regular hashtable that
the order of entries in its internal data structures depends
only on the hash function and its inputs. It doesn't know
anything about the order of entries in your hash literal.

I've shown you examples using Ruby *1.8.6* that clearly
demonstrate that it gives a **** about order of appearance
in a literal, or insertion order.

Stefan

 

[David A. Black]

Hi --

Regardless of whether they're *supposed* to, the Rubies I've used before
1.8.7 actually do iterate in insertion order. Could be machine/OS specific,
a fluke, whatever. The point is that 1.8.7 definitely does not iterate in
the order that keys appear in a literal, at least on my machine. If this is
supposed to now be defined behaviour I'll investigate further and file a bug
if necessary.

I think it probably is machine-dependent, or something like that. I've
never heard of hash order being anything but "not guaranteed" before
1.9 -- except that if you iterate twice, I believe it definitely
happens in the same order both times (though I'm not 100% sure that
that's actually specified or guaranteed either).


David

--
David A. Black / Ruby Power and Light, LLC
Ruby/Rails consulting & training: http://www.rubypal.com
Coming in 2009: The Well-Grounded Rubyist (http://manning.com/black2)

http://www.wishsight.com => Independent, social wishlist management!

 

[Rick DeNatale]

[Note: parts of this message were removed to make it a legal post.]

Hi Matz, I'm glad to see that you've entered this fray.

I trust that you know me well enough to know, that while I have strong
opinions, I have no ill-will, particularly to you or your language.

Hi,


* Rails had problem from method name conflict. This was
unfortunate, caused by Rails' fragile monkey patching and our bad
for lack of thorough Rails test before 1.8.7. It is fixed now.

* erb also had a bug in 1.8.7 initial release. It is fixed now.

* two other problems are reported (SWIG and hash order), and as far
as I understand, they both contain bugs that haven't disclosed for
1.8.6 by accident.

I consider them minor and not being worse than previous releases.
Most of them are addressed already. Am I missing something?

First, the next comment is directed to a subset of the Ruby community,
which doesn't include you Matz,

I know that Rails isn't liked by everyone in the Ruby community, but for
better or worse, it's an important factor to the health of a significant
portion of 'our' community. Those who don't use Rails or other large Ruby
frameworks over time haven't experienced how we got to the point where these
problems have been addressed.

At this point, it's a little hard to recover the history, but, although
Rails 2.2 is now compatible with Ruby 1.8.7 it has taken work, and new
releases, from both the Ruby team and the Rails team.

Rails users faced with upgrades, need to deal with the evolution of both
Ruby and Rails. To a large extent Rails over the last year to 18 months has
been even more volatile than Ruby. Porting a Rails app from Rails 1.2.x to
Rails 2.0 (There was no Rails 1.3) was, as the change in major version
number indicates a fairly significant jump. And Rails made changes in Rails
2.2 which, for applications doing certain complex things with ActiveRecord,
VERY difficult to port. That's not Ruby's fault, it's just a fact of life.

So there are quite a few Rails Apps, which need to stay on a version of
Rails which is compatible with Ruby 1.8.6, but won't work with Ruby 1.8.7.

|Back then nobody stepped up and said that if code working on 1.8.6

|broke it is a bug so I just left it at that - the compatibility seemed
|to be not so great.

Of course, none of us is perfect, particularly without the benefit of
hindsight.

As a practitioner and advocate of agile techniques, I like to have a
comprehensive test suite to expose compatibility issues when changes are
made.

Unfortunately in the case of a language like Ruby, it's impossible to have
such a test suite, It would need to draw from a significant portion of the
applications implemented in Ruby, and applications implemented on frameworks
like Rails which are implemented on Ruby, etc. etc. Not really practical.

If compatibility of 1.8.7 is really "not so great":

* if it is fixable, we can fix.

* if it is minor, we can upgrade 1.8.7 (or later) whenever we want,
as we did in the past.

* if upgrading is not affordable for enterprisey people (yes, I
understand them; upgrading is costly even when incompatibility is
minor),

I'm not sure who you are calling enterprisey people here. I know that you
have no intent to disparage, but that term does carry a certain meaning of
"the other guys" with some of the Rubyists, (and others) I run into.

To me "enterprisey" means someone working in a large corporate enterprise,
who is following strategies, architectures and standards, promulgated by
enterprise vendors, like Microsoft, and IBM (to the extent IBM still has the
influence to do this), probably because of edicts handed down from on high
from the executives who play golf with sales reps from those vendors.

I'm pretty sure that such "enterprisey" types make up a very small subset of
the Ruby/Rails community. We don't have many sales reps who play golf with
corporate executives.

The vast majority of those using Rails are fairly small companies trying to
leverage the almost unique advantages of Ruby and Rails to be able to
quickly deploy into production, and incrementally improve core business
applications. And the vast majority of Rails developers are either working
for those small companies, or for a small Rails consultancy, or as
independent freelance contractors.

they can keep using 1.8.6, which is maintained right now.

Very true, I've got no argument with that, but it's important to understand
that those who are using packaged distributions of Ruby from debian
(including Ubuntu), RedHat, macports etc. have more trouble keeping 1.8.6
because normal system administration has a nasty tendency to replace 1.8.6
with 1.8.x where x > 6 because they don't differentiate versions with
differences in teeny version numbers as backwards incompatible. While 1.8.7
is backwards compatible with 1.8.6 in theory. Whether or not it really is,
depends on whether or not all of the ancillary ruby code on the particular
system, like Rails, Gems, and local applications have been kept, or brought
up to date.

Savvy developers get around this by eschewing packaged versions and
installing the components of the Rails stack, including Ruby, from source.
But there are a lot of developers who trust the package maintainers, and run
into these issues, and end up blaming Ruby. We could say "well that's their
problem," if we don't care about the impact on the overall perception of
Ruby. Perhaps that's the right attitude, but I'm not so sure.

Would it have been better for those with these problems had Ruby 1.8.7 not
included the changes from 1.9 which broke apps and frameworks? I think that
the answer is clearly yes.

But, for better or worse, it did, and we have to live with the consequences,
that doesn't mean we need to be happy about it.

As EngineYard raised their hands, the maintenance of 1.8.6 will be

kept, even after 1.8.8.

If more evidence of great incompatibility is not shown, I will

consider it FUD, or FUD-like at most, even though I see no bad
intention from anyone.

I hope that this doesn't come across as FUD, or even FUD-like, I'm just
trying to put a bit more context on why compatibility isn't a black or white
thing, and the sources of real FUD.

The real FUD doesn't come from discussions like this, it comes from those
who either have run into these problems and blame Ruby. We have no control
over what they say, we can only try to consider the effects of decisions on
the broader community, within the bounds of our human lack of complete
knowledge.

 

[Rick DeNatale]

[Note: parts of this message were removed to make it a legal post.]

Hi --

On Sat, 14 Feb 2009, Rick DeNatale wrote:

Sometime after that discussion, Matz, who I know has much more than a

Hashes aren't ordered in 1.8.7, though.

Which makes this part of the thread rather moot, doesn't it?!

Thanks for pointing that out.

 

[James Coglan]

[Note: parts of this message were removed to make it a legal post.]

NO THEY DO NOT. It is the nature of a regular hashtable that
the order of entries in its internal data structures depends
only on the hash function and its inputs. It doesn't know
anything about the order of entries in your hash literal.

I've shown you examples using Ruby *1.8.6* that clearly
demonstrate that it gives a **** about order of appearance
in a literal, or insertion order.

I'm really not up for a slanging match so let me just say that, in code I've
written, Ruby < 1.8.7 has demonstrated the behaviour I described. This may
be a fluke or a bug, in which case I will investigate further and file bugs
if necessary. What I do know is that I've had code in a library released
over 2 years ago and used by a fair number of people relying on this
behaviour and heard no complaints until 1.8.7 arrived. I realise that
hashtables are not ordered in 1.8 and have since made my code more robust.
I'm not questioning your examples -- you can but experiment to see what
works -- and I will try to extract the code from my project that
demonstrates my point when I have some time.

I'm not the least bit interested in who's right or wrong or in berating the
Ruby code devs or indeed anyone else, and was hoping for a productive
discussion, not people repeatedly asserting that I don't know how a hash
table works and trying to shut me up. When I've time I'll see if what I have
is a Ruby bug or something else. Until then I'm canning this conversation.

 

[Rick DeNatale]

[Note: parts of this message were removed to make it a legal post.]

Someone suggested having 1.8.8 being 1.8.6 syntax and semantics with
only bug fixes, security fixes and efficiency enhancements. That appeals
to me ... that way, "1.8.7" is a development release and "1.8.8" is
stable.

Bad idea, IMHO.

This would mean that Ruby 1.8.8, would be backwards compatible with Ruby
1.8.6, but NOT with Ruby 1.8.7, which might be good for needing to say with
the Ruby 1.8.6 definition, but not those who are already using Ruby 1.8.7.