Any interest in lightweight coroutines in Java ala C# 2.0 iterators?

[Ken Sprague]

Was reading the new Microsoft C# 2.0 specification and was interested to see
that they are adding something they call iterators to the language. These
seem very similar to Python generators in that they are methods that save
their state between invocations and return to the exact point where they
left off when called again (that was a brief and not altogether complete
description).

These constructs can be used to create lightweight coroutines ("cooperative"
multi-threading) that are very useful for certain classes of programs, such
as state machines and event simulations. It seems possible to add this
feature to Java in a manner similar to the way enums will be added in Java
1.5 (by the compiler creating a class and adding code behind the scene).

Is there any interest for adding this feature to the Java language?

Ken

 

[John C. Bollinger]

Was reading the new Microsoft C# 2.0 specification and was interested to see
that they are adding something they call iterators to the language. These
seem very similar to Python generators in that they are methods that save
their state between invocations and return to the exact point where they
left off when called again (that was a brief and not altogether complete
description).

These constructs can be used to create lightweight coroutines ("cooperative"
multi-threading) that are very useful for certain classes of programs, such
as state machines and event simulations. It seems possible to add this
feature to Java in a manner similar to the way enums will be added in Java
1.5 (by the compiler creating a class and adding code behind the scene).

Is there any interest for adding this feature to the Java language?

If I want this kind of thing I generally use a class to accomplish it,
possibly an inner class of some appropriate class. Often I need not do
anything special at all; for instance, if the persistent state already
belongs to a suitable object. How much benefit do I stand to gain from
having it as a language feature? In order for it to be worthwhile to
me, I would want the feature to provide a significant advantage in
simplicity and readability of the syntax, while still offering suitably
powerful and flexible semantics.

I worry that such a feature might be good at handling the simple cases
where it isn't really needed, but no better than the existing
alternatives (or even downright unsuitable) for more complex cases. Do
you have a link, or can you post some details / examples?


John Bollinger
(e-mail address removed)

 

[Ken]

The Microsoft C# web page is:



http://msdn.microsoft.com/vcsharp/team/language/default.aspx



There is a link on that page to the proposed C# 2.0 Language Specification,
which has an example. The Python generator concept (pretty much the same as
C#'s iterators) is explained at:



http://www.python.org/peps/pep-0255.html



The implementation of these language constructs is analogous to the type
safe enum construct proposed for Java 1.5. An enum declaration such as:



enum Season { spring, summer, fall, winter }



is converted by the Java compiler, behind the scenes, to a class. Joshua
Bloch, in his book "Effective Java", describes such a class in item 21, the
Typesafe Enum pattern. This allows the user to replace dozens of lines of
tedious boilerplate code with one line of code that succinctly expresses the
concept.



The C# 2.0 Language Specification gives an example of a C# iterator:



public IEnumerator<T> GetEnumerator() {

for (int i = count - 1; i >= 0; --i) yield items;

}



The C# 2.0 Specification gives the code for the inner class that this would
be converted to, I won't repeat it here. It is worthwhile to note that
again, the compiler turns a couple of lines of code into dozens, or even a
hundred or more, lines of code. The compiler saves the user from having to
write a lot of tedious and moderately tricky code that completely obscures
the intended algorithm.



For programs that could potentially use hundreds of types of
iterators/generators , such as games and discrete event simulators, this
language feature would end up saving the programmer from writing tens of
thousands of lines of code, while simultaneously making the resultant code
far clearer.



There is also some discussion in the Python community about how generators
easily allow complex, hierarchical data structures to be iterated over.
These can be viewed by doing a web search on "python generator". Such a
feature would fit in nicely with the new Java 1.5 "foreach" construct. The
following example assumes the ComplexXMLDocument class has a method that
produces an iterator, in a manner similar to the C# 2.0 IEnumerator.



Public void processDoc(ComplexXMLDocument doc) {

// An iterator is implicitly created by the doc
object.

for (node n: doc) {

// Process each node of the document.

 

[John C. Bollinger]

The Microsoft C# web page is:



http://msdn.microsoft.com/vcsharp/team/language/default.aspx



There is a link on that page to the proposed C# 2.0 Language Specification,
which has an example. The Python generator concept (pretty much the same as
C#'s iterators) is explained at:



http://www.python.org/peps/pep-0255.html



The implementation of these language constructs is analogous to the type
safe enum construct proposed for Java 1.5. An enum declaration such as:



enum Season { spring, summer, fall, winter }



is converted by the Java compiler, behind the scenes, to a class. Joshua
Bloch, in his book "Effective Java", describes such a class in item 21, the
Typesafe Enum pattern. This allows the user to replace dozens of lines of
tedious boilerplate code with one line of code that succinctly expresses the
concept.

I have no general objection to this kind of thing; my questions and
concerns have to do with this specific thing. I think it is right and
reasonable to approach any new language feature proposal with a degree
of skepticism and critical analysis.

The C# 2.0 Language Specification gives an example of a C# iterator:



public IEnumerator<T> GetEnumerator() {

for (int i = count - 1; i >= 0; --i) yield items;

}



The C# 2.0 Specification gives the code for the inner class that this would
be converted to, I won't repeat it here. It is worthwhile to note that
again, the compiler turns a couple of lines of code into dozens, or even a
hundred or more, lines of code. The compiler saves the user from having to
write a lot of tedious and moderately tricky code that completely obscures
the intended algorithm.

I observe that both the examples that Microsoft provides contain exactly
one yield return statement, no yield break statements, and no try /
catch blocks. Much of the autogenerated code provided with the examples
supports the general state machine specification described for this type
of construct, and in many cases would be rather superfluous. The
several dozen autogenerated lines of code replace about a dozen that I
would have written by hand (only that many because of observation of
coding conventions).

For programs that could potentially use hundreds of types of
iterators/generators , such as games and discrete event simulators, this
language feature would end up saving the programmer from writing tens of
thousands of lines of code, while simultaneously making the resultant code
far clearer.

There is room, and evidently at least some desire, for a feature of this
sort. After reviewing the Python PEP and the C# 2.0 spec, however, I
find that I do not like some of the specific semantics of the construct
in those languages (see below), and I think there would be some
significant challenges to implementing it in Java in an analogous way.
(And I'm quite curious about how C# will get around similar challenges.)

There is also some discussion in the Python community about how generators
easily allow complex, hierarchical data structures to be iterated over.
These can be viewed by doing a web search on "python generator". Such a
feature would fit in nicely with the new Java 1.5 "foreach" construct. The
following example assumes the ComplexXMLDocument class has a method that
produces an iterator, in a manner similar to the C# 2.0 IEnumerator.



Public void processDoc(ComplexXMLDocument doc) {

// An iterator is implicitly created by the doc
object.

for (node n: doc) {

// Process each node of the document.

.

}

}

Yes, that makes the top level code very simple indeed. I'm not quite
persuaded, however, that the code that would then go into an iterator /
generator declaration would be particularly simpler. Indeed, it seems
to me that the more complex the iterator body, the less, proportionally,
is gained from the autogeneration of an iterator framework around it.
And the less complex the iterator body, the less useful autogeneration
would be in the first place.

I furthermore think there is a particular problem with the way C#
specifies that its implementation interacts with exceptions and try /
catch. Observe that a yield return exits a try block _without the
finally clause being executed_. Control then might or might not later
resume in the try block from the point at which it left. This is ugly,
and I predict that it would be a source of problems in Java and will be
a source of problems in C#.

The general idea of a yield return is also strange -- it is like a
hybrid of return and invocation of a callback, with uncertainty even at
runtime as to which character it will exhibit.

I think that there are also potential implementation problems related to
using multiple yield return and / or yield break statements in one
iterator, and related to iterators with more complex internal state.

It might be nice to have a code-assist type language feature to simplify
creation of iterators, but I don't think that the specific design of the
Python / C# feature is appropriate for Java. (And I'm not convinced
that it's appropriate for C#, either.)


John Bollinger
(e-mail address removed)

 

[Ken]

Thanks for the very thoughtful input. You bring up some interesting points.

Ken