New Generator

[Greg Fitzgerald]

Ruby gurus,

There is a side-effect in the current Generator implementation. When
iterating to an element, it caches the element *after* the one it
returns each iteration. Meaning that when you first construct the
generator, it caches the first element. When you call next() it will
set aside the cached value to return, and then iterate to the second
element, make it the new cached value, and return the first. You will
only notice this side-effect in code where you don't iterate through
every element in the generator, such as a 'find' operation. The
side-effect becomes problematic if either it takes significant time to
retrieve the next element (crossing a network) or when iterating is a
destructive operation (pulling data off a stream).

Below is the code that demonstrates the issue and provides a new
implementation of Generator that corrects it. Although I haven't done
tons of testing on it, I believe the API behaves in all ways like the
original.

Do you all think this may be something worth correcting in the next
release of Ruby? If so, is there a document around somewhere for how
to go about submitting a patch?


require 'generator'

def test_generator()
timeStart = Time.now
puts "start constructing at #{Time.now - timeStart}"
gen = Generator.new do |g|
for num in [1,2,3,4,5]
sleep(3)
g.yield(num)
end
end

puts "finished constructing at #{Time.now - timeStart}"

puts "getting first element at #{Time.now - timeStart}"
for num in gen
puts "#{num} at #{Time.now - timeStart}"
break if num == 3
puts "getting next element at #{Time.now - timeStart}"
end
puts "done at #{Time.now - timeStart}"

puts "\n\n"
end

puts "Ruby's Generator"
test_generator()

class Generator
def initialize(&closure)
@closure = closure
return rewind()
end

def each()
rewind()

while self.next?()
yield self.next()
end
return self
end

def rewind()
if @position != 0
@currentElement = callcc do |cc|
@generatorEntryPoint = cc
@closure.call(self)
@closureEntryPoint = nil
@generatorEntryPoint.call()
end

@nextElement = @currentElement
@nextElementCached = true
@position = 0
end
return self
end

def pos()
return @position
end

def current()
return @currentElement
end

def next?()
if @nextElementCached == false
begin
@nextElement = retrieve_next()
@nextElementCached = true
rescue EOFError
end
end
return @nextElementCached
end

def end?()
return !self.next?()
end

def next()
@currentElement = @nextElementCached ? @nextElement : retrieve_next()
@position += 1
@nextElementCached = false
return @currentElement
end

def retrieve_next()
retval = callcc do |cc|
@generatorEntryPoint = cc
@closureEntryPoint.call()
end
raise EOFError, 'no more elements available' if @closureEntryPoint == nil
return retval
end

def yield(obj)
return callcc do |cc|
@closureEntryPoint = cc
@generatorEntryPoint.call(obj)
end
end
end

puts "Corrected generator"
test_generator()

 

[Zev Blut]

Hello Greg,

If you do not get any responses to your proposed modification of the
Generator class, then you may want to try the ruby-core mailing list.
If you think the current implementation has a bug, then you can also
report it here:
http://rubyforge.org/tracker/?atid=1698&group_id=426&func=browse

Best,
Zev