memoize and yaml

[Brian Buckley]

Hello,

I am using the "memoize" module to eliminate having to redo complex,
time-consuming calculations. It works great. However, when I yamlize
my "memoized" object off to the database and then retrieve it back,
all the memoizing is lost.

My Ruby skills are still in development. In addition to normal
yamlizing, how could one also yamlize the memoize information, so that
yamlizing and back retains the methods that are to be memoized and
also retains any prior saved (memoized) calculations?

The test below on Foo illustrates what I am trying to do. Also below
is the whole Memoize module.

Any tips or pointers appreciated. Thanks!

--Brian Buckley


require 'test/unit'
require 'yaml'
require 'memoize'

class TestMemoizeAndYaml < Test::Unit::TestCase
class Foo
include Memoize
attr_reader :x, :y
def initialize(x, y)
@x, @y =3D x, y
end
def calc1(a, b)
sleep 1 # fakes big, complex calc
ans =3D calc2 + a + b
end
def calc2(c =3D 0)
sleep 1 # another big, complex calc
ans =3D x + y
end
end

def exercise_foo(foo)
3.times{ foo.calc1 10,20 }
end

def test_memoize_and_yaml

foo =3D Foo.new(1, 2) # first a foo without memoizing
exercise_foo(foo) # without memoizing exercise_foo takes 6 seconds
- good

# now memoize it
foo.memoize :calc1 # memoize calc1
foo.memoize :calc2 # and a 2nd method
exercise_foo(foo) # good - now it takes only 2 secs because of the memo=
izing
exercise_foo(foo) # and now no seconds because of the memoizing

foo =3D YAML::load(foo.to_yaml) # to the database and back
exercise_foo(foo) # but now exercise_foo is taking 6 secs again
# how can I retain memoized state?
# need to retain both the methods and the hashes of calculation histori=
es
end
end

#FYI... here is the Memoize module
module Memoize
MEMOIZE_VERSION =3D "1.1.0"
def memoize(name)
meth =3D method(name)
cache =3D {}
(class << self; self; end).class_eval do
define_method(name) do |*args|
cache.has_key?(args) ? cache[args] : cache[args] ||=3D
meth.call(*args)
end
end
cache
end
end

 

[Florian Groß]

I am using the "memoize" module to eliminate having to redo complex,
time-consuming calculations. It works great. However, when I yamlize
my "memoized" object off to the database and then retrieve it back,
all the memoizing is lost. [...]

#FYI... here is the Memoize module
module Memoize
MEMOIZE_VERSION = "1.1.0"
def memoize(name)
meth = method(name)
cache = {}
(class << self; self; end).class_eval do
define_method(name) do |*args|
cache.has_key?(args) ? cache[args] : cache[args] ||=
meth.call(*args)
end
end
cache
end
end

This memoize module stores the cache in a local variable that can be
referenced from within the memoize method wrapper.

It also memoizes the methods at the instance level meaning that after
storing and loading an instance the methods won't be memoized anymore.

Here's a version that stores the cache in an instance variable and that
replaces the method definitions at class definition time:

module Memoize
def memoize(name)
name = name.to_sym
old_method = instance_method(name)
remove_method(name)

define_method(name) do |*args|
@cache ||= {}
signature = [name] + args

if @cache.include?(signature) then
@cache[signature]
else
@cache[signature] = old_method.bind(self).call(*args)
end
end
end
end

Please note that you can use neither of these memoize() methods for
memoizing methods that can take a block argument as it will be dropped.

Oh, and if you call a memoized method with an argument that can't be
serialized by YAML this definition will probably cause trouble.

 

[Brian Buckley]

old_method =3D instance_method(name)

I see the intent. I have not worked at it yet but I dropped your code
in as you wrote it and I am getting a NoMethodError: undefined method
`instance_method' on the above line.

--Brian

 

[Florian Groß]

I see the intent. I have not worked at it yet but I dropped your code
in as you wrote it and I am getting a NoMethodError: undefined method
`instance_method' on the above line.

Make sure to use the code on a module or class:

class Foo
extend Memoize

def foo(x)
puts "Calculating #{x} ** 32"
x ** 32
end
memoize :foo
end

 

[Brian Buckley]

class Foo

extend Memoize

def foo(x)
puts "Calculating #{x} ** 32"
x ** 32
end
memoize :foo
end

Got it. I had used "include" rather than "extend". Your solution
applies memoization to all objects of a class. The original memoize
applied memoize on a per object basis.

I need to test and digest.

 

[Daniel Berger]

Got it. I had used "include" rather than "extend". Your solution
applies memoization to all objects of a class. The original memoize
applied memoize on a per object basis.

I need to test and digest.

Would folks like both options? memoize_all or something? Or just leave it alone?

- Dan

 

[Sean O'Halpin]

Would folks like both options? memoize_all or something? Or just leave =

it alone?

With a slight adjustment, Florian's version can be used at the top
level too (at the cost of the instance variable @cache in main):

module Memoize
def memoize(name)
name =3D name.to_sym
if self.to_s =3D=3D "main"
old_method =3D class<<self;self;end.instance_method(name)
klass =3D Object
else
old_method =3D instance_method(name)
klass =3D self
end
klass.sendremove_method, name)
klass.senddefine_method, name) do |*args|
@cache ||=3D {}
signature =3D [name] + args

if @cache.include?(signature) then
@cache[signature]
else
@cache[signature] =3D old_method.bind(self).call(*args)
end
end
end
end

class D
extend Memoize
attr_accessor :value
def initialize(value)
@value =3D value
end
def t_memo(*args)
puts "calculating #{args.inspect}"
args.map { |x| x * value }
end
memoizet_memo)

end

d =3D D.new(10)
p d.t_memo(7,8)
p d.t_memo(7,8)

p d.t_memo(1,2)
p d.t_memo(1,2)

e =3D D.new(20)
p e.t_memo(1,2)
p e.t_memo(7,8)
p d.t_memo(7,8)

p d.instance_eval { @cache }

p e.instance_eval { @cache }

extend Memoize

def t_memo(*args)
puts "calculating #{args.inspect}"
args.map { |x| x * 5 }
end

memoize :t_memo

p t_memo(1,2,3)
p t_memo(1,2,3)
p t_memo(7,8)

p @cache


__END__
calculating [7, 8]
[70, 80]
[70, 80]
calculating [1, 2]
[10, 20]
[10, 20]
calculating [1, 2]
[20, 40]
calculating [7, 8]
[140, 160]
[70, 80]
{[:t_memo, 1, 2]=3D>[10, 20], [:t_memo, 7, 8]=3D>[70, 80]}
{[:t_memo, 1, 2]=3D>[20, 40], [:t_memo, 7, 8]=3D>[140, 160]}
calculating [1, 2, 3]
[5, 10, 15]
[5, 10, 15]
calculating [7, 8]
[35, 40]
{[:t_memo, 7, 8]=3D>[35, 40], [:t_memo, 1, 2, 3]=3D>[5, 10, 15]}

Regards,

Sean

 

[Brian Buckley]

module Memoize

def memoize(name)
name =3D name.to_sym
old_method =3D instance_method(name)
remove_method(name)

define_method(name) do |*args|
@cache ||=3D {}
signature =3D [name] + args

if @cache.include?(signature) then
@cache[signature]
else
@cache[signature] =3D old_method.bind(self).call(*args)
end
end
end
end

Two questions of this solution. (It works and is what I need now so
I'm just looking to round out Ruby skills.)

1 Why is the 'remove_method(name)' line necessary? Does not the next
define_method call get rid of the method anyway by overriding?

2 The module requires use of 'extend' instead of 'include'. Is there
a comparable 'include' solution, and (if there is) would it be
preferred because, for example, one could not use this solution on a
class that already extends something else?

Thanks!

Brian Buckley

 

[Florian Groß]

module Memoize
def memoize(name)
name = name.to_sym
old_method = instance_method(name)
remove_method(name)

define_method(name) do |*args|
@cache ||= {}
signature = [name] + args

if @cache.include?(signature) then
@cache[signature]
else
@cache[signature] = old_method.bind(self).call(*args)
end
end
end
end

Two questions of this solution. (It works and is what I need now so
I'm just looking to round out Ruby skills.)

1 Why is the 'remove_method(name)' line necessary? Does not the next
define_method call get rid of the method anyway by overriding?

Overriding methods usually produces warnings. It's best to remove the
methods first.

2 The module requires use of 'extend' instead of 'include'. Is there
a comparable 'include' solution, and (if there is) would it be
preferred because, for example, one could not use this solution on a
class that already extends something else?

Actually, I think this is the cleanest solution. In theory you could
also offer a custom append_features() which defines the methods on the
class or module including Memoize itself. That's more complex and not as
easy to understand, though, in my opinion.

 

[Brian Buckley]

Overriding methods usually produces warnings. It's best to remove the

methods first.

I was wondering if warnings might be the motive. I am glad to get confirma=
tion.

Actually, I think this is the cleanest solution. In theory you could
also offer a custom append_features() which defines the methods on the
class or module including Memoize itself. That's more complex and not as
easy to understand, though, in my opinion.

Cool. I had been thinking I wanted to use your version of Memoize in
conjunction with Rails' ActiveRecord::Base but ActiveRecord::Base
requires one to 'extend' it to use it. However it occurred to me that
since ActiveRecord::Base does not extend anything I could just extend
it instead of my actual class. It seems to work.

class ActiveRecord::Base
extend Memoize
end

class Foo < ActiveRecord::Base
def calc
...
end
memoize :calc
end

--Brian Buckley

 

[Florian Groß]

Cool. I had been thinking I wanted to use your version of Memoize in
conjunction with Rails' ActiveRecord::Base but ActiveRecord::Base
requires one to 'extend' it to use it. However it occurred to me that
since ActiveRecord::Base does not extend anything I could just extend
it instead of my actual class. It seems to work.

class ActiveRecord::Base
extend Memoize
end

class Foo < ActiveRecord::Base
def calc
...
end
memoize :calc
end

Please note that extend is not the same thing as inheriting from another
class. You can inherit from ActiveRecord::Base and still have a
extend(Memoize).