autoload with a block?

[David Masover]

I've just discovered Ruby's autoload feature. It looks to imply something like
this:

autoload :Foo, 'foo'
Bar.autoload :Baz, 'bar/baz'

This implies that to have namespace'd modules autoloaded properly, I would end
up with a structure like this:


autoload :Foo, 'foo'

(and, in foo.rb)

module Foo
autoload :Bar, 'foo/bar'
end


What if I want to abstract away such a system? In other words, what if I want
a light, portable version of Rails-style autoloading? In a perfect world, I
would be able to do something like this:

autoload :Foo do
require 'foo'
Foo.autoload :Bar ...
end

In that simplest form, it would at least centralize all my loading stuff in
one place. In a more meta form, I could run through some search paths I'm
likely to use (in my own app, for instance), and generate an autoload scheme
based on that.

Does this sound like a good idea? Or should I be hacking it together with
const_missing, instead?

 

[David Masover]

No one had anything to say about this... Hmm.

In case you didn't realize: I'm testing all of this in Ruby 1.9.


I thought of a sneakier approach: The rdoc specifies that autoload will call
Kernel::require:

http://ruby-doc.org/core-1.9/classes/Kernel.html#M006100

And it does, indeed, seem to behave this way -- so long as I don't redefine
Kernel::require. A simple test:

$ irb1.9
irb(main):001:0> module Kernel
irb(main):002:1> alias_method :require_without_me, :require
irb(main):003:1> def require(*args, &block)
irb(main):004:2> puts 'Require called:'
irb(main):005:2> p args
irb(main):006:2> p block
irb(main):007:2> require_without_me *args, &block
irb(main):008:2> end
irb(main):009:1> end
=> nil
irb(main):010:0> autoload :Foo, 'foo'
=> nil
irb(main):011:0> Foo
loaded foo.rb
=> Foo
irb(main):012:0> require 'foo/bar'
Require called:
["foo/bar"]
nil
loaded bar.rb
=> true
irb(main):013:0>



For what it's worth, both foo.rb and foo/bar.rb have 'puts' statements in
them, declaring that they were loaded.

Maybe it's irb at fault? Hmm... I created this file:

module Kernel
alias_method :require_without_me, :require
def require(*args, &block)
if args.first == 'test/file/not/loading'
puts 'Not loading the test file'
else
require_without_me(*args, &block)
end
end
end

puts 'The following works:'
require 'test/file/not/loading'

puts 'The following does not:'
autoload :Foo, 'test/file/not/loading'
Foo



I'd expect it to end with a constant not found, or module not found.
Instead, I get:

$ ruby1.9 test.rb
The following works:
Not loading the test file
The following does not:
test.rb:17:in `<main>': no such file to load -- test/file/not/loading
(LoadError)



Doesn't look like I typoed.


So, in other words, autoload is either using its own interpretation of
require, or it's hardcoding (in C) a call to the original require, rather
than going through the Ruby code. (I wonder how this would have cooperated
with Rubygems, had that not been included?)

So far, the crushing limitation of autoload is that I can't easily autoload
namespace'd things, without littering the namespace itself with autoload
methods -- or eager-loading all parent namespaces.

Well, that, and I can't seem to customize its behavior _at_all_ other than by
rewriting it from scratch, making it somewhat useless.

 

[Trans]

So, in other words, autoload is either using its own interpretation of
require, or it's hardcoding (in C) a call to the original require, rather
than going through the Ruby code. (I wonder how this would have cooperate= d
with Rubygems, had that not been included?)

I reported this bug a long time ago. It is a serious issue with one of
my programs. Because of it I am forced never to use autoload, and tell
all users of my program that they must do the same. I would have
expected this to be fixed for 1.9, and back-ported. It's disappointing
to hear it still remains an issue.

T.

 

[David Masover]

It is a serious issue with one of
my programs. Because of it I am forced never to use autoload, and tell
all users of my program that they must do the same.

I think I've found a solution, for myself -- basically a const_missing hack.

Actually, I cracked open the ActiveSupport source, curious to see how they did
it. And I spent probably an hour or two, staring at the code, before shaking
my head and starting from scratch.

Rails lets you write beautiful code, sometimes, but the Rails source sure
isn't beautiful, most of the time. (In my not-so-humble opinion.)

Still, I can't help but wonder -- if ActiveSupport had to do such backflips to
get it working, maybe there's a serious assumption in my own code?

Let me know if I'm missing something.
I'm still trying to think of ways this could break.



# You can use anything that provides an inflector.
# I'm already using Sequel here, so it works for me.
require 'sequel'

class Module
def const_missing_with_autoload(name)
our_name = (self==Object) ? '' : self.name + '::'
begin
require (our_name+name.to_s).underscore
const_get name
rescue LoadError
const_missing_without_autoload(name)
end
end
alias_method :const_missing_without_autoload, :const_missing
alias_method :const_missing, :const_missing_with_autoload
end

 

[David Masover]

I think I've found a solution, for myself -- basically a const_missing hack.

Ok, that didn't work. All kinds of edge cases, and I can see why things like
autoload have been written.

So I wrote my own... but there's still one problem I can't seem to solve here:

class AutoLoader < BasicObject
def initialize parent, mod, file
@parent, @mod, @file = parent, mod, file
end
end

# Pushed out here so I can get at Kernel.require.
AutoLoader.send :define_method, :method_missing do |*args, &block|
@parent.send :remove_const, @mod
Kernel.require @file
@parent.const_get(@mod).send *args, &block
end



Typical usage would be:

self.class.const_set :Foo, AutoLoader.new(self.class, :Foo, 'foo')

And it works great -- exactly the way I want -- for _almost_ all cases.
I still can't do this:

module Foo
end

without actually loading foo.rb yet. (Somehow, Ruby knows my BasicObject isn't
a module, without ever tripping the method_missing there.)

Yet the same thing _does_ work with autoload:

autoload :Foo, 'foo'
module Foo; end

For what it's worth, Rails' const_missing has an even worse problem: If I do
the "module Foo" syntax above, I will define Foo, which means const_missing
will never be triggered, which means foo.rb will never be loaded.



It looks like my choices are either to learn to write a (non-portable) C
extension (and maybe still not have it work?), or to create a temporary file,
point autoload at that, and have it do the rest.

That sucks.



Is there any way to know for certain when a file is loaded, when it's autoload
that's doing it, short of modifying the file? (I'm specifically wanting to be
_notified_, so that I can prepare sub-constants to be autoloaded as well --
that is, when Foo is loaded, I want to automatically run Foo.autoload :Bar.)

Is there any way to force autoload to use a different mechanism?

Is there any way, in Ruby, to emulate autoload with actual feature parity?

Or am I really going to have to create a temporary file for each and every
source file I might possibly ever want to use?

 

[Trans]

That sucks.

Is there any way to know for certain when a file is loaded, when it's aut= oload
that's doing it, short of modifying the file? (I'm specifically wanting t= o be
_notified_, so that I can prepare sub-constants to be autoloaded as well = --
that is, when Foo is loaded, I want to automatically run Foo.autoload :Ba= r.)

Is there any way to force autoload to use a different mechanism?

Is there any way, in Ruby, to emulate autoload with actual feature parity= ?

Or am I really going to have to create a temporary file for each and ever= y
source file I might possibly ever want to use?

The things to do is of course to fix the Ruby source to A) use a
callback for autoload and B) to use the normal Kernel.require call.
Then submit a patch and pray the higher ups care enough to check it
in.

Now the question is are you a C coder

T.

 

[David Masover]

=20

Actually, I have figured out something that works reasonably well, for now.=
=20
Given a path that can be safely autoloaded, it will check for files in ther=
e=20
that look like they could be modules.

It also provides a mixin, which, when included, checks through its existing=
=20
paths for possible nested modules and adds them.

I suspect it's twice as long as it needs to be, though, and certainly less=
=20
elegant to write.

If anyone's interested, I can clean it up and release it. It's roughly=20
equivalent to the same functionality in ActiveSupport, except that it's muc=
h=20
shorter (dependencies.rb is around 500 lines, and I'm well under 100), and=
=20
requires different kinds of babysitting.

That is: With ActiveSupport, if foo.rb is not loaded, the following:

class Foo; end

will cause foo.rb to _never_ be autoloaded, because const_missing isn't=20
checked on assigning things to a constant.

However, in my version, to support foo/bar.rb, I have to do at least:

module Foo
include AutoLoad
end

I can put it in foo.rb, or anywhere that's loaded by the time I need Foo::B=
ar,=20
but it is required.

The things to do is of course to fix the Ruby source to A) use a
callback for autoload and B) to use the normal Kernel.require call.
Then submit a patch and pray the higher ups care enough to check it
in.

I'd add a C) Have const_missing actually called in enough places to make it=
=20
possible to implement vanilla autoload on top of const_missing.

Having autoload be more flexible in the action it performs would be helpful=
,=20
but it'd be even more helpful for it to be flexible in the constants it=20
applies to.

Now the question is are you a C coder

Yes and no.

I have written one-line kernel patches, which did things I'm ashamed to adm=
it.=20
I've cracked open abandoned projects, to try to make them, for example,=20
compile on 64-bit. I've read the source of poorly-documented parts of=20
PowerDNS to figure out what it was doing.

To be a programmer on any Unix is to have to be at least somewhat aware of=
=20
what C is and what it can do, even if you never use it.

But I have never done any significant development in C. Even if I had, it=20
would (I assume) still take some time to learn the Ruby codebase and=20
conventions.

So, someone else needs to do this

 

[Nobuyoshi Nakada]

Hi,

At Fri, 15 Aug 2008 15:57:21 +0900,
David Masover wrote in [ruby-talk:311365]:

autoload :Foo do
require 'foo'
Foo.autoload :Bar ...
end

In that simplest form, it would at least centralize all my loading stuff in
one place. In a more meta form, I could run through some search paths I'm
likely to use (in my own app, for instance), and generate an autoload scheme
based on that.

I don't think it makes things simpler, but it sounds
interesting. You can file it in the ITS, if you want.
<http://redmine.ruby-lang.org/projects/ruby/issues?set_filter=1&tracker_id=2>