Matz says namespaces are too hard to implement - why?

[Charles Oliver Nutter]

Yeah, and then there is still the question what happens to recursion?
Since the checks are lexically, you could override a recursive method
with another one and the second invocation ends up calling the old
version. Spooky although I guess you could do some cute tricks with
this.

One way or the other I don't claim to know where selector namespaces
are actually useful, so I can't say which way would be better to go. But
I'll gladly illustrate how to implement them either way.

- Charlie

 

[thefed]

To find a solution to my want/need for namespaces,

Are there any module-specific variables? Kind of like a controlled
global?

I've been looking into using Fluid to get what I'm looking for, but
I was hoping that the variable usage could span multiple files.

Help?
-------------------------------------------------------|
~ Ari
if god gives you lemons
YOU FIND A NEW GOD

 

[Jos Backus]

One way or the other I don't claim to know where selector namespaces are
actually useful, so I can't say which way would be better to go. But I'll
gladly illustrate how to implement them either way.

Here's a ruby-core thread which looks related. The features described in there
could likely be implemented through selector namespaces.

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/9960

 

[Eivind Eklund]

What about this example, based on your code:

require 'namespaces'
namespace :foo, %{
class Array
def each
# do something new
end
end
p Array.new.collect
}

The default implementation of Enumerable#collect calls each. Would you
expect collect to see the original each method or the one you've
provided in the namespace?

Original each.

The way I think about it, importing a namespace is gaining access to a
group of methods that you wouldn't otherwise get access to. You can
do namespacing manually by calling all your methods
"mynamespace_method" instead of just "method", and checking for this
in method_missing, a la (but mind overriding $1 and fixing up
exception backtraces):

def method_missing(method, *rest)
if method.to_s =~ /^mynamespace_(.*)/
send($1, *rest)
else
super
end
end

I would also believe that with type inference, namespacing should be
able to make code go *faster*, not slower. When you do namespace
lookups, you can do early resolution of calls more easily, as you can
find the method being available through the namespace and short
circuit. (I have no idea if you're doing inference in JRuby - are
you?)

See the above example; if you only want namespaces so that within a
given block of code method calls to where you want them to, that's
simpler to implement. But it breaks some amount of consistency you might
expect. It seems like if selector namespacing is useful (which I'm
unsure of) it would only be generally useful if it could also affect
calls further down the chain. Maybe I'm wrong?

You're expecting consistency at a different level than me, at least.

At some point you have to be able to say "this code is being
namespaced". If you want to do that at runtime, then either you need to
modify already-parsed code (which won't work across libraries or calls)
or you need every invocation to check for namespaces. If you want to do
that at parse/compile time you need a pragma or keyword, and you still
can't do it across libraries or calls without installing a namespace
check for every invocation.

I think the appropriate point in time is compile time; not being able
to modify name spaces at run time may be a tolerable cost. The point
of name spaces, as I see them, is to avoid name conflicts while still
retaining "nice" names; while it might be fun to be able to play with
them at runtime, ones that are restricted to compile time may be
better than not having name spaces at all.

I'm still on the fence about how useful they are at all - use of plain
prefixing to create an explicit namespace might be as useful as adding
implicit ones, and all of this might only be useful in connection with
adding some sort of protocol (interface definition) support.

Eivind.

 

[MonkeeSage]

Original each.

The way I think about it, importing a namespace is gaining access to a
group of methods that you wouldn't otherwise get access to. You can
do namespacing manually by calling all your methods
"mynamespace_method" instead of just "method", and checking for this

Well, but that's the trivial case that Charles talked about. The
practical case would be where all the code called from a given
selector namespace respects the definitions / constants / variables of
that namesapce; so if you define Array#index to return "foo", then any
methods called within that namespace will use the modified
Array#index. And that's just where it gets hairy -- causing extra
lookups (at best, one; more likely, several) for every vcall, to
determine if we're in a namespace. (Someone correct me if I've not
followed the thread right.)

in method_missing, a la (but mind overriding $1 and fixing up
exception backtraces):

def method_missing(method, *rest)
if method.to_s =~ /^mynamespace_(.*)/
send($1, *rest)
else
super
end
end

I would also believe that with type inference, namespacing should be
able to make code go *faster*, not slower. When you do namespace
lookups, you can do early resolution of calls more easily, as you can
find the method being available through the namespace and short
circuit. (I have no idea if you're doing inference in JRuby - are
you?)


You're expecting consistency at a different level than me, at least.


I think the appropriate point in time is compile time; not being able
to modify name spaces at run time may be a tolerable cost. The point
of name spaces, as I see them, is to avoid name conflicts while still
retaining "nice" names; while it might be fun to be able to play with
them at runtime, ones that are restricted to compile time may be
better than not having name spaces at all.

I'm still on the fence about how useful they are at all - use of plain
prefixing to create an explicit namespace might be as useful as adding
implicit ones, and all of this might only be useful in connection with
adding some sort of protocol (interface definition) support.

Eivind.

Regards,
Jordan

 

[Paul Brannan]

The complication is that under normal circumstances a method has no
knowledge of whether it's being called inside a namespace or not, since
the installation of the namespace itself is just another method call. So
every method in the system would have to check whether they are being
called under a namespace.

The technique I used in:

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/161439

was to modify caller's frame to signify that a particular namespace is
active.

If use and namespace were keywords rather than method calls, they could
do this without the use of caller_binding.

This technique is slow, but iirc only for methods defined in a
namespace (but the cost is paid regardless whether the namespace is
active).

If namespaces were lexical elements, I think the performance penalty
could be minimized, as it would be known at compile time which method
lookup tables to use for a given call.

Paul

 

[Paul Brannan]

So now I wonder, am I missing something? Is the devil in the details, or
is one of the incomplete things in my proof of concept the show stopper?
I see that it would be some work, but I don't see how it would be hard.
Please enlighten me

AFAICT, one problem with your implementation is it's not possible to
have multiple namespaces active at the same time. This is something
that I think would be very desirable, otherwise it's not possible for a
single method to make use of methods defined in two different namespaces
(libraries) simultaneously.

Paul

 

[Eivind Eklund]

Well, but that's the trivial case that Charles talked about. The
practical case would be where all the code called from a given
selector namespace respects the definitions / constants / variables of
that namesapce;

My point was that I do not see why this would be any more practical.

I don't know what your usecase is; from my point of view, the idea of
namespaces is to separate the call space for implementations of
different extensions to a set of types (in a more static language,
each type would be a different class; this is less relevant for Ruby),
so that different extensions do not step on each other.

As far as I can tell, there are slightly different tradeoffs between
using lexical scoping (which I propose) and dynamic scoping (which you
propose), with the dynamic scoping enabling some hacks that the
lexical scoping doesn't, and the lexical scoping having the benefits
of ease of understanding and speed of code.

This is the standard tradeoff between lexical and dynamic scoping; in
languages that provide both dynamic and lexical scoping for variables
(Perl, Lisp) my experience is that we end up using about 1000x more
lexical than dynamic scoping, and in languages that only provide
lexical scoping (e.g, Ruby), we easily work around the lack of dynamic
scoping and hardly notice that it isn't there.

If you (or anybody) can show me usecases that show that dynamic
scoping is essential for implementation of namespaces in Ruby, of
course I'm all ears. So far, it's just seems like the need for
dynamic scoping has been taken as a given, and I personally feel that
dynamic scoping make programs harder to understand than lexical
scoping, and that without compelling usecases it seems more reasonable
to use lexical scoping, which also avoids the speed penalty.

Eivind.

 

[Charles Oliver Nutter]

If namespaces were lexical elements, I think the performance penalty
could be minimized, as it would be known at compile time which method
lookup tables to use for a given call.

Yep, this would work, but it only namespaces for a single frame. Is that
what we want?

- Charlie

 

[Charles Oliver Nutter]

Original each.

The way I think about it, importing a namespace is gaining access to a
group of methods that you wouldn't otherwise get access to. You can
do namespacing manually by calling all your methods
"mynamespace_method" instead of just "method", and checking for this
in method_missing, a la (but mind overriding $1 and fixing up
exception backtraces):

I was thinking about this a bit more today and I don't see how it's
particularly useful. If the namespacing is only local to some block or
other lexical structure...why don't you just call the methods directly?
What does it gain you?

It seems like namespacing is only "really" useful if you can also have
namespace changes apply to calls further down the stack. In a lexical
context, I don't see what it gains you over just calling different methods.

I would also believe that with type inference, namespacing should be
able to make code go *faster*, not slower. When you do namespace
lookups, you can do early resolution of calls more easily, as you can
find the method being available through the namespace and short
circuit. (I have no idea if you're doing inference in JRuby - are
you?)

Not...yet

I think the appropriate point in time is compile time; not being able
to modify name spaces at run time may be a tolerable cost. The point
of name spaces, as I see them, is to avoid name conflicts while still
retaining "nice" names; while it might be fun to be able to play with
them at runtime, ones that are restricted to compile time may be
better than not having name spaces at all.

I'm still on the fence about how useful they are at all - use of plain
prefixing to create an explicit namespace might be as useful as adding
implicit ones, and all of this might only be useful in connection with
adding some sort of protocol (interface definition) support.

Ditto. If they're only lexically scoped to make "nice" names, I don't
see the benefit. I could see how they might be useful if you wanted to
run an entire call stack with temporary modifications, but of course
that's the trickier version.

- Charlie

 

[Paul Brannan]

Yep, this would work, but it only namespaces for a single frame. Is that
what we want?

I think what I would want is:
- apply a namespace to a particular scope
- apply a namespace to a single method
- apply a namespace to an entire class definition

All of these can be determined at compile time.

What benefit would there be to a namespace being active for child
frames? I think it might cause side-effects that are hard to track.

I wonder what Smallscript does.

Paul

 

[Paul Brannan]

I was thinking about this a bit more today and I don't see how it's
particularly useful. If the namespacing is only local to some block or
other lexical structure...why don't you just call the methods directly?
What does it gain you?

It seems like namespacing is only "really" useful if you can also have
namespace changes apply to calls further down the stack. In a lexical
context, I don't see what it gains you over just calling different methods.

It might not be desirable to call the methods directly if you don't know
what the type of the object is. For example, if I have a method that
was passed an object as a parameter, I might import a namespace to get a
particular behavior for arrays in case the object passed in is an array.
But the object might not be an array, in which case I can't just change
the name of the method I call.

I do see the point about #each, but I'm having a hard time imagining
what the implications are of having the namespace apply further down the
stack. What do we do if I call a method that imports a namespace which
conflicts with the one I've just imported?

Namespaces in Ecmascript appear to be lexically scoped:

http://docs.huihoo.com/web/js/es4/core/namespaces.html

I'm curious if implementors of other languages have thought about this
issue. I wonder if we could get David Simmons to chime in, since he's
the one who pointed us here in the first place.

Paul

 

[Charles Oliver Nutter]

I do see the point about #each, but I'm having a hard time imagining
what the implications are of having the namespace apply further down the
stack. What do we do if I call a method that imports a namespace which
conflicts with the one I've just imported? ...
Namespaces in Ecmascript appear to be lexically scoped: ...
I'm curious if implementors of other languages have thought about this
issue.

...

I really don't know, and to be honest I haven't seen a particularly
compelling use case for either lexically scoped namespaces
or...what...temporally scoped?

So we want to limit class modifications to a specific range of
execution. Then there's two cases:

1. we want to do it lexically-scoped. In that case, we can just call our
own utility methods directly, and if the object isn't typed as we
expect, it passes through. Sure, syntactic sugar...but who cares, is it
worth impacting the rest of the language impl to support it?
2. we want to do it temporally-scoped, affecting downstream calls too.
Well then, I'd love to see the use case for this, and you may have a
hard time convincing me that it's worth crippling every invocation with
some indirection mechanism that can be overridden on a whim.

What is the use case? I'd love to see.

- Charlie

 

[Paul Brannan]

I really don't know, and to be honest I haven't seen a particularly
compelling use case for either lexically scoped namespaces
or...what...temporally scoped? ...
What is the use case? I'd love to see.

I would like other users to be able to import mathn without it breaking
my code.

Ideally this should work:

class Foo
def foo
# [ruby-talk:69859]
use namespace Mathn
m = Matrix[[1,2], [3,4]]
return m*m.inv #=> [[1,0], [0,1]]
end
end

class Bar
def bar
return 5/2 #=> 2
end
end

but I guess this is dynamic scoping rather than lexical scoping.

("temporal" to me implies that the change applies outside the current
thread -- see also [ruby-talk:196082])

While potentially more powerful, dynamic scoping makes it harder to know
what side-effects importing the namespace might have.

But what if the namespace could be turned on/off at the object or method
level? And what if that namespace change automatically be propogated to
objects or methods created within a scope where that namespace were
active?

In the above example, m would automatically get the Mathn namespace
since it was created in a scope where that namespace was active. Then
the code would work as expected.

This could be implemented similar to the way mix-ins are implemented
today, avoiding the performance penalty for every object.

Paul

 

[Rick DeNatale]

I really don't know, and to be honest I haven't seen a particularly
compelling use case for either lexically scoped namespaces
or...what...temporally scoped? ...
What is the use case? I'd love to see.

I would like other users to be able to import mathn without it breaking
my code.

Ideally this should work:

class Foo
def foo
# [ruby-talk:69859]
use namespace Mathn
m = Matrix[[1,2], [3,4]]
return m*m.inv #=> [[1,0], [0,1]]
end
end

but what about this:

def divide(a, b)
use namespace Mathn
a / b
end

Now, I'd think you would want:

1 / 2 #=> 1

divide(1,2) #=> 1 / 2

The problem here it seems that you would need more than attach a
namespace to an object. Here we want the Integer/Fixnum classes to
act differently in the context of executing the divide method, even
for objects instantiated outside and passed in.

 

[Paul Brannan]

but what about this:

def divide(a, b)
use namespace Mathn
a / b
end

Now, I'd think you would want:

1 / 2 #=> 1

divide(1,2) #=> 1 / 2

The problem here it seems that you would need more than attach a
namespace to an object. Here we want the Integer/Fixnum classes to
act differently in the context of executing the divide method, even
for objects instantiated outside and passed in.

I'm having trouble understanding why this case wouldn't work as
expected. Can you elaborate?

Thanks,

Paul

 

[Rick DeNatale]

I'm having trouble understanding why this case wouldn't work as
expected. Can you elaborate?

What causes the integer 1 to use the '/' method monkeypatched in Mathn
in one case and not the other?

 

[Paul Brannan]

...
What causes the integer 1 to use the '/' method monkeypatched in Mathn
in one case and not the other?

In the first case (1 / 2), the Mathn namespace is not active, so the
usual integer division method applies.

In the second case (divide), the Mathn namespace is active, so Integer#/
defined in the Mathn namespace applies.

Paul

 

[Rick DeNatale]

In the first case (1 / 2), the Mathn namespace is not active, so the
usual integer division method applies.

In the second case (divide), the Mathn namespace is active, so Integer#/
defined in the Mathn namespace applies.

Okay, I read your pastie again, and I understand the unnatural acts
being performed a bit more. All I can say is that it makes me want to
run away in horror.

There are also some other little issues, for example, the way mathn
changes Fixnum and Bignum

class Fixnum
alias / quo
end

class Bignum
alias / quo
end

and in Ruby 1.9 these become:

class Fixnum
remove_method :/
alias / quo
end

class Bignum
remove_method :/
alias / quo
end