Puzzling regex behaviour

I

Ian Macdonald

Hello,

Can anyone explain this to me?

$ echo $LANG
nl_NL
$ irb -f
irb(main):001:0> foo = "préférées"
=> "pr\351f\351r\351es"
irb(main):002:0> foo =~ /[^[:alnum:]]/
=> nil
irb(main):003:0> foo =~ /\W/
=> 2

First question: Why does the final statement return 2 instead of nil?
All characters in foo are alphabetic characters in this locale.

Then:

$ echo $LANG
nl_NL
$ cat ./foo
#!/usr/bin/ruby -w

foo = "préférées"
p foo =~ /[^[:alnum:]]/
p foo =~ /\W/
$ ./foo
2
2

Huh?

Second question: Why does the first regex match now return 2 instead of
nil?

To my way of thinking, both statements should always return nil, whether
or not they are typed into irb or run in a stand-alone script. At the
very least, both statements should return the same answer, regardless of
the context.

What am I missing here?

Ian
--
Ian Macdonald | tachyon emissions overloading the system
(e-mail address removed) |
http://www.caliban.org/ |
|
|
 
R

Robert Klemme

Hello,

Can anyone explain this to me?

$ echo $LANG
nl_NL
$ irb -f
irb(main):001:0> foo = "préférées"
=> "pr\351f\351r\351es"
irb(main):002:0> foo =~ /[^[:alnum:]]/
=> nil
irb(main):003:0> foo =~ /\W/
=> 2

First question: Why does the final statement return 2 instead of nil?
All characters in foo are alphabetic characters in this locale.

Then:

$ echo $LANG
nl_NL
$ cat ./foo
#!/usr/bin/ruby -w

foo = "préférées"
p foo =~ /[^[:alnum:]]/
p foo =~ /\W/
$ ./foo
2
2

Huh?

Second question: Why does the first regex match now return 2 instead of
nil?

To my way of thinking, both statements should always return nil, whether
or not they are typed into irb or run in a stand-alone script. At the
very least, both statements should return the same answer, regardless of
the context.

What am I missing here?
Click to expand...

Maybe there is an initialization in .irbrc that leads to a changed
locale inside IRB. Or your IRB belongs to a different Ruby version on
that system.

Other than that, I guess you tripped into the wide and wild country of
i18n - many strange things can be found there. Maybe \w and \W only
treat ASCII [a-z] characters as word characters.

Kind regards

robert
 
I

Ian Macdonald

Maybe there is an initialization in .irbrc that leads to a changed
locale inside IRB.
Click to expand...

Nope; I had hoped it would be that easy, but as you can see from my
snippet of output, I started irb with -f, which bypasses ~/.irbrc.
ENV['LANG'] also prints nl_NL in irb, so that can't be it.
Or your IRB belongs to a different Ruby version on that system.
Click to expand...

I compiled it myself, so there has been no mix-and-matching.
Other than that, I guess you tripped into the wide and wild country of
i18n - many strange things can be found there. Maybe \w and \W only
treat ASCII [a-z] characters as word characters.
Click to expand...

It does seem that way, as Perl also appears to treat them this way.

However, I'm still puzzled why there's a difference between irb and a
stand-alone script.

Ian
--
Ian Macdonald | If you are what you eat, I guess that makes
(e-mail address removed) | me a cheese danish. -- Anonymous
http://www.caliban.org/ |
|
|
 
D

David Balmain

However, I'm still puzzled why there's a difference between irb and a
stand-alone script.
Click to expand...

Maybe your editor saves the script in UTF-8 format. The irb example
clearly encodes the string in ISO-8859-1. That could explain the
difference.
 
D

David Balmain

Maybe your editor saves the script in UTF-8 format. The irb example
clearly encodes the string in ISO-8859-1. That could explain the
difference.
Click to expand...

For example;

~$ echo $LANG
en_US.ISO-8859-1
~$ irb -f
irb(main):001:0> "pr\351f\351r\351es" =3D~ /[^[:alnum:]]/
=3D> nil
irb(main):002:0> "pr\303\251f\303\251r\303\251es" =3D~ /[^[:alnum:]]/
=3D> 3

Not exactly what you had but it probably has something to do with the
encoding of the =E9.

--=20
Dave Balmain
http://www.davebalmain.com/
 
I

Ian Macdonald

Maybe your editor saves the script in UTF-8 format. The irb example
clearly encodes the string in ISO-8859-1. That could explain the
difference.
Click to expand...

For example;

~$ echo $LANG
en_US.ISO-8859-1
~$ irb -f
irb(main):001:0> "pr\351f\351r\351es" =~ /[^[:alnum:]]/
=> nil
irb(main):002:0> "pr\303\251f\303\251r\303\251es" =~ /[^[:alnum:]]/
=> 3

Not exactly what you had but it probably has something to do with the
encoding of the é.
Click to expand...

My editor is vim and I run it in the nl_NL locale, so it doesn't start
in UTF-8 mode. To double-check:

:set encoding?
encoding=latin1

And if we dump my little script:

$ od -c foo
0000000 # ! / u s r / b i n / r u b y
0000020 - w \n \n f o o = " p r 351 f 351
0000040 r 351 e s " \n p f o o = ~ /
0000060 [ ^ [ : a l n u m : ] ] / \n p
0000100 f o o = ~ / \ W / \n

You can see that it is, indeed, saved as Latin-1, not UTF-8.

The mystery continues. ;-)

Ian
--
Ian Macdonald | It's not whether you win or lose, it's how
(e-mail address removed) | you place the blame.
http://www.caliban.org/ |
|
|
 
I

Ian Macdonald

I can reproduce this 1.8.4
Click to expand...

Just to be clear, you are confirming that the following code:

foo = "préférées"
p foo =~ /[^[:alnum:]]/

prints nil in irb and 2 in a stand-alone script when in both cases your
locale is preset to nl_NL?

Ian
--
Ian Macdonald | On a clear disk you can seek forever.
(e-mail address removed) |
http://www.caliban.org/ |
|
|
 
R

Rob Biedenharn

I can reproduce this 1.8.4
Click to expand...

Just to be clear, you are confirming that the following code:

foo =3D "pr=E9f=E9r=E9es"
p foo =3D~ /[^[:alnum:]]/

prints nil in irb and 2 in a stand-alone script when in both cases =20
your
locale is preset to nl_NL?

Ian
--=20
Ian Macdonald | On a clear disk you can seek forever.
(e-mail address removed) |
http://www.caliban.org/ |
Click to expand...

I'm beginning to wonder if the original question is even accurate. =20
Doing nothing more than changing the encoding and re-saving the file =20
(where the value for foo was a cut-n-paste from the email), there =20
doesn't seem to be any discrpeancy between ruby and irb. (This =20
output is from ruby 1.8.5, but 1.8.2 was the same)

rab:code/ruby $ file regexp_and_alnum_versus_w.rb
regexp_and_alnum_versus_w.rb: ISO-8859 text
rab:code/ruby $ cat regexp_and_alnum_versus_w.rb
foo =3D "pr?f?r?es"
alnum =3D /[^[:alnum:]]/
dubya =3D /\W/

puts "foo\n =3D> #{foo.inspect}"
[ alnum, dubya ].each do |re|
puts "foo =3D~ #{re}\n =3D> #{foo =3D~ re}"
end
rab:code/ruby $ ruby regexp_and_alnum_versus_w.rb
foo
=3D> "pr\351f\351r\351es"
foo =3D~ (?-mix:[^[:alnum:]])
=3D> 2
foo =3D~ (?-mix:\W)
=3D> 2
rab:code/ruby $ irb -r regexp_and_alnum_versus_w.rb
foo
=3D> "pr\351f\351r\351es"
foo =3D~ (?-mix:[^[:alnum:]])
=3D> 2
foo =3D~ (?-mix:\W)
=3D> 2NameError: undefined local variable or method `eixt' for main:Object
from (irb):1rab:code/ruby $ file =20
regexp_and_alnum_versus_w.rbregexp_and_alnum_versus_w.rb: UTF-8 =20
Unicode text
rab:code/ruby $ cat regexp_and_alnum_versus_w.rb
foo =3D "pr=E9f=E9r=E9es"
alnum =3D /[^[:alnum:]]/
dubya =3D /\W/

puts "foo\n =3D> #{foo.inspect}"
[ alnum, dubya ].each do |re|
puts "foo =3D~ #{re}\n =3D> #{foo =3D~ re}"
end
rab:code/ruby $ ruby regexp_and_alnum_versus_w.rb
foo
=3D> "pr\303\251f\303\251r\303\251es"
foo =3D~ (?-mix:[^[:alnum:]])
=3D> 2
foo =3D~ (?-mix:\W)
=3D> 2
rab:code/ruby $ irb -r regexp_and_alnum_versus_w.rb
foo
=3D> "pr\303\251f\303\251r\303\251es"
foo =3D~ (?-mix:[^[:alnum:]])
=3D> 2
foo =3D~ (?-mix:\W)
=3D> 2

-Rob

Rob Biedenharn http://agileconsultingllc.com
(e-mail address removed)
 
R

Robert Klemme

I can reproduce this 1.8.4
Click to expand...

Just to be clear, you are confirming that the following code:

foo = "préférées"
p foo =~ /[^[:alnum:]]/

prints nil in irb and 2 in a stand-alone script when in both cases your
locale is preset to nl_NL?
Click to expand...

Another idea: maybe the readline lib interferes with encodings somehow
in IRB? What happens if you invoke your script from within IRB via "load"?

Kind regards

robert
 
I

Ian Macdonald

Another idea: maybe the readline lib interferes with encodings somehow
in IRB? What happens if you invoke your script from within IRB via "load"?
Click to expand...

It runs as if run from the command line:

irb(main):001:0> load 'foo'
2
2

Ian
--
Ian Macdonald | Man who falls in vat of molten optical
(e-mail address removed) | glass makes spectacle of self.
http://www.caliban.org/ |
|
|
 
I

Ian Macdonald

I'm beginning to wonder if the original question is even accurate.
Doing nothing more than changing the encoding and re-saving the file
(where the value for foo was a cut-n-paste from the email), there
doesn't seem to be any discrpeancy between ruby and irb. (This
output is from ruby 1.8.5, but 1.8.2 was the same)

rab:code/ruby $ file regexp_and_alnum_versus_w.rb
regexp_and_alnum_versus_w.rb: ISO-8859 text
rab:code/ruby $ cat regexp_and_alnum_versus_w.rb
foo = "pr?f?r?es"
alnum = /[^[:alnum:]]/
dubya = /\W/

puts "foo\n => #{foo.inspect}"
[ alnum, dubya ].each do |re|
puts "foo =~ #{re}\n => #{foo =~ re}"
end
rab:code/ruby $ ruby regexp_and_alnum_versus_w.rb
foo
=> "pr\351f\351r\351es"
foo =~ (?-mix:[^[:alnum:]])
=> 2
Click to expand...

What is your locale? I strongly suspect it's either unset or set to C.
In those cases, I get the same results as you.

If you use en_US or nl_NL, you'll find (or at least, I find) that
'foo =~ /[^[:alnum:]]/' returns nil in irb and 2 from a stand-alone
script.

In fact, even irb returns a different value from the command line. This
is bizarre:

$ irb -f < foo2
foo = "préférées"
"pr\351f\351r\351es"
foo =~ /[^[:alnum:]]/
2
foo =~ /\W/
2

$ irb
irb(main):001:0> foo = "préférées"
=> "pr\351f\351r\351es"
irb(main):002:0> foo =~ /[^[:alnum:]]/
=> nil
irb(main):003:0> foo =~ /\W/
=> 2

As you can see, interactively irb returns nil for that first regex match.

Ian
--
Ian Macdonald | The cost of living hasn't affected its
(e-mail address removed) | popularity.
http://www.caliban.org/ |
|
|
 
P

Phrogz

As you can see, interactively irb returns nil for that first regex match.
Click to expand...

Aren't you making the assumption that it's the regex at fault here,
and not the encoding of the string when you enter it in irb?

What if you do:

gavinkistner$ cat set_foo.rb
$foo = "préférés"

gavinkistner$ irb
irb(main):001:0> load 'set_foo.rb'
=> true
irb(main):001:0> $foo =~ /[^[:alnum:]]/
???
 
I

Ian Macdonald

As you can see, interactively irb returns nil for that first regex match.
Click to expand...

Aren't you making the assumption that it's the regex at fault here,
and not the encoding of the string when you enter it in irb?

What if you do:

gavinkistner$ cat set_foo.rb
$foo = "préférés"

gavinkistner$ irb
irb(main):001:0> load 'set_foo.rb'
=> true
irb(main):001:0> $foo =~ /[^[:alnum:]]/
???
Click to expand...

$ irb
irb(main):001:0> load 'foo'
=> true
irb(main):002:0> $foo
=> "pr\351f\351r\351es"
irb(main):003:0> $foo =~ /[^[:alnum:]]/
=> nil

It's still nil, I'm afraid.

Ian
--
Ian Macdonald | ..disk or the processor is on fire.
(e-mail address removed) |
http://www.caliban.org/ |
|
|
 
I

Ian Macdonald

It runs as if run from the command line:

irb(main):001:0> load 'foo'
2
2
Click to expand...

I beg your pardon. I must have had the locale set incorrectly on that
run. It runs as if typed interactively into irb:

$ irb
irb(main):001:0> load 'foo'
nil
2

Ian
--
Ian Macdonald | A small town that cannot support one lawyer
(e-mail address removed) | can always support two.
http://www.caliban.org/ |
|
|
 
P

Phrogz

I beg your pardon. I must have had the locale set incorrectly on that
run. It runs as if typed interactively into irb:

$ irb
irb(main):001:0> load 'foo'
nil
2
Click to expand...

Phewsh. Combined with the behavior you reported for loading a global
and then matching in IRB, I had feared the world had gone insane. At
least its consistently weird and the regexp match is, in fact, the
culprit.
 
R

Rob Biedenharn

Phewsh. Combined with the behavior you reported for loading a global
and then matching in IRB, I had feared the world had gone insane. At
least its consistently weird and the regexp match is, in fact, the
culprit.
Click to expand...

Why don't you just find out which characters are in the [:alnum:] and
\w sets?
alnums = (0..0377).select {|c| c.chr =~ /[[:alnum:]]/ }.map {|c|
Click to expand...
Click to expand...
c.chr}.join
=> "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"=> "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz"

$ LANG=nl_NL irb
alnums = (0..0377).select {|c| c.chr =~ /[[:alnum:]]/ }.map {|c|
Click to expand...
Click to expand...
c.chr}.join
=> "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz\252
\265\272\300\301\302\303\304\305\306\307\310\311\312\313\314\315\316
\317\320\321\322\323\324\325\326\330\331\332\333\334\335\336\337\340
\341\342\343\344\345\346\347\350\351\352\353\354\355\356\357\360\361
\362\363\364\365\366\370\371\372\373\374\375\376\377"=> "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz"

sheesh!

-Rob

Rob Biedenharn http://agileconsultingllc.com
(e-mail address removed)
 
I

Ian Macdonald

Why don't you just find out which characters are in the [:alnum:] and
\w sets?

$ LANG=nl_NL irb
alnums = (0..0377).select {|c| c.chr =~ /[[:alnum:]]/ }.map {|c|
Click to expand...
Click to expand...
c.chr}.join
=> "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz\252
\265\272\300\301\302\303\304\305\306\307\310\311\312\313\314\315\316
\317\320\321\322\323\324\325\326\330\331\332\333\334\335\336\337\340
\341\342\343\344\345\346\347\350\351\352\353\354\355\356\357\360\361
\362\363\364\365\366\370\371\372\373\374\375\376\377"=> "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz"
Click to expand...

Yes, but all this really does is indicate that the irb behaviour is
the correct one.

When I run this in a stand-alone script, I get this:

$ LANG=nl_NL ./foo
"0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"

It's almost as if the locale isn't being propagated to the process via
the environment. But...

$ LANG=nl_NL ruby -e "puts ENV['LANG']"
nl_NL

...it _is_ being propagated.

Is is the same for you?

Ian
--
Ian Macdonald | When a man is tired of London, he is tired
(e-mail address removed) | of life. -- Samuel Johnson
http://www.caliban.org/ |
|
|
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Is [:alnum:] equivalent to [A-Za-z0-9]? 1
Strange SimpleDelegator behaviour regarding blocks 2
regex failure notifcation 1
help with regex 3
Singleton class's original class 11
Behaviour of Enumerables reject vs. select mixed into Hash 50
Strange whitespace parsing behavior on Ruby 1.8.7 (patchlevel249/302) 12
using a variable to build a regex 3

Members online

No members online now.
Total: 77 (members: 0, guests: 77)
Robots: 264

Forum statistics

Threads
473,982
Messages
2,570,185
Members
46,736
Latest member
AdolphBig6

Latest Threads

Top