String problem

[Fresh Mix]

What wrong?

# irb
irb(main):001:0> xxx = "лошадь"
=> "\320\273\320\276\321\210\320\260\320\264\321\214"
irb(main):002:0> xxx.length
=> 12

 

[Tom Cloyd]

What wrong?

# irb
irb(main):001:0> xxx = "лошадь"
=> "\320\273\320\276\321\210\320\260\320\264\321\214"
irb(main):002:0> xxx.length
=> 12

I assume you're wondering why each character appears to be represented
by two bytes - and I believe it's because the encoding is, of necessity,
UTF-8 or something very similar. If I recall correctly, this encoding is
designed to be able to represent the world's alphabets, etc., rather
than merely the limited character set used in western European
languages, and so two bytes must be used to allow for all the possibilities.

If I don't have this quite right (or right at all), I'm sure I'll be set
right by those who know more here (and they are legion!).

t.


--

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tom Cloyd, MS MA, LMHC - Private practice Psychotherapist
Bellingham, Washington, U.S.A: (360) 920-1226
<< (e-mail address removed) >> (email)
<< TomCloyd.com >> (website)
<< sleightmind.wordpress.com >> (mental health weblog)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 

[Robert Klemme]

I assume you're wondering why each character appears to be represented
by two bytes - and I believe it's because the encoding is, of necessity,
UTF-8 or something very similar. If I recall correctly, this encoding is
designed to be able to represent the world's alphabets, etc., rather
than merely the limited character set used in western European
languages, and so two bytes must be used to allow for all the possibilities.

If I don't have this quite right (or right at all), I'm sure I'll be set
right by those who know more here (and they are legion!).

Actually I do not call myself in when it comes to encodings in Ruby.
But I believe there is one important bit of information missing that's
needed to properly answer the OP's question: what Ruby version did you use?

Kind regards

robert

 

[Fresh Mix]

what Ruby version did you
use?

$ ruby -v
ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux]

 

[7stud --]

# irb
irb(main):001:0> xxx = "лошадь"
=> "\320\273\320\276\321\210\320\260\320\264\321\214"
irb(main):002:0> xxx.length
=> 12

What wrong?

In 1.8.* versions, ruby doesn't recognize unicode, where characters are
represented by multiple bytes. ruby thinks everything in an ascii
character where characters are represented by one byte.

Try this:

xxx = "лошадь"
puts xxx.length

--output:--
12

$KCODE = "u"
require 'jcode'

puts xxx.jlength

--output:--
6

xxx.each_char do |u|
puts u
end

--output:--
л
о
ш
а
д
ь

 

[7stud --]

In 1.8.* versions, ruby doesn't recognize unicode, where characters are
represented by multiple bytes. ruby thinks everything in an ascii
character where characters are represented by one byte.

Corrections:

In 1.8.* versions, ruby doesn't recognize unicode, where characters [may
be]
represented by multiple bytes. ruby thinks everything [is] an ascii
character where characters are represented by one byte.

 

[7stud --]

But I believe there is one important bit of information missing that's
needed to properly answer the OP's question: what Ruby version did you
use?

Why is that relevant? Can unicode be switched off in ruby 1.9?

 

[Robert Klemme]

Why is that relevant? Can unicode be switched off in ruby 1.9?

It is relevant because handling of encodings has significantly changed
between 1.8 and 1.9, which I believe your other posting demonstrates.

Cheers

robert

 

[7stud --]

Regular
expressions in 1.8.* recognize UTF-8, EUC-JP, and Shift_JIS. So you
can handle Unicode strings by using regular expressions.

Too vague.

And those interested in how all that works may find this series on my
blog helpful:

http://blog.grayproductions.net/articles/understanding_m17n

Excellent website. <c-word here>

Here is something that is unclear:

----------
To use the jcode library, set $KCODE and then require the library.
Setting $KCODE first is important, and you will receive a warning if you
require jcode without setting it (as long as you took my advice and
turned ****them*** on)...

http://blog.grayproductions.net/articles/the_kcode_variable_and_jcode_library
---------

In the sentence:

-------
Setting $KCODE first is important, and you will receive a warning if you
require jcode without setting it (as long as you took my advice and
turned them on)...
--------

'it' and 'them' are pronouns, which should refer to nouns. The pronoun
'it' looks like it might refer to 'jcode' when 'it' actually refers to
'$KCODE'. That is pretty easy to sort out.

However, what does 'them' refer to? 'them' should refer to a plural
noun, so if you actually stop and try to sort it out rather than just
dismissing the whole paragraph in confusion, 'them' looks like it must
refer to '$Kcode' and 'jcode'. However, that doesn't make sense because
you don't 'set' jcode--you require jcode.

Apparently, 'them' refers to 'warning', which is not only grammatically
incorrect but it is very hard to make that association. In any case, in
that sentence if you change 'it' and 'them' to $KCODE and 'warnings'
respectively, you will change a confusing and unreadable sentence into a
sentence whose clarity will be unmatched in modern literature:

-----
Setting $KCODE first is important, and you will receive a warning if you
require jcode without setting $KCODE (as long as you took my advice and
turned warnings on with -w)...
______

I'd bet that 90% of the readers of your article stop reading at that
exact spot.

 

[Aldric Giacomoni]

Too vague.




Excellent website. <c-word here>

Here is something that is unclear:

----------
To use the jcode library, set $KCODE and then require the library.
Setting $KCODE first is important, and you will receive a warning if you
require jcode without setting it (as long as you took my advice and
turned ****them*** on)...

http://blog.grayproductions.net/articles/the_kcode_variable_and_jcode_library
---------

In the sentence:

-------
Setting $KCODE first is important, and you will receive a warning if you
require jcode without setting it (as long as you took my advice and
turned them on)...
--------

'it' and 'them' are pronouns, which should refer to nouns. The pronoun
'it' looks like it might refer to 'jcode' when 'it' actually refers to
'$KCODE'. That is pretty easy to sort out.

However, what does 'them' refer to? 'them' should refer to a plural
noun, so if you actually stop and try to sort it out rather than just
dismissing the whole paragraph in confusion, 'them' looks like it must
refer to '$Kcode' and 'jcode'. However, that doesn't make sense because
you don't 'set' jcode--you require jcode.

Apparently, 'them' refers to 'warning', which is not only grammatically
incorrect but it is very hard to make that association. In any case, in
that sentence if you change 'it' and 'them' to $KCODE and 'warnings'
respectively, you will change a confusing and unreadable sentence into a
sentence whose clarity will be unmatched in modern literature:

-----
Setting $KCODE first is important, and you will receive a warning if you
require jcode without setting $KCODE (as long as you took my advice and
turned warnings on with -w)...
______

I'd bet that 90% of the readers of your article stop reading at that
exact spot.

Alright, so the man who created Ruby doesn't write well enough for you,
and neither does one of the big guys in the community. Maybe you'd like
to offer yourself as proofreader for them, instead of programmer for
others? I liked James' series of articles and was able to read it just
fine... And english is my 4th language.

-- Aldric

 

[James Gray]

I'd bet that 90% of the readers of your article stop reading at that
exact spot.

I think that's dramatically overstating the problem.

I do really appreciate your feedback though. Obviously I want this
content to be as helpful to everyone as possible. I've adjusted the
sentence as you recommend. Thanks.

James Edward Gray II

 

[Adam Gardner]

I think that's dramatically overstating the problem.

I do really appreciate your feedback though. Obviously I want this
content to be as helpful to everyone as possible. I've adjusted the
sentence as you recommend. Thanks.

James Edward Gray II

Aw, you think so? I kinda liked the idea that, since I understood it
immediately, I was in the top 10% of all Ruby programmers, themselves
already a smart bunch.

 

[7stud --]

I've adjusted the
sentence as you recommend. Thanks.

Ahh..pure poetry. I didn't know about the 'u' flag for a regex. Thank
you.