iconv problems with different machines

[Raymond O'Connor]

Hi,

I have the following piece of code:

ic = Iconv.new('US-ASCII//TRANSLIT', 'UTF-8')
puts ic.iconv("Aüthor")

1. on my local machine (OSX 10.5) when I run this, I get the output:
A"uthor

2. when I run this same code on my debian server (via rake executed
through a capistrano task) I get the output: A?thor

3. when I run this same code on my debian server (via irb), I get:
Author

Both 1 and 3 are acceptable output to me, however I cant figure out how
to get my program to output the correct result on my server when I run
it through a capistrano task. Is there some environment variable I need
to set? From reading other posts, I've tried adding at the top of my
file:
$KCODE = "u"
require 'jcode'
ENV['LANG'] = 'en_US.UTF-8'
ENV['LC_CTYPE'] = 'en_US.UTF-8'

still doesn't fix the issue. Any help would be greatly appreciated.

Thanks,
Ray

 

[Raymond O'Connor]

Actually I found some other posts about this same issue from awhile
ago... Appears there's no solution.

I stopped using the iconv library and instead switched to the iconv
system command and that seems to work. Not the best solution, but at
least it works....

 

[Xavier Noria]

Actually I found some other posts about this same issue from awhile
ago... Appears there's no solution.

I stopped using the iconv library and instead switched to the iconv
system command and that seems to work. Not the best solution, but at
least it works....

I have not been able to understand where is exactly the difference, =20
but looks like depending on the system/version/something the =20
transliteration tables are just different. At ASPgems we wrote this =20
hand-crafted normalizer which we know is portable for sure (note that =20=

it uses Rails #chars and does a bit more stuff, but you see the idea):

def self.normalize(str)
return '' if str.nil?
n =3D str.chars.downcase.strip.to_s
n.gsub!(/[=C3=A0=C3=A1=C3=A2=C3=A3=C3=A4=C3=A5=C4=81=C4=83]/, =
'a')
n.gsub!(/=C3=A6/, 'ae')
n.gsub!(/[=C4=8F=C4=91]/, 'd')
n.gsub!(/[=C3=A7=C4=87=C4=8D=C4=89=C4=8B]/, 'c')
n.gsub!(/[=C3=A8=C3=A9=C3=AA=C3=AB=C4=93=C4=99=C4=9B=C4=95=C4=97]/, =
'e')
n.gsub!(/=C6=92/, 'f')
n.gsub!(/[=C4=9D=C4=9F=C4=A1=C4=A3]/, 'g')
n.gsub!(/[=C4=A5=C4=A7]/, 'h')
n.gsub!(/[=C3=AC=C3=AC=C3=AD=C3=AE=C3=AF=C4=AB=C4=A9=C4=AD]/, =
'i')
n.gsub!(/[=C4=AF=C4=B1=C4=B3=C4=B5]/, 'j')
n.gsub!(/[=C4=B7=C4=B8]/, 'k')
n.gsub!(/[=C5=82=C4=BE=C4=BA=C4=BC=C5=80]/, 'l')
n.gsub!(/[=C3=B1=C5=84=C5=88=C5=86=C5=89=C5=8B]/, 'n')
n.gsub!(/[=C3=B2=C3=B3=C3=B4=C3=B5=C3=B6=C3=B8=C5=8D=C5=91=C5=8F=C5=8F=
]/, 'o')
n.gsub!(/=C5=93/, 'oe')
n.gsub!(/=C4=85/, 'q')
n.gsub!(/[=C5=95=C5=99=C5=97]/, 'r')
n.gsub!(/[=C5=9B=C5=A1=C5=9F=C5=9D=C8=99]/, 's')
n.gsub!(/[=C5=A5=C5=A3=C5=A7=C8=9B]/, 't')
n.gsub!(/[=C3=B9=C3=BA=C3=BB=C3=BC=C5=AB=C5=AF=C5=B1=C5=AD=C5=A9=C5=B3=
]/, 'u')
n.gsub!(/=C5=B5/, 'w')
n.gsub!(/[=C3=BD=C3=BF=C5=B7]/, 'y')
n.gsub!(/[=C5=BE=C5=BC=C5=BA]/, 'z')
n.gsub!(/\s+/, ' ')
n.delete!('^ a-z0-9_/\\-')
n
end

-- fxn

 

[Raymond O'Connor]

Hi Xavier,

I like that solution even better. Thanks for sharing!

Best,
Ray

 

[marc]

Raymond O'Connor said...

Hi,

I have the following piece of code:

ic = Iconv.new('US-ASCII//TRANSLIT', 'UTF-8')
puts ic.iconv("Aüthor")

1. on my local machine (OSX 10.5) when I run this, I get the output:
A"uthor

2. when I run this same code on my debian server (via rake executed
through a capistrano task) I get the output: A?thor

3. when I run this same code on my debian server (via irb), I get:
Author

Both 1 and 3 are acceptable output to me, however I cant figure out how
to get my program to output the correct result on my server when I run
it through a capistrano task. Is there some environment variable I need
to set? From reading other posts, I've tried adding at the top of my
file:
$KCODE = "u"
require 'jcode'
ENV['LANG'] = 'en_US.UTF-8'
ENV['LC_CTYPE'] = 'en_US.UTF-8'

still doesn't fix the issue. Any help would be greatly appreciated.

I've found a lot of bugs with the MRI Iconv and now only use it with
JRuby - which, I suspect, uses the Java SE convertors.