Detect file encoding utf-8

[Rebhan, Gilbert]

Hi,

I want to check the file encoding of files in a directory.
Until now i have tried =3D

# found in an older thread in comp.lang.ruby
class String
def utf8?
unpack('U*') rescue return false
true
end
end
# found in an older thread in comp.lang.ruby

utf=3DArray.new
others=3DArray.new
Dir["Y:/test/**/*.xml"].each do |path|
open(path) { |f|=20
(f.read.utf8?) ? uts<<path : others<<path
}
end

and also tried the chardet Library (no ruby documentation included)
like that

require 'UniversalDetector'

utf=3DArray.new
others=3DArray.new
Dir["Y:/test/**/*.xml"].each do |path|
open(path) { |f|=20
UniversalDetector.chardet(f.read) =3D~ /utf-8/ ?
uts<<path : others<<path
}
end
puts utf.join(",")
puts others.join(",")


Are there better / simpler ways ?

Regards, Gilbert

 

[Richard Conroy]

You could use some regular expressions, to search for code points in
your source string that are outside of what is legal for UTF-8.

Basically you assume it is UTF-8, and then reject it if it contains illegal
or unknown code points.

 

[Xavier Noria]

I want to check the file encoding of files in a directory.

Have you tried charguess?

http://raa.ruby-lang.org/project/charguess

-- fxn

 

[Gilbert Rebhan]

Have you tried charguess?

http://raa.ruby-lang.org/project/charguess

No, how to install it ?

only =

charguess.c
extconf.rb
MANIFEST
sample.rb

in the tarfile.

Regards, Gilbert