Language detection module..

[AR]

I had some system maintaince..

Text are long at least 15Kb up to some megs. It shouldn't be so hard to
find out in which language is text, because it is long enought. About 70%
of texts are labeled in which lnguage they are, rest are not. But among
remaining 30% there is at least 90% English, some in German, few French
and few Italian and maybe some Duch, Chinese, .. Russian.. At least I need
tool that will filter out English texts as accurate as possible.

So I think that some kind of statistical approach would be fine...

About 100%.. yes.. I know it is impossible...

 

[Neil Shadrach]

pkent:
[ other wise advice snipped ]

If it uses the w-circumflex character a fair bit it could be Welsh

Though at this point in time the accent is often omitted because it has not been well-spported in the past.
I'm sure the same is true for other languages ( eg Poles using "l" in place of "l with stroke" )
As far as Welsh is concerned the presence of the particles "yn" and "yr" is a
pretty good discriminator. You'd be pretty hard pressed to find a Welsh text without them and they don't
appear to occur in other languages [1]. Do other languages have good discriminators? I presume groups of
related languages would be a problem.



[1] I'm basing this on Google rather than academic rigour

 

[pkent]

pkent:
[ other wise advice snipped ]

If it uses the w-circumflex character a fair bit it could be Welsh

Though at this point in time the accent is often omitted because it has not
been well-spported in the past.

Also very true. I've noticed other chars that used to be better
supported on windows than Macs - possibly thorn and/or eth, but I
haven't got the code charts to hand to check. Anyway, roll on more and
more use of unicode, and possibly more and more characters _in_ Unicode

[1] I'm basing this on Google rather than academic rigour

Aahh, that's good enough

P