Fails to recognize simple English words/phrases

[ykessler]

Hi Peter- This project looks exciting, however we can't get the most basic tests working when trying simple English words or phrases:

require 'whatlanguage'
=> true
"test".language
=> :dutch
"hello".language
=> :arabic
"how are you".language
=> :arabic

[peterc]

Yes, due to it having a word-oriented probablistic approach, it's pretty useless unless you have at least a few words (the last example is a bit annoying in that regard though).

It turns out there are better (though more complicated) ways to do language detection that both use less memory and are more accurate on smaller texts and I've been experimenting with the techniques demonstrated in some papers for a future version (namely n-gram histograms and Unicode character ranges - since English text doesn't tend to use Arabic orthography or vice versa).

[ykessler]

I think a first-pass test of char ranges is absolutely the way to go- for special character languages like Hebrew and Arabic that would in fact be all you need, and even a single word could be determined definitively and efficiently.

I also stumbled upon another dead-simple test to root out ideographic languages from the rest:

class String
  def multibyte?
    chars.count < bytes.count
  end
end

"可口可樂".multibyte? #=> true
"qwerty".multibyte? #=> false

Taken from http://stackoverflow.com/a/7201988/165673

[cmaujean]

I'm experiencing similar issues:

Loading development environment (Rails 3.2.18)
2.0.0-p576 :001 > require 'whatlanguage'
 => false
2.0.0-p576 :002 > wl = WhatLanguage.new(:all)
 => #<WhatLanguage:0x007fba5244d748 @selection=[:all]>
2.0.0-p576 :003 > "The body of the answer".language
 => :english
2.0.0-p576 :004 > "The body of the answer.".language
 => :arabic

[cmaujean]

This is a blocker for me. I have more examples:

2.0.0-p576 :004 > "You know that this is the body of the answer.".language
 => :arabic
2.0.0-p576 :005 > "Once upon a time, you know that this is the body of the answer.".language
 => :arabic
2.0.0-p576 :006 > "A funny thing happened on the way to the market".language
 => :norwegian
2.0.0-p576 :007 > "A funny thing happened on the way to the market.".language
 => :norwegian
2.0.0-p576 :012 > "I can end many things with answer.".language
 => :hungarian
2.0.0-p576 :013 > "I can end many things with answer".language
 => :english

[cmaujean]

even worse:
2.0.0-p576 :014 > "I was walking to Café de la Presse the other day and that is when I got my answer.".language
=> :norwegian

[peterc]

Thanks for bringing this up. It's almost comically bad on some of the examples! :) I'm going to be looking into it as it shouldn't be that bad, and these results are suggesting the bitfield isn't wide enough to get the right level of accuracy.

[obahareth]

How about using unicode ranges to determine a string's language? (Obviously you wouldn't make a function for every language like this silly short example here)

class String
  def arabic?
    index(/[\u0600-\u06FF]/) != nil
  end
end

puts 'اللغة التي هي هذه؟'.arabic? #=> true
puts 'You know that this is the body of the answer.'.arabic? #=> false

EDIT
Wow I just noticed my page was rendering weird after posting the comment and my idea has already been suggested.

[subimage]

In the "comically bad" camp here. Was hopeful this would be useful on some article headlines for a project I'm working on but it seems like there's lots of work to do!

I Puritani review – Bellini, gracefully poised in the Troubles
- english

PauseMusicale.com
- farsi

Quickdraw: Big Grams, Chvrches, Death Cab For Cutie, Small Black, Youth Lagoon
- english

video con Salmo e Noyz Narcos
- english

Jag Och Min Far
- english

stream these :: Palm + Dan Friel + Guerilla Toss + Stove + Night Birds + Swings + Wildhoney
- english

Angeldust, from Panorama, out on 4... – Walzerkönig. Music blog from Vienna, Austria
- spanish

Two Bodies (Henri Remix)
- russian

The Home of Great Thinkers
- arabic

Rhonda Smith Interview
- norwegian

Alps Review from Music Emissions
- english

All Yours [Album Review]
- french

SXSW 2015 - OEB Spreadsheet #48 Sunday morning...
- russian

Martin Castillo -(Version Official Nueva Era Music)2012 2013 video at savevid.com
- english

Mi destino -4CzxAQmFcd0 video at savevid.com
- english

filous — How Hard I Try (Robotaki Remix)
- arabic

[mmahalwy]

Any updates on this?! I am getting this problem too and it's a huge huge problem!

Example:

"believe".language => :arabic
"oh".language => :arabic
"you".language => :arabic

[peterc]

The official advice based on experiments is now that on texts under around several words in length, you are unlikely to get reliable results, simply due to the algorithm used and the tradeoffs of using bloom filters for efficiency. We will add/change to a Unicode codepoint presence test in future (as was used in a comment above).

[sixtyfive]

So this is closed, but would it not make sense to add some sort of dictionary-based approach? One of the first examples in this issue was "How are you?" ... I just tried that as well, more than two years later, and the result has changed from :arabic to :hungarian:

main » wl.language "How are you?"
=> :hungarian

[peterc]

It does use a dictionary based approach. But because the dictionaries are heavily "compressed" using a Bloom filter, you get a certain percentage of false positives. With texts beyond a handful of words, this is no problem, but for anything very short, it does not work.

But yes, a full dictionary approach could be used, but then the memory use would be significantly higher and performance much lower. Moving to a n-gram detection system will resolve much of this but it's all about having the time to implement it.

[sixtyfive]

Hmm, I get that single words would pose problems, but "how are you" is a proper sentence and not of unusual length as far as English sentences go. For the project I'm currently working on I had to choose CLD now, which is a shame because I liked whatlanguage better, API-wise.

[peterc]

CLD will definitely give you better results, but has some key downsides of its own. WhatLanguage is extremely old now (the core is about 9 years old) and does have some major limitations, which I am always quick to point out.

The latest research I've seen for doing this sort of thing in a memory efficient way is to use a combination of n-gram and codepoint histograms. I understand how this works and could reimplement it with a few days of work, but it's not time I have right now, unfortunately.

One thing that someone could do would be to have a dictionary of the X most popular words in each language and that could "reinforce" the results. That would probably take care of most shorter texts, but it would definitely be a hack.