Content replacement features -- use removed content to inform measures of added content in diffs

[eranroz]

revscoring and AbuseFilter (and other tools) allow to catch easily vandalism that use some "bad regex"/bad words. However, the existing tools don't have ability to identify word replacements:
E.g "Barack Obama is president" => "Barack Obama is terrorist". While "terrorist" is not a bad word, a replacement of some other word to terrorist is most probably bad.

While it isn't always obvious there is "alignment" between words in the previous and the new revisions, if such exist the tool can use it.

[halfak]

We already have a diff Datasource that can be used to examine removed text -- which is what you would use to identify such replacements. I'm updating the title to reflect that this is a request for a new class of Feature.

[halfak]

OK. In order to get something like this together, we need a proposal for how we might get signal for badword replacements. E.g. we use wordnet[1] to compare the similarity of words that were replaced. We use could also use something like Levenshtein distance. That would make a change like "faster"->"fastest" score low, but "banana"->"pineapple" score highly.

1. https://wordnet.princeton.edu/

[halfak]

BTW, we'll likely need to define a new datasource of word/token pairs from revscoring.datasource.diff.operations

[he7d3r]

Notice that for Portuguese Wikipedia, which has content in both variants pt-BR and pt-PT, replacements such as "polinómio" (pt-PT) → "polinômio" (pt-BR) are considered damaging by the community, and are usually reverted to the variant used by the first author (example).

So, on ptwiki we will get high revert/damaging scores for word replacements with small Levenshtein distances.

[halfak]

he7d3r, typo fixes will also be caught in this small Levenshtein distance. It seems like we need to match same-word-different-spelling situations. How will we know that the new word is the same as the old word?

[he7d3r]

We probably won't know it is the same, unless we have a dictionary mapping one to the other. But we could at least know that there was a word replacement in which a word from pt-PT (respec. pt-BR) was replaced by a word from pt-BR (respec. pt-PT): just check if it is a "misspelling" in one language variant (e.g. not in the dictionary for pt-BR) but not the other (the dictionary for pt-PT). (edit: BTW: this reminds me of syntactic congruence as a way to roughly detect words in the same gramatical classes)

Having a list of word equivalences between language variants is difficult to maintain, but if we go that route, there are a few things we can use as a start for Portuguese:

• Lists of equivalences: https://pt.wikipedia.org/wiki/WP:Dicion%C3%A1rio
• Regexes for common equivalences: https://pt.wikipedia.org/wiki/MediaWiki:Gadget-LanguageConverter.js