Limitations of the translation API

[Xowap]

Hi there,

I'm not entirely certain of where to post this but given that I picked up this repo from the Chromium source I'm gonna guess this is the best place to land it.

The translation API is already an amazing thing to have available in the browser, and the current implementation brings me joy and wonders. However I'm seeing improvements that I would really like to have available.

Current issues

For example, if like me you are developing an extension which allows for translating the user's messages, especially in a WhatsApp/IM context, you might find that some notions are hard to take into account:

The speaker's gender

For example in Spanish, "I am tall" would translate to "Soy alto" or "Soy alta" depending whether you're a man or a woman.

This is obviously not the only gender-related issue, but it's a big one if you are working on IM (thus mostly first-person sentences).

Relative references

Particularly in the context of a conversation, you might be referencing the previous message, that you are not passing to the translation API. This can impact the declination of adjectives, disambiguate meanings, etc.

For example in French you could have:

• "Qui s'occupe de la baguette ?" ("who takes care of the baguette?")
• "Je la prends" ("I'll take it")

Again, a gender-related issue. The sentence could be entirely different if it were not for a baguette. For example "pain" is masculine so you would say "Je le prends".

In this case if you are only translating the second sentence, and you cannot contextualize the translation with the first one, then you don't know which gender to use in there.

Regionalisms

The language is different depending on the region. Obviously Brits will get upset whenever they see "color" instead of "colour" but it can get a lot more dramatic with other languages. For example French French is entirely different from French Canadian on any word that appeared after the 17th Century. Or even regions of France will not agree on some words (have a look at the "chocolatine" debate). Examples are countless.

My point is that different regions and different people are going to pick different words or idioms and this is absolutely not bound by country or language.

Source language

The Translator API requires a sourceLanguage, which is in my opinion not bringing any value.

For example, in my current company, our standard meeting will happen in two if not three languages. Take a transcript of that, you will have French, Spanish and English in there. What is the source language of that?

On the other hand, when you're going to be translating, you will probably want the smoothest user experience and thus not ask for the source language. Most of the time you are probably going to use the LanguageDetector.

And finally, the goal of translation is not to be gate-keeping incorrect spelling or grammar. We want to understand the source material, whichever its language, and generate an output in the target language.

So essentially today you need to start two APIs (language detection and translation), download the associated models, all that to determine a single source language while the source material is very probably mixed.

Proposed improvements

As an avid LLM user, I have been using LLMs to do translation for quite a while now, and my workflow essentially is:

• Give a bit of context (previous message, lexical field, etc)
• A bit about myself
• Target language
• What I want to translate

For example:

I'm a man, please translate this message for my doctor in Spanish from Spain: Could you renew my receta? Thanks!

This workflow works well for me, because:

• It does not ask me for the source language
• I can ask whatever context is relevant to the conversation, using my judgement

At the end of the day this suggests that a more powerful API could be possible. For example modify the create options:

/**
 * Core options required for creating a translator
 */
export interface TranslatorCreateCoreOptions {
    /** Optional source language hint in BCP-47 format (e.g., "en", "es", "zh") */
    sourceLanguage?: string;
    /** Target language in BCP-47 format (e.g., "en", "es", "zh") */
    targetLanguage: string;
    /** Semi-formal information about the dialect/region to translate for */
    targetDialect: string; // "es", "south-west of France", etc
}

As you can see:

• sourceLanguage becomes optional, and it's a hint not a hard requirement (helps with disambiguation, but that's it)
• targetDialect gives more information about which variants of the language the developer wants. Semi formal, because you might want to instruct regionalisms, tone, use of specific slang, etc

And add to the translate options:

/**
 * Options for translation operations
 */
export interface TranslatorTranslateOptions {
    /** AbortSignal to cancel the translation */
    signal?: AbortSignal;
    /** Plain-text general context of the conversation (speaker's gender, lexical field, etc) */
    context?: string;
    /** Previous parts of the text/conversation being translated */
    previousText?: string;
}

Essentially, we're adding two things in there:

• context is essentially an add-on to the system prompt, to give any info deemed valuable by the developer. Given all the possible nuances that can happen, it seems futile to me to model it any further, we can probably let the developer give us plain-text details of what they know in regards to the expected translation
• previousText that would be the previous parts of the conversation, essentially we'd ask the LLM to complete this text by translating the input text

Conclusion

After using these APIs for my app, those are the things that I feel are needed for my use-case. I'm sure that I'm all kinds of wrong between not posting in the right form, place or with the right constraints in mind. But let me know if this can go any further, and if so I will happily contribute where required.

[tomayac]

This is fantastic feedback and this repository is exactly the right forum to voice it. I agree with all the shortcomings that you have identified, with one suggestion (that was actually mentioned as a limitation in your prose): instead of sourceLanguage, you likely rather want sourceLanguages (plural) as the hint.

FYI our Product Manager @KenjiBaheux.

[Xowap]

Thanks!

Indeed, sourceLanguages could be a list. My idea was to keep backward compatibility, but it makes a lot of sense to have it accept a list of languages 🤓

[KenjiBaheux]

Thanks for this very useful feedback!

We've seen similar scenarios where a little bit of context (e.g. localization rules, genders, domain), or previous text / translated text should result in more accurate translations. This would typically require using an LLM which isn't how the API has been implemented in Chrome as we were trying to balance performance, quality, reach and capabilities.

That said, I'll check if our translation models might support "previous text" to guide the translation of new text input (consistency). For the more advanced scenarios, we'd need to look into using an LLM and augmenting the current API surface.

Regarding the sourceLanguage(s) aspect, it might be doable with an LLM backed version of the API but may not work well with the language pairs based models we are currently using.

[Xowap]

From what I understand, these are the Google Translate models re-purposed for this, right?

In terms of performance, it is definitely blowing LLMs out of the water, so that would be an important factor to be aware of as a developer.

I wonder if some of the issues could be circumvented with the existing model by placing a sentinel character separating the context and the message, translating the whole thing and then cutting back at the sentinel?

• Keeping gender works even with a whole paragraph in the middle
• No confusion on the baguette
• You can influence the meaning of words
• But it's not as smart as a LLM (for context, "check" translates in most languages in either "verify" or "put a checkmark", which is by far the most common pitfall in computer UI translation, according to me)

[tomayac]

From what I understand, these are the Google Translate models re-purposed for this, right?

You can mostly think of it as this, yes.

In terms of performance, it is definitely blowing LLMs out of the water, so that would be an important factor to be aware of as a developer.

+1.

I wonder if some of the issues could be circumvented with the existing model by placing a sentinel character separating the context and the message, translating the whole thing and then cutting back at the sentinel?

• Keeping gender works even with a whole paragraph in the middle
• No confusion on the baguette
• You can influence the meaning of words
• But it's not as smart as a LLM (for context, "check" translates in most languages in either "verify" or "put a checkmark", which is by far the most common pitfall in computer UI translation, according to me)

That's actually a pretty clever hack and I just confirmed it works with the Translation API:

const translator = await Translator.create({
  sourceLanguage: 'en',
  targetLanguage: 'es',
});

await translator.translate('I am a woman\n\n✂️\n\nI am tall.');
// 'Soy mujer ✂️ Soy alta.'

Maybe this kind of microsyntax with internally hidden pre- and post-processing could be shoe-horned on top of the existing API:

const translator = await Translator.create({
  sourceLanguage: 'en',
  targetLanguage: 'es',
});

await translator.translate('I am tall.', {
  context: 'I am a woman',
});
// 'Soy alta'

[Xowap]

That's definitely a nice step forward!

On the other hand, I've made a mistake in my PoC, given that half the conversation is in the wrong language. If you mix it up it doesn't work anymore. So I guess you need first to translate back the context. Not unachievable at all 🤓

Another thing is: while I think it would be a folly to have a definitive model, maybe there is a way for the implementer to provide hints on how to deal with common patterns. For example:

await translator.translate('I am tall.', {
    context: await translator.buildContext({
        speakerGender: 'female',
    }),
});

Two rationales behind this idea:

• Allow the browser implementer to generate different system prompts depending on the nature of the model (let's say Chrome Mobile gives you the current model and Chrome Desktop with a GPU gives you a LLM, etc)
• Also in the current case it means that I need to detect the speaker's language, write "I am a XXX" in said language and then generate the translation, so that's a lot of annoying steps. It would be much easier if the API did it for me

[Xowap]

So I whipped this PoC:

interface TranslatorContextBuildRequest {
    /**
     * Indicates the gender to use for the first person messages. Sorry for
     * people who don't identify to these genders, I'm not sure at this point
     * what can be done at a satisfactory level outside of those two genders.
     */
    firstPersonGender: "male" | "female";
}

interface Translator2TranslateOptions extends TranslatorTranslateOptions {
    /**
     * A textual context for this translation
     */
    context?: string;
}

const SENTINEL = "✂️";

export class Translator2 extends Translator {
    static override async create(
        options: TranslatorCreateOptions,
    ): Promise<Translator2> {
        const translator = (await super.create(
            options,
        )) as unknown as Translator2;
        Object.setPrototypeOf(translator, Translator2.prototype);
        return translator;
    }

    public async buildContext(request: TranslatorContextBuildRequest) {
        let genderSentence =
            request.firstPersonGender === "male"
                ? "I am a man."
                : "I am a woman.";

        if (this.sourceLanguage !== "en") {
            const translator = await Translator.create({
                sourceLanguage: "en",
                targetLanguage: this.sourceLanguage,
            });
            return await translator.translate(genderSentence);
        } else {
            return genderSentence;
        }
    }

    public async translate(
        input: string,
        options?: Translator2TranslateOptions,
    ): Promise<string> {
        let realInput = input;

        if (options?.context) {
            realInput = `${options.context}${SENTINEL}${realInput}`;
        }

        let result = await super.translate(realInput, options);

        if (options?.context) {
            const sentinel = result.indexOf(SENTINEL);

            if (sentinel === -1) {
                console.warn("Sentinel not found in translation result");
                return result;
            }

            return result.slice(sentinel + SENTINEL.length);
        } else {
            return result;
        }
    }
}

It works significantly less well than in Google Translate, also it was fairly difficult to get the sentinel to stay there and intact (for example it's eating up \n characters). It does work however. I'm gonna guess that the model could be tweaked relatively easily 🤓