Tokenizer evaluation

[joenaess]

Felix: I think it's definitely a good idea to decide on a preliminary vocab size and a reasonable vocab size range, but I would also like to vary the vocab size and investigate the effect on what is called internal and external performance below. Similar to what I have done in the past: [https://arxiv.org/abs/2304.14780]

[joenaess]

Feature Checks: A framework to verify that all the features we agree on are successfully implemented. In addition, the tests should also thoroughly check the tokenizer in applications, in order to avoid bad surprises later.
Internal Evaluation (Fast Feedback): We will measure fertility and compression ratio for a quick signal of tokenizer efficiency.
Python

Simplified fertility calculation logic

total_tokens += len(tokenizer.encode(text))
total_words += len(text.split())
fertility = total_tokens / total_words

External Evaluation (Ground Truth): For promising candidates, we will perform standardized, abbreviated model training runs. The definitive measure of a tokenizer's quality will be the downstream task accuracy (e.g., acc_norm) after training a model for a fixed number of steps.

[stefan-it]

Hi @joenaess maybe that's interesting for the project, but I recently found this framework which is implementing common metrics (also multilingual).

I have modified it so e.g. datasets can be read from the Model Hub.

[joenaess]

Excellent, thank you we will look into it. Would you also been interested in working on this @stefan-it ? @timpal0l can you have a look?

[stefan-it]

Hi @joenaess , I would be super interested to work on that, yes!

[joenaess]

Great @stefan-it. Create a branch and work on this in a subfolder or similar, implement, test etc and then make a PR please :-) Report progress here is great and MM thread in t4.3 channel for discussion.

[joenaess]

@stefan-it are you on Mattermost?

[stefan-it]

Hi @joenaess , sorry for the late reply, I missed the GitHub notification.

I am not on Mattermost (but I have e.g. Discord) and I am new to the OpenEuroLLM project. Is there a main discussion plattform where I could join?

[joenaess]

Yes. The project uses MM, so you need to get on there asap :-) I can put you in touch with @timpal0l on discord as well however... I am _jonas44 and Tim is timisbister. And which organisation do you belong to @stefan-it ?