SentencePiece Tokenizer does not encode </s> as special token

[le1nux]

When tokenizing the </s> token, it does not get tokenized into a single special token, instead it gets split into multiple subtokens.

[le1nux]

cc @fromm-m @mali-git @flxst

[flxst]

I think this is to be expected, see

• SentencePieceProcessor.encode() breaks down special tokens that exist in vocabulary google/sentencepiece#777
• Unable to tokenize <s>, <pad>, </s>, and <unk> correctly in Python google/sentencepiece#667
• What is the difference between --user_defined_symbols and --control_symbols google/sentencepiece#215

In modalities, we put the tokenizer's EOD token between documents here.

[le1nux]

This was a good hint here:
google/sentencepiece#667 (comment)

I wonder if this creates issues for our instruction tuning, where we do play around with the special tokens.
What's your opinion on this @lllAlexanderlll ?

[lllAlexanderlll]

Yes, that is problematic, if we want to use the token - we could use any special token e.g. the "placeholder_tokens"
However, adding new tokens is not possible yet due to #208. Currently, a given single token is marked as special and hence tokenized prior to non-special tokens, which is needed to reliably not resolve prefixed tokens and special tokens into a single token.