Why aren't the inputs tokenized before used for calculation?

It seems that the implementation does not try to properly tokenize the inputs. Punctuations like question marks and commas are not removed, and the token cases are not lowered to maintain consistency. I am wondering, is this on purpose? I am not sure what is done for the official implementation but this tokenization strategy seems very off.