Support National Language Shift Tables

[juliangut]

Currently only supported Turkish, Spanish and Portuguese

Tests added only for charset validation. Message split should not be affected as National Language Shift Tables only affects charset and not number of bytes per char

[Codesleuth]

Awesome! Thanks @juliangut, please bear with me as I'm traveling and will have to review this when I get time.

[juliangut]

Hello @Codesleuth, have you had time to review this PR?

[Codesleuth]

I'm currently looking at it - will post soon.

[juliangut]

Hi @Codesleuth how did you find the PR? does it feel right?

[Codesleuth]

I'm going to hold off accepting this because of the reasons outlined in the comment. I'm thinking of an API in the future that will do something like this:

splitter.split('∞')

{
  "characterSet": "GSM",
  "shiftTable": "Portuguese language (Latin script)",
  "parts": [
    {
      "content": "∞",
      "length": 1,
      "bytes": 1
    }
  ],
  "bytes": 1,
  "length": 1,
  "remainingInPart": 159
}

And then in the case of mixed it would be:

splitter.split('∞Ø')

{
  "characterSet": "Unicode",
  "parts": [
    {
      "content": "∞Ø",
      "length": 2,
      "bytes": 4
    }
  ],
  "bytes": 4,
  "length": 2,
  "remainingInPart": 68
}

I think this will be a better API and won't give false positives for the mixed case being detected as GSM.

[Codesleuth]

I can see how this appears to make sense if all you need to do is count all possible valid characters that exist in any shift table, but this creates an invalid situation where the whole text doesn't fit any common shift table, and should be detected as Unicode. Take the following two messages:

1. ∞ valid in only Portuguese language (Latin script)
2. Ø valid in either Spanish language (Latin script) or the Basic Character Set

If each example was a full message, they would each be valid. However, if we take a message containing both characters:

∞Ø

As far as we know, this is an invalid message because there is no common shift table that supports both at the same time, so it should be detected as Unicode.

This library's character set auto-detection mechanism is pretty important, and we should think about how that can be preserved.

[juliangut]

I've updated the PR, when validating a whole message with this method is not used anymore, review message below

[juliangut]

Validating against each shift table char-table independently solves the problem of mixed shift table characters combined within the same message

[juliangut]

Hi @Codesleuth I can't argue against a new unified API, that would be great. About the messages with chars of mixed shitf tables you are right, I've just updated the code and tests accordingly, please review my comments

[Codesleuth]

I'm so sorry for the delay! I will be working on this repo this weekend, which will give me time to review and try out this PR properly. This PR is missing functional tests that cover the options and shift table output, but I can add them as I work on it.