Incorrect UTF-8 splitting causes visual glitches

[ctrlcctrlv]

Not all substrings of a valid UTF-8 string are themselves valid UTF-8 substrings, but slack++ treats them as such, leading to visual glitches and (possibly) #15.

Bug discovered in the course of debugging #14 - more information available there, as well as possible solutions.

Example:

(U+2014 displayed as ~@~T)

[FedeDP]

@ctrlcctrlv Can you please try now? For both #15 and #16.

[ctrlcctrlv]

Unfortunately - this issue is still alive and well. Now that #14 is fixed and I can scroll further back in my chat history, I can find even worse-looking examples of it.

This text should look like

You just summoned a Ruthless Angel. あなたは残酷な天使を召喚しました。
Your IP address is

I am a Ruthless Angel.
私は、残酷な天使。
ERROR! Your post is too excessive.
ｴﾗｰ! あなたの投稿は過剰すぎます。
I am regulated you.
私はあなたを規制しました。
So you will not be able to write for a while.
なので(従って)、あなたはしばらく書き込めません。

If you want to reproduce this, simply go to ja.wikipedia.org, copy some text and post it into a Slack room. Then look at it in slack++.

At my company, half of my coworkers are Japanese and we manage 2channel - so bad Unicode support is readily apparent in every channel. :(

[ctrlcctrlv]

We need to change the session interface to accept std::wstring instead of std::string, and use std::wstring everywhere until the very last step of inserting into the ncurses window. We will then convert it from wchar_t (std::wstring) to char* for ncurses....

[ctrlcctrlv]

https://stackoverflow.com/questions/402283/stdwstring-vs-stdstring

[ctrlcctrlv]

Is it OK for us to use iconv? It's a new dependency, but it's a very common one - most Linux/BSD distros should have it, if not as a header than at least as a shared object.

If we can use iconv I can probably fix this easily...but trying to do it with the standard libraries only is a pain and every solution I can find will break compilation on Windows.

[ctrlcctrlv]

Or how about this library? https://github.com/nemtrif/utfcpp

This would make it so we don't have to use a lot of C stuff which could give us more memory problems.

[ctrlcctrlv]

I have a patch for you that requires no external libraries other than C++14, but it has one problem I'm waiting for advice from SO on before opening a PR.

https://stackoverflow.com/questions/46871566/how-can-i-know-if-a-character-will-fit-into-the-grid-and-if-it-wont-how-many

[FedeDP]

You can use iconv if you wish; utfcpp seems too hard to find for some distro (i am on archlinux and it does not exist in both repo and aur).

By the way yeah std::wstring is surely the answer here. I was sure that std::string already handled wide chars and i did not even check on doc...
I don't understand your SO question: are you looking for a function that tells you how many chars is wide?
There is already a handy function: https://linux.die.net/man/3/wcsnlen .
You can use this function when splitting new messages/history messages, and bound them to number of cols available :)

[ctrlcctrlv]

If you're planning on a more thorough fix to also allow resizing the window (i.e. deprecating the SlackUI::add_message and SlackUI::add_history functions) I won't continue working on my commit which would just fix those functions.

My SO question was about how we can know how many columns a character takes. A CJK character always (to my knowledge) takes two columns, which means that even if we avoid the problem we have right now where we're splitting UTF-8 strings at the wrong place, we can still end up with strings that don't fit in the window because the characters are physically too "wide' to fit in one column.

This project right here is great and would be simple to translate into C++ : https://github.com/jquast/wcwidth/blob/master/wcwidth/wcwidth.py

The reason we don't want to use the system wcwidth is because it is often incorrect, either because it doesn't know about the latest Unicode standard or because of the user's locale.

[FedeDP]

I'll try later with wcsnlen (maybe on a new branch), then i'll wait for your verdict :)
If wcsnlen is enough to fix the issue, for now i'll merge it.
Then I/we (if you are still willing to help it is much appreciated!) will try to fix the root issue with a new Chat messages structure.

[ctrlcctrlv]

Alright 👍 I agree that we should have a new Chat messages structure, but I need more time to think about what that should look like :)