ws: Could not decode a text frame as UTF-8

hi .. i will just report issue happens to me
i have chat website: nodejs +socket.io for server

some users send posts with emoji and some times result in disconnect all our clients because of:

socket.io-1.4.4.js:1 WebSocket connection to 'wss://***.com:1010/socket.io/?EIO=3&transport=websocket&sid=***' failed: Could not decode a text frame as UTF-8.
WS.doOpen @ socket.io-1.4.4.js:1

i did some debug in server side and print out escapped msg of the post who cause this
here are some of them

%u0627%u0646%0A.%uDFBB.%uDFBB

%u0645%u0627%u0644%u0643%20%u0644%u0632%u0648%u0645%20%u0628%u0641%u0631%u062D%u062A%u064A%20%u0627%u0630%u0627%20%u0628%u062D%u0632%u0646%u0646%u064A%20%u0645%u0627%u0644%u0642%u064A%u062A%u0643%uD83D%uDE34%uD83C%uDF38%0A

in short: just emit this from server to client connected with ws

// \uDF38
socket.emit('data',{msg:'\uDF38'});
// client side error: failed: Could not decode a text frame as UTF-8.

so my quickfix is this method [dont use this fix see my next post]

// server escape recived msg then broudcast it or store it
msg=msg.replace(/([\ud83c\udf00-\udfff])/gi,function(e){return escape(e)});

client side:

msg= unescape(msg);

why i dont just full escape the msg?
*that cost some bandwidth and increase latency if your clients didnt chat in english

any solution is welcome .. thanks

[carpii]

@dleeq8 I'm currently looking at a similar issue in my node.js based chatroom

I don't think your solution is that safe though.
It seems to allow users to type escaped characters in ASCII, and automatically have them unescaped when they are sent to other users (even if their message did not contain invalid utf).
This could cause all sorts of problems client side, especially with html entities or bypassing whitelists.

A better solution (and the one Im in the process of writing), would be to validate each incoming message.
If an incoming message is not valid UTF-8, I don't see a reason the even emit or broadcast it to other users.

Just disconnect the client who sent it.

@carpii
you are right in: users can bypass whitelist by sending escaped \u* or %u* and maybe lead to xss

you says: "Just disconnect the client who sent it."
let me just tell you how users cause utf8 problems here

users shares broadcast full of emoji and arabic at whatsapp groups/bbm then someone copypaste it to my chat and that cause utf8 problems by mistake

they are not hackers to be banned or kicked out ..

so what you think about this fix? escaping: "%", ""

replace(/([\ud83c\udf00-\udfff]|\%|\\)/gi,function(e){return escape(e);});

tested with: "%7D�" escaped to "%257D%uDF11" client unescape it "%7D�"
btw [&,<,>,",'] are not problems to me the problem is ws only ! xss is filtered somewhere else

thanks

[carpii]

Emoji and Arabic text should still be valid UTF-8 (Some of the more recent Emoji's use 4-byte characters, but are still valid UTF-8)

If you don't want to kick the user who sent them, an alternative might be to just silently ignore it (ie don't broadcast or emit it to other users)?
As soon as you try to convert invalid utf-8 into valid utf-8, you increase the risk of introducing a vulnerability

If you do want to validate it via regexp, its much more complicated than your example (since you need to check byte sequences, not just individual bytes)
Maybe this will help, although I have not tested it, and its based on an RFC from 2003...

http://stackoverflow.com/questions/13116685/regexp-to-check-if-code-contains-non-utf-8-characters

[darrachequesne]

To detect those lone surrogates, you can use the following regex:
var regex = /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?:[^\uD800-\uDBFF]|^)[\uDC00-\uDFFF]/;

Source: https://gist.github.com/mathiasbynens/bbe7f870208abcfec860

[darrachequesne]

That should be fixed in latest release (2.0.0).

Related: socketio/engine.io-parser#82