Sanitize strings by removing lone surrogates

[darrachequesne]

Following #68

Using a local utf8.js file, until mathiasbynens/utf8.js#19 is merged.

The strict: false flag make the encoder/decoder replace every lone surrogate with the U+FFFD character, instead of throwing Lone surrogate U+... is not a scalar value.

Note:

• the wtf-8 package is utf8 without the checkScalarValue, throwing the former error.
• Other errors (like Invalid continuation byte) can still happen when decoding, but are caught within the try..catch

cc @Nibbler999 @calzoneman

[calzoneman]

Is there a corresponding tryEncode(), or are we assuming that the server will not attempt to write invalid UTF-8 data?

[darrachequesne]

@calzoneman do you have a failing test-case?

EDIT: invalid utf8 is already covered, isn't it?

[calzoneman]

I don't; I just noticed that you mentioned that invalid continuation byte errors are handled by the try-catch, and I didn't see one for encoding. But if you've checked that that's already covered, that's fine.

[darrachequesne]

Yes, Invalid continuation byte errors are thrown in the decodeSymbol method, that is, when decoding from UTF-8 to UTF-16|UCS-2 (see).