jsx/jiffy compatibility

[talentdeficit]

hey. with maps we can sidestep the object representation problem. interested in aligning jiffy and jsx a little more closely? i've added support for uescape and return_maps and i think jiffy in force_utf8 mode does the same thing as jsx does by default (conversely, i think jsx in dirty_strings mode does what jiffy does by default). jsx is less strict than jiffy by default i think but we can probably work around that

ideally, we'd settle on a common interface such that we could both publish a shim json module that has an interface like:

%% jsx in regular operation, jiffy in `native` mode
json:decode(JSON, []) = json:decode(JSON, [native]).
json:encode(Term, []) = json:encode(Term, [native]).

we can already support uescape and pretty trivially. we can either force jsx into strict mode if we want to be pedantic or add support for various loose encoding to jiffy and then all we have to argue about is whether or not strings should be coerced into valid utf8 or not

also i don't know what you do about escaping forward slashes or u+2028 and u+2029 but those seem easy to work out

[davisp]

How apropos! I just today poked at jsx for an escriptized tool I was playing with. I was definitely confused for awhile trying to figure out the API cause I kept thinking you were rejecting some of my input terms. Eventually I figured out that I just had bad data though...

As for a shim project, I also just happened to run across this project today as well:

https://github.com/fishcakez/jixy

I didn't look too hard but it might be a good place to start. I'm not sure about us both providing a JSON module though. Seems like that'd be ripe for namespace clashing. People that ask for stream based processing of JSON I always send to JSX so its not inconceivable that people are using both applications in a single project.

For slashes, I used to escape / but then was talked out of doing that cause it made paths and URLs terrible for no real benefit. The JSON language lists the thing as escapable but doesn't mandate that its escaped (though it doesn't not mandate it either, another wonderful corner of the JSON "spec").

For U+2028 and U+2029 I seem to recall those being awkward cause they're referenced in the ECMAScript grammar or something and people freaked out about them. Jiffy doesn't do anything special for them currently but it might be something to add.

As to valid UTF-8 I'm a bit biased by doing database work where I hew towards rejecting anything that's invalid so I don't get in trouble for silently mutating data. I'd be open to considering a dirty_strings mode or something as an option I guess. There's already a force_utf8 encoding option so there's no reason to not have something on the decoding side.

The one big thing you left out is our error handling. We'd probably want to make sure that we combine that so that users wouldn't have to figure out different errors based on which library was selected and all that. If we did the third wrapper app we could port all our tests there and have them run against both libraries and what not. Hopefully running things through travis-ci would work when any three of the projects were updated.

And then there's the nuclear option where we just merge into a single repository. Its late here so I'm probably not thinking straight but it does seem like that'd handle the whole situation most cleanly. Except for the fact that we'd have to think of a name.

[talentdeficit]

maybe just one json module that has both jsx and jiffy as dependencies?

i'm mostly interested in there being a common interface that returns functionally the same results for encode and decode. according to that jixy wrapper we handle some characters differently so we'd have to resolve that. i have a pretty good set of tests for bad utf that i'll try to port to jiffy so we can try and figure out what we do differently. also we have different float precision? it's probably just a formatting thing with those

i have a decent set of tests for bad utf8 and for floats. i'll try to port them to jiffy today or tomorrow so we can figure out where we differ

also maybe we want to support encoding each other's object representations? i can support {[{...}]} pretty easily in jsx, not sure how much work it would take to support proplists in jiffy

[talentdeficit]

oh, as for name. i have this we can use. it's just some methods that work on/with json as returned by jsx right now, but if we unify interfaces it'll work just as easily on jiffy

https://github.com/talentdeficit/json

[talentdeficit]

or, thinking about it as i eat lunch, maybe we just want to restrict input to maps only for objects and if people want {[{...},..]} or [{...},..] they can use jiffy/jsx?

[davisp]

I agree that the json application would probably be the best place to add an interface.

For floats, I use Google's double-conversion library which does the pretty-printing of floats so that you get human readable 1.1 instead of 1.10000000000000008881784197001252. I pulled in one of the test files from double-conversion for that which we could reuse.

For unicode characters I'm guessing our defaults on replacements is probably the issue. We can open an issue on jixy to see if there was anything else but I'm fairly confident that Jiffy is quite strict here so it'd most likely just how we deal with broken UTF-8.

For the object representation I can spend some time trying to get that going. Its been a feature request to return proplist based JSON. Theoretically it'd be more difficult to add to Jiffy so I'll try and take a crack at it soonish to see if that'll be something that we can make Just Work.

One issue I see is that the map support might be a bit of a thing. Jiffy supports them but in a backwards compatible way. I still use R14B01 in production so at least at the Jiffy level I'd want to maintain that. Given the unification I'd like to have that compatibility but its not a requirement by any means. I'll try and find some time to go through the json app to get a feel for what that would entail.

The only other thing that comes to mind is more of a rebar/build thing in how we might want to allow users to specify which of jiffy or jsx they want. For us in development we'd probably want to have both so that when we run tests we know that we get identical behavior from both but users will most likely only want one or the other. I'm not entirely sure what the best solution even looks like here.

[tsloughter]

You should consider signatures for this https://github.com/erlware/erlware_commons/blob/master/doc/signatures.md

Only annoying thing is it really works best for data structures that you are already passing around as an argument -- like a dict. It may be awkward for this.