Allow data as json

[jrief]

I use a REST service to serialize the cart. This REST service returns the carts content serialized in JSON format. Therefore I would have to convert it back to a Python dict, which doesn't make sense, since later it is has to be converted to JSON anyway.

Therefore I'd like to add a check, which bypasses the serialization, if the data object already is valid JSON.

[keis]

Thanks for the PR!

It makes sense to have an easy way of sending raw data using the library but I don't think the json check is the correct way. The way it's written will break on py3 as the input to json.loads must be str but the payload must be bytes . The logic for when to decode/encode on both py2/py3 will be a bit messy so I rather avoid that all together.

the easier isinstance(data, bytes) check (bytes being an alias for str in py2 IIRC) don't have that problem but would allow you to send broken json.

[jrief]

Python states EAFP:

"Easier to ask for forgiveness than permission".

Therefore it is safer, faster and cleaner having json.loads() to raise a TypeError if data is not a string, rather than testing for a string/bytearray.
Have a look how some internal Python functions are implemented, to see where EAFP is used. hasattr() is a good example.

[keis]

What we really need to check is that it's something that can transmitted through the urllib stuff which is limited to str/bytes because of the digest. Checking for valid json might be a nice addition but ultimately not needed imo

But the real problem is that json.loads will actually fail for the specific type we need it to be for py3 (bytes not str). I guess you could change 88 to be req.data = data.encode('utf-8') to get around that (and you do get some EAFP for a string type). But then we are keeping you away from sending already properly encoded strings without us first decode it only for the library to encode it again.

[jrief]

If json.loads() receives anything but a string/bytearray, it will raise a TypeError. Then the exception handler will convert data to JSON anyway - this is the default behavior. Therefore this implementation is safe for Py2/Py3.

[keis]

json.loads in py3 will not work with bytes or bytearray. it will work for str but that will later fail trying to pass it to hashlib as that expects bytes

[jrief]

json.loads is not intended to read bytes or bytearray, it will throw a TypeError and so behave as the current implementation. Therefore I don't understand the problem.

I'm happy with any kind of alternative implementation accepting a native JSON string. Please keep in mind, that my REST serializer stores its data as OrderedDict, thus keeping the order of dicts when serializing. This is the main reason, why I want to keep my existing JSON string. If I would do
json.loads(json.dumps(data)) I would lose that order.

[keis]

With this implementation is impossible to find a "jsonblob" object to pass that will actually work. give it a str and json.loads passes but you still get a TypeError trying to put a str into hashlib instead of bytes. use bytes and json.loads gives you the TypeError.

I get your use case and agree it would be good to be able to pass in a raw payload. I don't really mind either way if it's str or bytes that the proper type to give but at least one should work and something eq between 2 and 3 like "unicode or str"/str or "encoded str"/bytes.

What format are you keeping the REST response in?

[jrief]

Sure, I could do the conversion to JSON only if isinstance(data, (list, dict)) evaluates to true. Then all stream based formats would be passed through without attempting to convert them.

What format are you keeping the REST response in?

This is similar to a Python dict, but may include OrderedDict elements. Therefore the JSON encoder/decoder, as shipped with Django-RESTframework, serializes/deserializes that data in the correct order. Normal Python dicts get resorted in arbitrary order.

[keis]

Something like that could work but is perhaps an even worse violation of second guessing what the json module can encode/decode.

Perhaps it would be better to add a more explicit way of bypassing the encoding step say by a "body" option to complement the existing "data" option.

re the format I was more wondering if the value you are sending in is utf-8 encoded already or in "unicode"

[jrief]

a17dcc0 is another approach to solve this.

An alternative would be to allow to override the cls=MyJSONEncoder in json.dumps(data). Then I could use the JSONEncoder as offered by the REST framework.

[keis]

Will that really work?

>>> hasattr("foo", '__iter__')
True
>>> hasattr(b"foo", '__iter__')
True
>>> hasattr({}, '__iter__')
True

[jrief]

Ups,

only on

Python 2.7.6 (default, Nov 18 2013, 15:12:51)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> hasattr("foo", '__iter__')
False

but not on

Python 3.3.3 (default, Dec  2 2013, 01:40:21)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> hasattr("foo", '__iter__')
True

that's weird.

How about isinstance(data, (list, tuple, dict)) ?

[keis]

I think that would be fine