Methods for modifying collections?

[mecworks]

I see a lot of methods for inspecting collections but none for modifying/adding elements, updating, etc.

For example, it would be good if there were a way to append an item.

is this just done using list.append()? how does validation occur in that case? I appended some data as an item, which happened successfully but then when I tried to do an collection.item.to_dict(), I got an error, obviously, because it was malformed data.

[ricardokirkner]

Sorry for the late answer. You are right, the current code is mostly about parsing a C+J document, not so much about modifying it. Nevertheless, it should be possible to append an item to a collection.

If you're just appending a plain dict, it's up to you to format it properly (until the code provides a helper method for this). Otherwise you could construct an Item object, with it's corresponding data payload, which should result a correct representation, like

c = Collection('href')
i = Item(data=[Data(name='foo', value='bar')])
c.items.append(i)

[mecworks]

Actually, the day I posted the question I figured it out. That is exactly how I ended up doing it. It works perfectly. Here's a small section of code where I'm appending items to make a collection:

# Devices
    def devices_c(self, ver=None):
        """
        A collection of device types
        :return: collection
        """
        self.version_check(ver)
        device_items = []
        for device_type in self.device_types:
            device_name = collection_json.Data(name='version', prompt='%s' % device_type)
            device_items.append(collection_json.Item(href='%s/api/%s/devices/%s/' % (self.href, ver, device_type.lower()), data=[device_name]))
        collection = collection_json.Collection(href='%s/api/%s/devices/' % (self.href, ver), items=device_items, version=ver)
        return collection

[mecworks]

I think the biggest thing that needs to happen is that examples be added to the documentation. Most the tools are already there.

[ricardokirkner]

Marc,

thanks for the example. I'm glad this is working out for you and you didn't drop it entirely because of this 'limitation'. To be honest, I'm not doing much with this library at the moment because I don't have a real use case for it, but feel free to file feature requests, and I'll try to spend some time now and then improving it. Of course, pull requests are also very welcome!

[mecworks]

Ricardo,

With the latest change in 0.1.1, my method for building building items by using list's .append() is not working anymore. I'm not sure exactly how to go about creating Data, Item and other objects as I have done before and there are no examples in the documentation of building collections.

Could you enlighten me on how I would go about doing what I was doing in version 0.1.0 in my code snippet above, in version 0.1.1?

Thanks for the help. Maybe some examples in the documentation of creating, Data, Item, Link objects and then creating collections using those would be helpful.

[ricardokirkner]

I Marc,

I'm really sorry about this. I'll look into the issue as soon as I can and get back to you with a proper solution (and more regression tests)

[dandesousa]

I saw this and noticed it was related to the changes I made recently. I can't seem to reproduce this. I'm willing to fix the issue and save you the trouble.

I tried the following:

>>> from collection_json import Collection, Item
>>> col1 = Collection("href")
>>> col1.items.append(Item("href"))
[<Item: href='href'>]

>>> col2 = Collection("href", items=[Item("href")])
>>> col2.items.append(Item("href2"))
>>> col2.items
[<Item: href='href'>, <Item: href='href2'>]

In both cases append seems to work. I'm on python 2.7 and 0.1.1 of the module. Could you include the exception or symptoms you see?

[mecworks]

Well, it looks like it's entirely my fault. In my code, I had actually modified/overwritten the collection_json Data class to add some extensions that I needed.

I added the following proposed extensions outlined by Mike Amundsen:

https://github.com/collection-json/extensions/blob/master/template-validation.md
https://gist.github.com/mamund/72328cc15082c533aea1

My code to overwrite Data:

import collection_json

class Data(collection_json.ComparableObject):
    """Object representing a Collection+JSON data object, extended with 'options', 'required' and 'regexp'"""

    def __init__(self, name, value=None, prompt=None, options=None, required=None, regexp=None):
        self.name = name
        self.value = value
        self.prompt = prompt
        self.options = options
        self.required = required
        self.regexp = regexp

    def __repr__(self):
        data = "name='%s'" % self.name
        if self.prompt is not None:
            data += " prompt='%s'" % self.prompt
        if self.options is not None:
            data += " options='%s'" % self.options
        if self.required is not None:
            data += " required='%s'" % self.required
        if self.regexp is not None:
            data += " regexp='%s'" % self.regexp
        return "<Data: %s>" % data

    def to_dict(self):
        """Return a dictionary representing a Data object."""
        output = {
            'name': self.name
        }
        if self.value is not None:
            output['value'] = self.value
        if self.prompt is not None:
            output['prompt'] = self.prompt
        if self.options is not None:
            output['options'] = self.options
        if self.required is not None:
            output['required'] = self.required
        if self.regexp is not None:
            output['regexp'] = self.regexp
        return output

collection_json.Data = Data

This modification worked fine with 0.1.0 but fails with 0.1.1. Here's the error:

/usr/bin/python2.7 /home/mchristensen/prog/embedded_pqa/rest_api/cj_collections.py
Traceback (most recent call last):
  File "/home/mchristensen/prog/embedded_pqa/rest_api/cj_collections.py", line 214, in <module>
    t = c.api_top()
  File "/home/mchristensen/prog/embedded_pqa/rest_api/cj_collections.py", line 93, in api_top
    api_items.append(collection_json.Item(href='%s/api/%s/' % (self.href, version), data=[api_name]))
  File "/usr/local/lib/python2.7/dist-packages/collection_json.py", line 332, in __init__
    self.data = data
  File "/usr/local/lib/python2.7/dist-packages/collection_json.py", line 33, in __set__
    instance.__dict__[self.name] = Array(self.cls, self.name, value)
  File "/usr/local/lib/python2.7/dist-packages/collection_json.py", line 245, in __init__
    super(Array, self).__init__(self._build_items(items))
  File "/usr/local/lib/python2.7/dist-packages/collection_json.py", line 256, in _build_items
    self.item_class.__name__, item))
ValueError: Invalid value for Data: <Data: name='version' prompt='Version 1.0a'>

Since I'm modifying the original code, I don't expect to have an official fix from you. Although, if you have any pointers, I'd appreciate it. Thanks for the great work. collection_json is very useful. Great work!

[dandesousa]

The change was subtle. I believe what happened is we added some additional conversions and our implementation binds the type that is checked at class definition time. Previously, you're monkey patch of the Data class overrode the type so that when we did the check it was fine, but now we actually check that it is in-fact an instance of the type 'Data' that is defined in the collection_json module and you cannot change this without monkey patching the actual Item class and every other class that uses it as well.

One possible solution I can think of is to inherit from the Data object and extend it. It would allow it to pass the check.

I also think your change and extension might be useful to others.

I'm not sure if it makes sense to include this in this library, as I'm not sure if Ricardo is interested in have this grow beyond the original specification.

I'm willing to try supporting some of these extensions on request, maybe in a different project, using this library as a base.

[mecworks]

I am thinking the same, I'd rather have the original code be to spec. However, what if there were a mechanism within Data (and possibly other classes) that allowed for extensions to be defined at class object instantiation time? I think that a way to extend the types by the user, while staying true to the core-spec in the code would be useful.

[dandesousa]

Having never used or read about any of the c+j extensions, I would like to look over some proposed examples and get a better understanding of the problem. What you suggest seems like a good idea as long as it covers most of the problem space for other extensions as well.

I'll try to take a look at it this weekend.

[mecworks]

Sounds good.

[mecworks]

Just an update, I changed the name of my class to MyData, inherited from collection_json.Data rather than from collection_json.ComparableObject and then called super(MyData, self).init(name, value, prompt) then monkey-patched collection_json and it all worked perfectly again. This can be a stop-gap for me until a better solution is coded up.

I was thinking, that we could code up some useful extensions in the classes and then enable them with conditionals based on flags that could be set within the instance or class variables that could be set.

Something like this:

class Data(ComparableObject):

    """Object representing a Collection+JSON data object."""

   ENABLE_OPTIONS_EXT = False
   ENABLE_REGEXP_EXT = False
   ENABLE_REQUIRED_EXT = False

    def __init__(self, name, value=None, prompt=None, options=None, required=None, regexp=None):
        self.name = name
        self.value = value
        self.prompt = prompt
        if Data.ENABLE_OPTIONS_EXT is True and options is not None:
            self.options = options
        if Data.ENABLE_REGEXP_EXT is True and regexp is not None:
           self.regexp = regexp
       # Handle all extensions in all methods (and other classes as needed) in a similar fashion.

Not sure if that satisfies the need to keep the default functionality to the spec while allowing a user to enable extensions at run-time. Extension support would have to be documented as experimental and fluid as they may never be accepted into the official spec.