Service/Organization Relationship - Documentation Inconsistency

[aemerson1]

The HSDS documentation (and schema?) currently describe organization as a field (or child object) of service while the examples (such as organization_full.json) have services as an array of service objects in organization.

[MikeThacker1]

@aemerson1 I may not fully understand your point so @mrshll1001 may wish to step in, but my understanding from what you have said is that the parent:child relationship with one organization having many services is correctly represented.

In a relational database, I'd expect each service record to have a field populated with the service's organization identifier as a foreign key.

In JSON objects, I'd expect:

• service to have a (parent) organization object
• organization to have an array of service objects

[aemerson1]

@MikeThacker1
To clarify the issue, the relationship you describe doesn't seem reflected in the documentation.

Here we can see that in the organization object, there is no array of service objects. Furthermore,

We can see here in the service object that it has an organization field.

This part of the documentation led me to believe that service was a parent object of organization, not the other way around (though service does have service's organization identifier as a foreign key).

The inconsistency I noted in the issue is that while this relationship is represented as in the pictures above in the documentation (and the schema), it is represented as you described in the examples in the specification repo.

I hope this makes the issue clearer! Please let me know if there's anything more I can do!

[MikeThacker1]

Thanks @aemerson1 . I think this is one for @mrshll1001 then. :)

[mrshll1001]

Hi @aemerson1,

This is a good catch and thank you for bringing it up.

In short; the intended model is that the service schemas has a reference to an organization, while the the organization schema has an array of services (since an organization may deliver many services). This allows for different slicing of the data to support different use cases.

The specific reason for the discrepency in the docs is that the documentation is generated from the "compiled" versions of the schemas seen in schema/compiled, and the compiled version of the organization.json schema has a services array in it.

However, you are correct in that the un-compiled version of the organization.json schema does not have an array of services. I'm not sure why, and I believe this is an error or an oversight.

The schemas are re-compiled using the hsds_schema_tools each time we make a change to the schema, so I'm not sure how the organization.services array continually makes it into the compiled version despite being absent from the source schemas. I'll be investigating this as a matter of priority now.

For validation purposes, please use the compiled versions of the schema for now; as I think that's the intended use case of them.

[mrshll1001]

I've found the reason for the discrepancy.

In the compile schemas function in the hsds_schema_tools, when compiling the organization schema there is a line of code which explicitly adds in the array of services as a pre-compile step before compiling.

• hsds_schema_tools L510

Unfortunately, there's no documentation for this decision either in the source code or the commit logs, and the original developer has left the Open Referral Sphere.

Just going on instinct, I suspect that the reason for this is to enable support for generating the datapackage.json file from the original — uncompiled — schemas which possibly would break with the nested/circular relationship between organization and service. I'd need to test this though.

On the off chance that @kindly is still active on Github and can spare 5 mins, it'd be great to get some clarity on this!

[greggish]

Just a note that I still haven't found much demand for continuing to produce datapackages...

[kindly]

@aemerson1 @mrshll1001 @greggish

I think that you can add an array of services in the organization object, and that fact is missing in the documentation. The compiled schemas are supposed to contain the entirety of what is in the specification. The uncompiled schemas only show the schemas if you assume a service is the top level object. When service is the top level object, then you should not add the list of services to the organization. The compiled schemas were added to make sure when producing the organization s, the reverse relationship was also available.

Allowing a list of services against the company was requested when doing HSDS 3 as part of the consultation. It was there to help consumers of the services, more easily reconcile what changes to services there were. As instead of trying to work out changes for all services, they could do it an organization at a time. However, adding a list of services to an organization was considered very optional and was only really relevant to larger publishers, and it did not mean that you should omit putting the organization against the service (you should do both). Also, what data needed to be in that list of services was very optional too and just putting the services ids for most cases would be enough. This feature was also there to maintain compatibility to some extent with the datapackage as that allowed any kind of relation between the entities. Producing data in JSON limits the amount of relationships you can add, but this was a special case that was requested to be added. It therefore was only added at the compile stage, and it was my bad that this was not really documented or explained anywhere.

I hope that clears this up.