make the test-suite more linked data friendly

[pchampin]

Summary: I argue that the IRIs of the manifests and tests for RDF 1.2 should be changed.

The problem

Consider the RDF 1.1 test suite: the IRI of each manifest is the URL of the manifest.ttl file (example); each test has an IRI of the form manifest.ttl#test-id (example).

• PROS: it closely follows the LD principles for machines, but...
• CONS: not so much for people ("3. When someone looks up a URI, provide useful information"), because the HTML description of each test is at a different URL (example). This could be solved with content-negotiation and/or some form of redirection, but sadly github-pages support neither.

The RDF 1.2 test suite made a different choice: each manifests is still described in a file manifest.ttl, but the IRI of the manifest itself is a fragment of the corresponding HTML page (example: in rdf12/rdf-n-triples/manifest.ttl, the IRI of the manifest is rdf12/rdf-n-triples#manifest). The same goes for the IRI of each test (example).

Clearly, the rationale of these choices was to mitigate the CONS above, and ensure that the IRI used by machines to identify a test provide useful information to people when dereferenced. The problem is that, conversely, we broke the 3rd principles for machines... Given that test suites are designed to help with test automation, I would argue that this is a net loss.

Breaking the 3rd principle is not just a philosophical issue: it has immediate practical consequences. In rdf12/rdf-n-triples/manifest.ttl, it says that the manifest mf:includes rdf12/rdf-n-triples/syntax/manifest.ttl. This is semantically incorrect, because that IRI does not identity a manifest (only a turtle file that happens to describe a manifest). This is also practically broken, because when I dereference it, I still need to "guess" the actual IRI of the manifest to get its description. Fortunately, in this case, there is only one manifest described in this file, so a simple heuristics gets me there, but that's brittle and ad-hoc. (Oh, and since RDF 1.2 manifests point to RDF 1.1 manifests, I also need to support both the "proper" LD way and this ad-hoc way...).

Proposed solutions

One option would be to revert to the old naming policy (that of the RDF 1.1 test suite). Not ideal, but better IMO.

Another option would be to use a service that supports content-negociation and redirection (w3.org? w3id.org?), and mint IRIs in this service for manifests and tests of the RDF 1.2 test suite. Those IRIs would redirect to the appropriate URL on github pages, based on the required content-type. E.g.

w3id.org/rdf-test/rdf/rdf12/n-triples →
    IF accept: text/turtle → REDIRECT to w3c.github.com/rdf-tests/rdf/rdf12/n-triples/manifest.ttl
    IF accept: text/html   → REDIRECT to w3c.github.com/rdf-tests/rdf/rdf12/n-triples/index.ttl

The IRI of a manifest would then be w3id.org/rdf-test/rdf/rdf12/n-triples#manifest,
that of a test would be w3id.org/rdf-test/rdf/rdf12/n-triples#test-id,

(again, I'm not attached to w3id.org, I'm happy to use another domain, including w3.org)

[rubensworks]

For reference, this is related to the discussion that took place here: w3c/rdf-star#269

In general, I agree that making the test suite more LD friendly would be a good thing. That would allow me to remove some of the ugly workarounds that I use in my own runner at the moment.

I do know that many people do not use this test suite through HTTP, but by directly cloning the repo and parsing the files directly. So any changes should also take that into account. (the proposed redirect approach might therefore be tricky)

It would also be good if we could make these changes in a backwards-compatible manner (but this may be difficult).
Since a recent change did result in breakage of existing runners.

[afs]

The tests are a long-term asset. The primary form are the files in this CG repo. They are machine readable data.

There is another constituency not mentioned above which is contributors. The CG formed so the tests could be maintained over the long term; that means hoping to receive PRs based on external local development. Hopefully, a cloned repo but if someone sends a patch that would be acceptable.

The CG happens to also produce github pages.

Another option would be to use a service

There have been other sites for HTML pages in the past but the CG only has github pages; we have to live with the limitations in order to get the ability to keep it up-to-date. For w3.org, security issues have been a limitation to maintenance.

I preferred the <> way but mf:assumedBase has been added to smooth the different aspects of HTML generation, file-based use and also making the machine-readable files less syntax dependent (TTL and JSON-LD).

We are limited at the moment by not having much capability to control the generation of github pages.

FWIW:
Apache Jena ships a copy of the tests in this repo in a release. A release is source code and is buildable downstream where there are a variety of environments, not all having reliable external access. The machine readable files and file layout are important to Jena.

[afs]

The same goes for the IRI of each test (example).

.../c14n/index.html#comment_following_triple

How did you get here because I get to .../c14n/#comment_following_triple.

[pchampin]

How did you get here because I get to .../c14n/#comment_following_triple.

I don't know :) I fixed the link in my original post.

[pchampin]

@rubensworks

For reference, this is related to the discussion that took place here: w3c/rdf-star#269

Wow, I completely forgot about this! So yes, same arguments, 3 years earlier...

I do know that many people do not use this test suite through HTTP, but by directly cloning the repo and parsing the files directly. So any changes should also take that into account. (the proposed redirect approach might therefore be tricky)

I am one of those people, so I sympathize with this concern :) Yes, the redirect approach would require some ad-hoc workarounds for local-use, but currently both local-use and web-use require them.

@afs

The tests are a long-term asset. The primary form are the files in this CG repo.

I assume that by that, you mean "the files in the repo rather than the resources published on w3c.github.io". I didn't consider that perspective, but that's a fair point. However...

They are machine readable data.

... as I point out above, the 1.1 test suite did a good job at being machine readable, both from the web and for local use. The 1.2 test suite sacrifices this to make the IRIs more human-friendly.

I preferred the <> way but mf:assumedBase has been added to smooth the different aspects of HTML generation, file-based use and also making the machine-readable files less syntax dependent (TTL and JSON-LD).

I must admit that I didn't even notice the mf:assumedTestBase property, and don't see how to use it (and it is not described in the mf: namespace document). We should document this. More generally, whether or not we change the IRIs of the tests/manifests to make them more linked data friendly, we should document how to consume the test suite (either from the web or from a local copy); all the more if we depart from Linked Data principles.

[afs]

How did you get here because I get to .../c14n/#comment_following_triple.

I don't know :) I fixed the link in my original post.

and the /# vs # is fixable in the production of the pages.

Things work via the browser ATM because the server is using 301 on e,.g, syntax# -> syntax/ but I don't know if that the the fixed github handling.

Changing trs: to end in / may be more stable.

[afs]

but currently both local-use and web-use require them.

Could you expand on that because I run the tests unmodified from both a clone of the repo (test development for contribution) and a file-tree copy (external system testing) -- it needs to be mf:assumedTestBase sensitive to fix action and results data (it isn't possible to write all tests with relative URIs) but nothing else.

This is true for RDF 1.1 except the base assumption was written into the test suite description. mf:assumedTestBase makes it part of the manifest.

[afs]

To add to the list of requirements, at least as a nicety, the URIs for tests that get into EALR reporting should be consistent. (IIRC that's a problem with a <> approach because if run as a copy, the URIs are wrong.)

The RDF and SPARQL working group is moving towards CR.
We need there to be a stable test suite soon.

Do we have software to process EARL reports?

[pchampin]

Ok, let me elicit my own workflow and assumptions:

My test harness works from a local copy, because it is faster and works offline. However, I treat the directory as a cache of the web version. More specifically, the test harness uses a document loader (similar to the JSON-LD's) to fetch any IRI, and I configure the document loader to retrieve all URLs starting with with https://w3c.github.io/rdf-tests/ from file:///wherever/my/local/copy/is/. But all other components (including the parsers) still sees them as coming from the HTTP URL, so the data I work with is exactly as if I was working from the web, and the EARL report uses the correct IRIs for tests.

I like this approach because it works equally well locally and remotely (you only need to switch the document loader to a naive one), and ensures separation of concerns. It has worked smoothly with the 1.1. test suite.

To be fair, this approach does not work as well when the server is being "smart". Some of this smartness can be embedded in the document loader without too much trouble (e.g. serve index.html when a directory is required, perform content-negotiation when the extension is omitted). Others, such as arbitrary redirections, are more tricky and would require more ad-hoc configuration of the document loader. So my proposal above is indeed not ideal in this respect.

I realize that I'm actually after two things:

1. we should document the best way(s) to process the manifests, i.e. ways that makes the developers' lives easier, and help them produce EARL reports as expected (in particular, reporting the "canonical" IRIs for tests). Ideally, this processing should be as straightforward as possible (i.e. avoid as much as possible ad-hoc workarounds).

2. we should ideally follow Linked Data principles in the data we publish on the web (if only to eat our own dogfood)

My initial gut feeling was that following LD principles (2) would help with (1), but this discussions has highlighted that the two may be in tension (notably because of the limitations of github pages, but not only).

If we have to prioritize, we should prioritize 1.

[pchampin]

The RDF and SPARQL working group is moving towards CR.
We need there to be a stable test suite soon.

Formally, no. What we need are implementation reports, and the test suite is merely a mean to that end.

More precisely: the test suite is a tool to help implementers assess their compliance to the spec (and help us decide whether we trust that assessment).

Granted, we don't want to pull the rug from under implementers' feet, especially not during the CR period. So breaking changes would be a very bad idea... (and arguably, my proposal above is a breaking change, I realize that). But that does not mean that the test suite has to be entirely frozen when we enter CR.

[afs]

Formally, no. What we need are implementation reports, and the test suite is merely a mean to that end.

SPARQL 1.0 and 1.1 has used EARL reports.
I am strongly against throwing that away.

This CG formed to maintain the RDF and SPARQL tests so they didn't fade away after a WG has ended.

[pchampin]

we are in violent agreement here @afs (on both the value of EARL reports and the value of maintaining the test suite in the long run).

I'm not suggesting that we drop EARL reports, just pointing out that they are not formally part on what the W3C process requires from the WG, so we have some leaway on how we manage the test suite.

edited "what the W3C requires" → "what the W3C process requires"

[afs]

No - we're not in agreement. This part of W3C has been using test suites.

RDF and SPARQL 1.2 tests have been accumulating for several years.
RDF and SPARQL 1.0, 1.1 test have been corrected (errata) and added to.
RDF 1.0 had a culture of tests for RDF/XML.

I believe that W3C ought to support that, not portray it as an optional extra.

It is sad that systems claim compliance but do not provide compliance reports. Implementation reports should have evidence. Evidence is part of data on the web.

[afs]

https://raw.githubusercontent.com/w3c/rdf-tests/refs/heads/main/ is useful to get files.

[pchampin]

No - we're not in agreement.

At least we are disagreeing on being in agreement :-)

More seriously, I agree with mostly everything you write above. I'm sorry if anything I wrote suggested otherwise (I realized that forgetting the word "process" above made it sound like it 😦 ).

Going back to the root of the argument:

@afs: We need there to be a stable test suite soon [for moving to CR]."
@pchampin: Formally, no. What we need are implementation reports, and the test suite is merely a mean to that end.

I should have been more nuanced, because obviously, we need the test suite to be stable enough that implementers can rely on it during the CR period.

What I meant is that the W3C process requires us to demonstrate implementation experience, but does not tell us how to do it. It is up to each WG to decide how, and yes, this WG has decided to use the test suite (and again, that's a decision that I fully support, and in line with the W3C culture, even though that's not prescribed by the process).

So the "stability" of the test suite is not as rigidly defined as the "stability" of the CR document itself. If we feel like, e.g. adding or changing a test during the CR period (e.g. because we find a bug in the test suite), that would not be breaching the process... (we would still have to decide what to do with EARL report pre-dating the change, if any, but again, that's the WG's prerogative).

To summarize:

• I agree that we need to make the test suite ready for the CR period (even though we can make some adjustments during the CR period).
• This proposal was in fact intended to make the test suite easier to use by implementers
• but, in retrospect, my first proposal above was probably too disruptive for existing test harnesses.
• I now believe that a clear documentation on how to consume the test suite is the way to go; once we have this written down, it will be easier to identify assumptions/design choices that could be improved -- and weight this against the pain for existing implementations to adapt.