Improve validation feedback?

[sdruskat]

Hi again,

This is something that all validation using generic JSON Schema validation packages have to deal with: the validation results may not be found very helpful in some cases, especially for users not experienced in schema validation. Improving the output is a lot of work and needs implementation of extra logic, but would help the user. Also, perhaps this could also be done in the upstream dependency itself. So this is more of a suggestion for discussion than a feature request.

E.g.,

• in this run, the error message is 1 data has additional properties (there is an extra key extra in the test file). It'd be great if in this case, the name of the addtl. property was provided.
• In this run, the error message is more helpful: 1 data["date-released"] referenced schema does not match. I think this is good enough. ✔️
• In this run, the error is 1 data.authors is required. Good enough ✔️

Additionally, I'm not sure that the field and message output are helpful/well-aligned with the output from the validation package:

------

                  field                          message
1 data["date-released"] referenced schema does not match

[dieghernan]

I would need to dig in further on this, as I am relying on jsonvalidate R package for validating and I don't have any prior experience on json validating.

A bit off-topic, but you mentioned on citation-file-format/citation-file-format#340:

I'll take it for a test drive with the key-complete CFF file we have for tests.

Is this one? If so, may I borrow it to complete my tests?

Thanks

[dieghernan]

Working on printing suggestion, now the error shows

Table: See errors

|field          |message          |
|:--------------|:----------------|
|data.authors.0 |no schemas match |

This is pure markdown, so it can be also copied and pasted like this:

Table: See errors

field	message
data.authors.0	no schemas match

[sdruskat]

I would need to dig in further on this, as I am relying on jsonvalidate R package for validating and I don't have any prior experience on json validating.

Yes, I think it's actually a bit of work, and may not always work as expected, as you'd have to catch the original error messages (which may also not be stable across versions of the validation package dependency), then process them and probably add heuristics to cover the CFF semantics.

A bit off-topic, but you mentioned on citation-file-format/citation-file-format#340:

I'll take it for a test drive with the key-complete CFF file we have for tests.

Is this one? If so, may I borrow it to complete my tests?

Yes, it's that one. Go ahead and use it for tests, absolutely.

[sdruskat]

Working on printing suggestion, now the error shows

Table: See errors

|field          |message          |
|:--------------|:----------------|
|data.authors.0 |no schemas match |

This is pure markdown, so it can be also copied and pasted like this:

Table: See errors
field message
data.authors.0 no schemas match

Nice one! That's definitely an improvement.

[dieghernan]

Just for info, what I get from this:

cff-validator/examples/key-error/CITATION.cff

Lines 1 to 19 in 90b5a70

	cff-version: 1.2.0
	message: "If you use this software in your research, please cite it as below."
	title: "cff-validator"
	abstract: "Validate your repository's CITATION.cff file using R software"
	authors:
	- family-names: "Hernangómez"
	given-names: "Diego"
	orcid: "https://orcid.org/ 0000-0001-8457-4658"
	error: yes # This key is an error on author
	- name: The Research Software project
	error_field: This is an error # This key is an error on the schema
	doi: https://doi.org/10.5281/zenodo.4442958 # doi with wrong format, shouldn’t be an url
	url: 123456 # url not valid
	license: abcdef # license not in schema
	keywords: # Error, person as keyword
	- family-names: "Hernangómez"
	given-names: "Diego"
	orcid: "https://orcid.org/ 0000-0001-8457-4658"
	error: yes

is:

Table: ./examples/key-error/CITATION.cff errors:

field	message
data	has additional properties
data.authors.0	no schemas match
data.doi	referenced schema does not match
data.keywords.0	is the wrong type
data.license	referenced schema does not match
data.url	referenced schema does not match

See Guide to Citation File Format schema version 1.2.0 for debugging.

File extracted from artifact in https://github.com/dieghernan/cff-validator/blob/main/examples/key-error/citation_cff_errors.md