Signposting by janvanmansum · Pull Request #7919 · IQSS/dataverse

janvanmansum · 2021-06-03T07:48:20Z

Context:
This Signposting feature was developed by Data Archiving and Networked Services (DANS-KNAW), the Netherlands. It is part of ODISSEI Portal work group and funded by ODISSEI, "Open Data Infrastructure for Social Science and Economic Innovations". ODISSEI has received funding from the Dutch Research Council (NWO).

What this PR does / why we need it:

This PR adds Signposting support to Dataverse 5.5.

Signposting is an approach to make the scholarly web more friendly to machines. It uses Typed Links as a means to clarify patterns that occur repeatedly in scholarly portals. For resources of any media type, these typed links are provided in HTTP Link headers. For HTML resources, they may additionally be provided in HTML link elements.

Which issue(s) this PR closes:
Closes #5962

Special notes for your reviewer:
TODO: when multi-license is merged, should adapt signposting as multi-license provides reliable way to fetch valid license URI which is not available in vanilla Dataverse yet.

Suggestions on how to test this:
cURL or Postman

To get level 1 profile

curl -I http://localhost:8080/dataset.xhtml\?persistentId\=doi:10.17026/dans-xdg-jtew

HTTP/1.1 200 OK
Server: Payara Server  5.2020.7 #badassfish
X-Powered-By: Servlet/4.0 JSP/2.3 (Payara Server  5.2020.7 #badassfish Java/Azul Systems, Inc./11)
Set-Cookie: JSESSIONID=b91e9e3a3cdbb7e583458a0b75d0; Path=/; HttpOnly
Link: 
<https://doi.org/10.17026/dans-xdg-jtew>;rel="cite-as", 
<https://doi.org/10.17026/dans-xdg-jtew>;rel="describedby";type="application/vnd.citationstyles.csl+json",
<http://localhost:8080/api/datasets/export?exporter=schema.org&persistentId=doi:10.17026/dans-xdg-jtew>;rel="describedby";type="application/json+ld", 
<https://schema.org/AboutPage>;rel="type",
<https://schema.org/Dataset>;rel="type", 
https://creativecommons.org/licenses/cc0/;rel="license", 
<http://localhost:8080/api/datasets/:persistentId/versions/1.0/linkset?persistentId=doi:10.17026/dans-xdg-jtew> ; rel="linkset";type="application/linkset+json"
Transfer-Encoding: chunked
Content-Type: text/html;charset=UTF-8
X-Frame-Options: SAMEORIGIN

To get level 2 linkset

Please note this line in level 1 profile, this give the link to linkset

<http://localhost:8080/api/datasets/:persistentId/versions/1.0/linkset?persistentId=doi:10.17026/dans-xdg-jtew> ; rel="linkset";type="application/linkset+json"

curl http://localhost:8080/api/datasets/:persistentId/versions/1.0/linkset\?persistentId\=doi:10.17026/dans-xdg-jtew

Here is a sample response in JSON

{
  "linkset": [
    {
      "anchor": "http://localhost:8080/dataset.xhtml?persistentId=doi:10.17026/dans-xdg-jtew",
      "cite-as": [
        {
          "href": "https://doi.org/10.17026/dans-xdg-jtew"
        }
      ],
      "type": [
        {
          "href": "https://schema.org/AboutPage"
        },
        {
          "href": "https://schema.org/Dataset"
        }
      ],
      "license": {
        "href": "https://creativecommons.org/licenses/cc0/"
      },
      "describedby": [
        {
          "href": "https://doi.org/10.17026/dans-xdg-jtew",
          "type": "application/vnd.citationstyles.csl+json"
        },
        {
          "href": "http://localhost:8080/api/datasets/export?exporter=schema.org&persistentId=doi:10.17026/dans-xdg-jtew",
          "type": "application/json+ld"
        }
      ],
      "item": []
    }
  ]
}

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

No

Is there a release notes update needed for this change?:

Additional documentation:

Please consult Signposting website

…ring)

…nposting

* uses external js * adding signposting configuration setting * signposting poc from eko * error fix * move the signposting location in code * Signposting with MicroSettings cleanup v1 xhtml revert to vanilla * clean up * bug fix and add default file type; TODO: Bug on the baseurl (empty string) * hide type when do not use default * set default when no config is set for signposting * modification according to reviews * move long json string from code to bunddle * allow empty config on the level 2 profile Co-authored-by: Kevin Condon <kcondon@hmdc.harvard.edu> Co-authored-by: ekoi <eko.indarto@dans.knaw.nl>

Signposting

pdurbin · 2021-06-09T14:33:12Z

there seem to be additional changed files that have nothing to do with Signposting

@janvanmansum this has been resolved by your recent commits. Thanks!

Some docs would be appreciated. Maybe a new page in the Admin Guide?

Also, if you could add a release note, that would be great. Here's the process: https://guides.dataverse.org/en/5.5/developers/making-releases.html#write-release-notes

Thanks!

coveralls · 2021-06-09T15:32:00Z

Coverage decreased (-0.06%) to 19.264% when pulling da8f52e on DANS-KNAW:signposting into b61b572 on IQSS:develop.

add release note and skip license link in case of None

qqmyers · 2021-07-23T20:51:53Z

doc/release-notes/5962-signposting.md

+ * The level 2 linkset can be fetched by visiting the dedicated linkset page for 
+   that artifact. The link can be seen in level 1 links with key name `rel="linkset"`.
+
+The configuration is stored as JSON string in the `Bundle.properties` file, key name is


I see from code below that this is just a default setting and that there is SignpostingConf setting as well. Minimally, the docs should indicate that it is set-able without editing the Bundle. More deeply - I'm not sure the Bundle is the right place for a default, i.e. it isn't something that should be internationalized. If it is easier to get a default for the whole setting, should it go in the code instead? Or?

qqmyers · 2021-07-23T21:00:13Z

src/main/java/edu/harvard/iq/dataverse/api/Datasets.java

+            if (signpostingConf.isEmpty()) return notFound("Configuration key for signposting is empty [SignpostingConf]");
+            if (dsv.getId() == null) return notFound("Dataset not found: Id is empty");


I'm wondering about the error responses - not sure I would want users to know whether the config key exists. One option would just be to send no message or "Signposting not enabled". I also wonder if 501/not implemented or 409/conflict be better for the first?
For the second - I assume the getDatasetVersionOrDie (findDatasetOrDie) above will return before you could hit this?

qqmyers · 2021-07-23T21:03:10Z

src/main/webapp/datasetFieldForEditFragment.xhtml

+    <p:watermark for="inputText" value="#{dsfv.datasetField.datasetFieldType.watermark}"></p:watermark>

-    <p:inputTextarea value="#{dsfv.valueForEdit}" id="description" pt:aria-required="#{dsf.required}"
+    <p:inputTextarea value="#{dsfv.valueForEdit}" id="description" tabindex="#{block.index+1}"


Are any of the differences in this file part of signposting?

qqmyers · 2021-07-23T21:09:41Z

src/main/java/edu/harvard/iq/dataverse/util/SignpostingResources.java

+        this.systemConfig = systemConfig;
+        this.workingDatasetVersion = workingDatasetVersion;
+        if (jsonSetting == null) {
+            jsonSetting = BundleUtil.getStringFromBundle("signposting.configuration.SignpostingConf");


related to my comment above - would it be simpler to just have a static string for the default in this class? Or to use defaults for the individual entries below (e,g, as useDefaultFileType defaults to true below - actually several fields get defaults that way already).

qqmyers · 2021-07-23T21:19:41Z

src/main/java/edu/harvard/iq/dataverse/util/SignpostingResources.java

+            ));
+
+            String authorURL = "";
+            authorURL = getAuthorUrl(da);


Note that metadata fields such as the components of the author need to be escaped/sanitized before they are used in html - the DatasetVersion.getJsonLd() method in v5.6+ is a good example of how to get safe values for individual fields and how to sanitize the entire response you're creating. Happy to discuss further.

qqmyers · 2021-07-23T21:22:18Z

src/main/java/edu/harvard/iq/dataverse/util/SignpostingResources.java

+
+    private String getItems(List<FileMetadata> fms) {
+        if (fms.size() > maxItems) {
+            logger.info(String.format("maxItem is %s and fms size is %s", maxItems, fms.size()));


drop to fine or remove before finalizing the PR...

qqmyers · 2021-07-23T21:31:57Z

src/main/java/edu/harvard/iq/dataverse/util/SignpostingResources.java

+            logger.warning(String.format("Error getting storageID from file; original error message is: %s", e.getLocalizedMessage()));
+        }
+
+        if (storageIO instanceof SwiftAccessIO) {


I have a few questions about this:

Given that files may be restricted, that an admin could move where files are stored to a different bucket or store type, etc., does it make sense to provide direct URLs to the storage?

Is Swift the only store that needs special processing? (S3, remote stores that are coming ~soon?)

If needed, should this be optional? (Personally, I think I'd just send the Dataverse filedownload URL and let it handle access control and potential temporary/permanent redirects when that URL is accessed.)

qqmyers · 2021-07-23T21:32:41Z

src/main/java/edu/harvard/iq/dataverse/util/SignpostingResources.java

+            String fileDownloadUrl;
+            SwiftAccessIO<DataFile> swiftIO = (SwiftAccessIO<DataFile>) storageIO;
+            try {
+                swiftIO.open();


Not completely sure about swift, but other stores, once you do an open(), you have streams that have to be closed to avoid resource leaks.

qqmyers · 2021-07-23T21:34:49Z

src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java

+     * @param sr corresponding SignpostingResources
+     * @return json linkset
+     */
+    public static JsonArrayBuilder jsonLinkset(SignpostingResources sr) {


Why is this needed? If you have an sr, can't you call sr.getJsonLinkset() yourself?

qqmyers · 2021-07-23T21:36:42Z

src/main/java/edu/harvard/iq/dataverse/DatasetPage.java

+     */
+    public String getSignpostingLinkHeader() {
+        if (!workingVersion.isReleased())
+            return "DRAFT";


why DRAFT? is this expected with SignPosting? Or should the header not be included if the dataset is draft?

qqmyers · 2021-07-23T21:40:47Z

src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java

+     * @param ds the designated Dataset
+     * @return json linkset
+     */
+    public static JsonObjectBuilder jsonLinkset(Dataset ds) {


Is this used? Also - since JsonPrinter is a singleton object, static doesn't change much.

qqmyers · 2021-07-23T21:41:15Z

src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java

+                .add("anchor", ds.getPersistentURL())
+                .add("cite-as", Json.createArrayBuilder().add(jsonObjectBuilder().add("href", ds.getPersistentURL())))
+                .add("type", Json.createArrayBuilder().add(jsonObjectBuilder().add("href", "https://schema.org/AboutPage")))
+                .add("author", ds.getPersistentURL())


author is the persistentUrl?

qqmyers · 2021-07-23T21:41:47Z

src/main/resources/META-INF/microprofile-config.properties

@@ -1,4 +1,5 @@
-dataverse.db.host=localhost
+dataverse.db.host=postgres


these aren't part of signposting!

qqmyers

I didn't attempt to run this or dig too much into the specific contents, but left a few comments that I hope are helpful. I'm not sure if the plan is to wait for multiple licenses to get merged before this goes forward or not, but thought I'd at least give a quick readthrough now.

qqmyers · 2021-07-23T21:50:03Z

src/main/java/edu/harvard/iq/dataverse/util/SignpostingResources.java

+    private String getAuthorUrl(DatasetAuthor da) {
+        String authorURL = "";
+        if (da.getIdValue() != null && !da.getIdValue().trim().isEmpty()) {
+            authorURL = da.getIdValue();


is this what you want? - any idValue gets sent back even if the getIdentifierAsUrl() would return a value? (Perhaps there shouldn't be an else in the next line?

qqmyers · 2022-02-02T16:15:07Z

@vicding-mi - FYI the multi-license PR has just been merged, so if that was a blocker here, it's gone. I haven't looked back through to see if the other comments have been addressed, but if you can make any updates for those and for multi-license that you think are needed, please do and then flag me and/or mark this for review so we can look at it again and move it towards being merged.

vicding-mi · 2022-02-08T08:00:28Z

Many thanks @qqmyers , I will recheck and proceed.

pdurbin · 2022-04-05T14:11:39Z

Closing in favor of this newer pull request:

Signposting on Dataverse 5.9 #8424

Vic Ding and others added 30 commits November 4, 2020 12:32

Merge remote-tracking branch 'upstream/develop' into develop

6a45404

Merge remote-tracking branch 'upstream/develop' into develop

770680e

cvm-sp

a38f0df

cvm-sp temp

c48df18

cvm-sp temp

2868228

cvm-sp temp

a55a378

tmp

db89d30

tmp

de12144

tmp

74dd56d

Merge remote-tracking branch 'upstream/develop' into develop

53f25b5

uses external js

673eef0

adding signposting configuration setting

6bc0f17

Merge remote-tracking branch 'upstream/develop' into develop

879f236

Merge remote-tracking branch 'upstream/develop' into develop

18ba37c

signposting poc from eko

3950734

error fix

5abe639

move the signposting location in code

3d644d1

Signposting with MicroSettings cleanup v1

65d69fe

Signposting with MicroSettings cleanup v1 xhtml revert to vanilla

7345f66

clean up

33fcb95

bug fix and add default file type; TODO: Bug on the baseurl (empty st…

a2e5b64

…ring)

hide type when do not use default

ddb8780

set default when no config is set for signposting

f2fb557

Merge branch 'signposting' of github.com:DANS-KNAW/dataverse into sig…

9de4c35

…nposting

modification according to reviews

1c6fde5

move long json string from code to bunddle

9752c6a

allow empty config on the level 2 profile

1c0cbbc

Merge branch 'develop' into signposting

0c40649

revision based on Herbert feedback

b441c65

janvanmansum and others added 3 commits June 9, 2021 15:41

Merge pull request #84 from vicding-mi/signposting

190a0f9

Signposting

Merge remote-tracking branch 'blessed/signposting' into signposting

bd07cb7

Removed duplicate method

5011558

Trigger build, to see if Travis succeeds

a656610

Remove unused function

ce893a4

jggautier mentioned this pull request Jun 24, 2021

Signposting should include license URIs made available in planned "multi-license" work #7969

Closed

Vic Ding and others added 2 commits July 1, 2021 10:13

add release note and skip license link in case of None

724e6e4

Merge pull request #98 from vicding-mi/signposting

da8f52e

add release note and skip license link in case of None

qqmyers reviewed Jul 23, 2021

View reviewed changes

qqmyers added the GDCC: DANS related to GDCC work for DANS label Sep 8, 2021

pdurbin closed this Apr 5, 2022

		if (signpostingConf.isEmpty()) return notFound("Configuration key for signposting is empty [SignpostingConf]");
		if (dsv.getId() == null) return notFound("Dataset not found: Id is empty");

		@@ -1,4 +1,5 @@
		dataverse.db.host=localhost
		dataverse.db.host=postgres

Conversation

janvanmansum commented Jun 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

To get level 1 profile

To get level 2 linkset

Uh oh!

pdurbin commented Jun 9, 2021

Uh oh!

coveralls commented Jun 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qqmyers Jul 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qqmyers left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qqmyers commented Feb 2, 2022

Uh oh!

vicding-mi commented Feb 8, 2022

Uh oh!

pdurbin commented Apr 5, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

janvanmansum commented Jun 3, 2021 •

edited

Loading

coveralls commented Jun 9, 2021 •

edited

Loading

qqmyers Jul 23, 2021 •

edited

Loading