BED-JSON transmittance as "number"

[christoph-maurer]

@danielmcquillen When I open for example https://igsdb-v2.herokuapp.com/api/v2/products/clearlite-4?format=bed-optical-json , then "transmittance" and "reflectance" are returned as strings instead of numbers.

      "dataPoints": [
        {
          "incidence": {
            "wavelengths": {
              "wavelength": 300
            },
            "direction": {
              "polar": 8
            }
          },
          "emergence": {
            "direction": {
              "polar": 8
            }
          },
          "results": {
            "transmittance": "0.0",
            "reflectance": "0.0476"
          }
        },

Is it easy for you to change it? I think they were numbers in the past.

[danielmcquillen]

@christoph-maurer The reason we store measurement data as strings in our json structure is we want to preserve the precision provided by the original submitter via their submission file.

As you probs already know, in JSON the best way to preserve precision is to use strings so the values are not read to doubles during parsing. Many consider this an anti-pattern, but until parsers change perhaps this is the best approach. End users can then cast to Decimal if they want exact precision or a double (IEEE754) if they want just good enough.

Happy to discuss more... @jcjonsson @simon-wacker thoughts?

[christoph-maurer]

@danielmcquillen Thank you for the explanation!

[simon-wacker]

@danielmcquillen In a perfect world JSON parsers would pass the ball to developers to decide how to parse numbers (as .NET does). When they are represented as strings without a proper JSON schema (probably some sophisticated regular expression), it is unknown which format they follow (for example, in German we use , instead of ., is scientific notation allowed or not, can I use _ for readability like 1_000_000, and so forth). When there is a proper JSON schema fixing the format, then I need a parser for that format into whatever kind of number I want, which if I'm lucky exists already in my language because the format is standardized and if I'm unlucky I need to write a custom parser for those number strings (and make sure it stays consistent with the JSON schema). So, even though I like the idea of not losing precision and I don't see another way in general than to use strings, I'd personally rather lose precision (in languages with bad JSON parsers) and use standard JSON numbers (this is of course but an opinion and people who really work with the data in real world and know about error propagation may strongly disagree). A more pragmatic issue with returning strings instead of numbers right now, is that the document does not conform to the JSON schema opticalData.json and other developers expect the document conform. So, we either need to update the JSON schema to allow strings here (with a fixed format) or make the JSON use plain-old JSON numbers.

[christoph-maurer]

@danielmcquillen I think that floating point values are very accurate for numbers smaller than 1 and have a considerable error for very large numbers, because the number of valid digits is limited. I don't think that it makes sense to develop a sophisticated regular expression and that each company develops their own parser. I propose that we share numbers as JSON numbers. I expect that more and more JSON parsers will offer options to parse numbers and that the error of values between 0 and 1 is very small.

[simon-wacker]

I turned the numbers into decimals as part of my refactorings of BED utility functions in https://github.com/LBNL-ETA/igsdb-webapp/pull/212/commits/48040bd5446b0c8109b3c20b1d3556b98b80b3c4#diff-e3ca2502842393a07e64aceef40414a9e068626d73383a7f0e2c5cfdadc3fd0cR374 (this can of course quickly be undone).

[danielmcquillen]

@christoph-maurer Discussed this issue with @RDmitchell and @jcjonsson who want to continue using strings for measured values in our own API, but are happy to have you update the BED serializer (as @simon-wacker appears to have done) so that the API does this change behind the scenes when returning data in BED format.

From an end-user's standpoint, I think this does not make the API less clear. It's reasonable to assume a certain format (BED) can effect changes on type of data returned.

DRF allows us to handle this pretty cleanly! Thanks @simon-wacker for anticipating this...