support for "map" typed runtime parameter

[keighrim]

New Feature Summary

We have seen a use case of passing a nested key-value pairs as a single (string) value to a runtime parameter. And then, we are currently working on a generalized stitcher implementation in the SDK (clamsproject/mmif-python#265), that will most likely to require some parameterized label "remapping" dict.

The string values for these parameters would then be internally parsed into hashmap/dict data structures to be used for document processing. So this issue is to discuss provisioning of SDK-level APIs to support such map-typed, string-valued runtime parameter with somewhat predictable common behaviors.

Related

No response

Alternatives

No response

Additional context

No response

[keighrim]

tl; dr

All we need is two special characters to build a notation for the standardized string syntax that can be parsed as a dict in python. However, our interfaces (URL or shell CLI) pose lots of constraints on choosing those two. My proposal is at the bottom, but very open to more suggestions. Especially I want to hear input from @owencking as he's our primary user.

---

Here's an early implementation (by @snewman-aa with some help from @kelleyl IIRC) without a standardized handling of such a parameter;

https://github.com/clamsproject/app-clipsearch/blob/8e7a0ffbf9a62725ca97ff8265ba9f578870c7f0/app.py#L84-L93

Specifically, the query parameter is used to pass a prompt string (for an LLM) and a short label string to save the results in the output MMIF. For example, a user can ask "images with one or two person closed up in the middle and have dense text overlay in the bottom third of the image" and annotate all the "found" images as a chyron-labeled TimePoint.

Notation-wise, this implementation is using@ to delimit the prompt string and the label string, then taking advantage of multivalued=True configuration of the parameter to allow two or more prompt-label pairs. Hence in practice, the URL for asking for chyron annotation (the example above) will be ;

$ curl --some --options \
    "http:localhost:xxxx?query=images%20with%20one%20or%20two%20person%20closed%20up%20in%20the%20middle%20and%20have%20dense%20text%20overlay%20in%20the%20bottom%20third%20of%20the%20image@chyron"

Or since the long string is a part of the URL query string, we can use + to escape spaces (for a little better readability, lucky us!).

$ curl --some --options \
 "http:localhost:xxxx?query=images+with+one+or+two+person+closed+up+in+the+middle+and+have+dense+text+overlay+in+the+bottom+third+of+the+image@chyron"

And finally, when a user wants multiple labeling at the same time, for example,

• "images with one or two person closed up in the middle and have dense text overlay in the bottom third of the image" -> chyron
• "images with total black of dark monotone background with two or more lines of horizontal text evenly distributed vertically " -> credits

$ curl --some --options \
 "http:localhost:xxxx?query=images+with+total+black+of+dark+monotone+background+with+two+or+more+lines+of+horizontal+text+evenly+distributed+vertically@credits&query=images+with+one+or+two+person+closed+up+in+the+middle+and+have+dense+text+overlay+in+the+bottom+third+of+the+image@chyron"

Some lessons from this early implementation;

• k-v pairs in a URL query string is delimited by & however, that character has a very specific meaning in bash (and many other unix shells). So we must assume that the whole URL string is quoted almost all the time (mostly with double quotes, to give some scripting freedom to users who are more fluent with shell scripting)
• spaces are not allowed in any part of URL. For "query string" part (between ? - #) we can use +, but it is still users' responsibility to construct and pass a proper URL string when they are sending requests.

---

So I did some research on possible options, and here is some additional information to consider

• we need two "special" character for
  1. delimiter between key and value
  2. delimiter between k-v pairs
• those selected two character must be prohibited from being used literally in any part of keys or values
• there are "unsafe" characters to use in URL and they must be escaped (to %hh form): https://www.rfc-editor.org/rfc/rfc1738.html#section-2.2

  # unsafe chars
  < >  # % { } | \ ^ ~ [ ] `
  

• query part of a URL is actually allowed with lots of special characters (but many of them are "unsafe" so needs to be escaped anyway): https://www.rfc-editor.org/rfc/rfc3986.html#section-3.4

  # (excerpt) CFG rules for query string
  sub-delims    = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "="
  unreserved    = ALPHA / DIGIT / "-" / "." / "_" / "~"
  pchar         = unreserved / pct-encoded / sub-delims / ":" / "@"
  query         = *( pchar / "/" / "?" )
  

• also in a recent conversation, we are tapping into a possible SDK-supported CLI for CLAMS apps (as an additional entry point along with HTTP), so now we need to consider character compatibility to shells. But unix shells have much less constraints in terms of allowed characters in argument values, and most of the time, proper (double-)quoting will solve even pretty tricky cases, instead of painfully escaping individual characters.

---

Given all that, my train of thought:

• This could be my total prejudice, but regular python programmers would want to use : as k-v delimiter, and , as delimiter between k-v pairs. Fortunately, both are NOT unsafe characters in URL(yay!). However, many SWT annotation labels already contain : character, so if we want to parameterize "post-binning" of the SWT app under this standardized map-type parameterization, we need to handle the "legacy" keys with colons. And then for , characters, I think they are very likely to be used in natural language-based keys or values. So I want to avoid using any common natural language punctuations.
• Using mutlivalued=True can be an elegant solution to eliminate the necessity for one delimiter character (k-v pair delimiter). Downside of it is that the user needs to repeat param= part in the query string. If the parameter name is long enough, that can be not only tedious annoyance, but also could pose some technical issues in terms of string length. But I still think the benefit outweighs, considering all the (could be totally imaginary) confusions around the concept of properly constructing the parameter value before sending a request. I think it's easier to make users understand by saying that "if you have two or more k-v pairs to pass, call this parameter multiple times" instead of "if you have two or more k-v pairs to pass, use this particular character to "join" all of them to create well-formed URL to send a request".

So finally, I'm proposing

1. use : to delimit between a key and a value:
   1. : is not a unsafe URL character, and has no special meaning other than the hostname:port delimiter
   2. : is so universally perceived as k-v delimiter in the primary languages we use (python and json)
   3. dealing legacy colons in SWT is an app specific issue
2. force multivalued=True for all type="map" parameters, and let users send multiple k-v pairs calling it repetitively.

And some fictional examples of SWT "postbin" parameterization under the proposal

1. as HTTP request

   # note colons in subtype labels are simply "collapsed" (S:H >> SH)
   $ clams source SOME_INPUT_SPEC | curl -d@- "http:localhost:5001?minFrameCount=5&labelbin=B:bars&labelbin=S:slated&labelbin=SH:slate&labelbin=SC:slate&labelbin=SD:slate&labelbin=SG:slate&labelbin=I:chyron&labelbin=N:chyron&labelbin=Y:chyron&labelbin=C:credit&labelbin=R:credits"
   # and of course, users aren't required to pass this long url all the time, since we will have reasonable default values for the `labelbin` param. 

2. as Python CLI

   # `labelbin` flag can take `?` number of arguments (https://docs.python.org/3/library/argparse.html#nargs)
   $ clams source SOME_INPUT_SPEC | python cli.py --minFrameCount 5 --labenbin S:slate SH:slate SC:slate SD:slate SG:slate I:chyron N:chyron Y:chyron C:credit R:credits

[owencking]

First, I agree 100% about using : to delimit keys and values. If that entails not supporting : in things like SWT labels, then we probably shouldn't include : within labels!

(In general, I favor restricting label names to concatenations of "unreserved" characters listed above (from RFC 3986, sec 2.3), with the possible exceptions of ~ and .. But that's a question for another time.)

As for the way to pass multiple parameters. Yes, I think doing it as multivalued makes sense. (If it were possible to pass a string that could be immediately parsed as JSON or a Python dict, then that would be nice. However, as a practical matter, we'll need to put quotes around the strings before we can use them. So, some pre-processing will be required.) The multivalued solution is elegant.

One additional wrinkle -- maybe a difficult one. We might need even more structure. For parameterizing binning, we might need to associate a key with a set of values, for example to express this kind of structure:

'post': {
  'bars': ['B'],
  'slate': ['S', 'S:H', 'S:C', 'S:D', 'S:G'],
  'opening': ['W', 'O', 'M'],
  'chyron': ['I', 'N', 'Y'],
  'credits':['C', 'R']   
}

I think , would be a very natural separator for multiple values. However, Keigh, you said

And then for , characters, I think they are very likely to be used in natural language-based keys or values. So I want to avoid using any common natural language punctuations.

Yes, I can imagine NLP apps where it'd be nice to use a comma within the value of some parameter. But (1) we are already giving up one common punctuation mark with :, and (2) maybe those sorts of natural language string value parameters make the most sense for top-level arguments, not within these structured "map" type arguments. So, maybe it would be okay to use , to delimit parts of these mappy things.

[keighrim]

Regarding the nested list parameters, for the time being I think confining all possible data types on the SDK level might not work on scale. We can still keep a minimum confinement of syntax and parser for string-to-string map within the SDK, and leave more complex data passing to app developers. For example, with the example above, the app developer can add additional parameter specification (well-documented in the app metadata ideally) that can handle comma separated strings for a list-type value. Concretely, users can use a URL like this

http://...?post=bars:B&post=slate:S,SH,SC,SD,SG&post=opening:W,O,M&post=chyron=I,N,Y&post=credits=C,R

and then the SDK will pass this dict to app code;

{ 
  "post": {
    "bars": "B",
    "slate": "S,SH,SC,SD,SG",
    "opening": "W,O,M",
    "chyron": "I,N,Y",
    "credits": "C,R"
  },
  ... # more params
}

Then app developer can add extra process to parse the string values into lists.

post_final = {}
for k, v in params['post']:
    post_final[k] = v.split(',')

In short, defining only string-to-string mapping at the SDK doesn't block app developers to implement their own more complex data types.

[afred]

@keighrim is it possible to have the CLAMS apps receive both runtime params and MMIF in the same object? Something like...

{
    "parameters": {
        "post": {
            "bars": ["B"],
            "slate": [ "S", "SH" ,"SC", "SD", "SG" ],
            "opening": [ "W", "O", "M" ],
            "chyron": [ "I", "N", "Y" ],
            "credits": ["C", "R"]
        },
        "speed": "88 mph",
        "gigawatts": 1.21,
        "fluxCapacitor": true
    },
    "MMIF" : {
        /* all the glorious MMIF */
    }
}

[keighrim]

We actually had something very similar with that for parameter passing method in our previous project (LAPPSgrid), but we changed the way in CLAMS since adding that "wrapper" json around MMIF will almost certainly requires users to use some additional piece of software to generate that non-MMIF json at the runtime.
For example, with the current (pure) MMIF-in, MMIF-out method, and given the apps are running as (micro) web services here and there, users can simply chain-call them to create a simple CLAMS pipeline

cat EXISTING_MMIF.mmif | curl -d@- -s http://app1:8080 | curl -d@- -s "http://localhost:8001?p1=v1" | curl -d@- -d "http://remote-app3:5555?param=value&more_param=more_value" > final_output.mmif 

But if the apps take a MMIF wrapper JSON, users might have to do something like this;

cat EXISTING_MMIF.mmif | wrap_mmif --flag-for-empty-param | curl -d@- -s http://app1:8080 | wrap_mmif --p1-flag v1-value | curl -d@- -s http://localhost:8001 | wrap_mmif --flags for --many params | curl -d@- -d http://remote-app3:5555 > final_output.mmif 

And in the LAPPS project, one of "utilities" that our team provided for users was the wrapper tool, but we realized it added not only a fair amount of maintenance cost for us, but also additional discipline and confusion to users.
In fact, the utility for that "wrapping" weren't even a single CLI app but a stack of java APIs, groovy-based DSL, and precompiled interpreter binary, so users had to write actual a code file instead of a shell script of chained curls, to allow direct encoding of complex, fully json-compatible data types (also all LAPPS webservices weren't pure HTTP, but all SOAP).

---

That being said, we can maybe provide an additional route option to users for that wrapped non-MMIF input. That would requires

• syntax and semantic specification for that wrapper part
• data presentation specs in the parameters
  • currently, CLAMS apps are written and documented under the assumption that users will only use simple strings to pass these parameters. How parameters are interpreted in internal data structures are somewhat abstracted and happening under the hood.
• integration (or just compatibility) with CLI-based entry points (additional CLI wrapper (on top of HTTP/flask app) #198 )

And I believe that should be a separate issue for future development.

[keighrim]

I created a new issue on the topic of passing parameters as a JSON object (instead of string maps), as it should be discussed in a broader context, not just for map-typed params.