feat(RFC): An expression language for format strings

[mwiebe]

PR for RFC

Tracking Issue: #112

This is a request for comments about the proposed RFC 5, 6, and 7.

This is a set of 3 RFCs to propose an expression language to use in template format strings based on a subset of the Python language. The three proposals are:

RFC 5 - Expression Language
RFC 6 - Expression Function Library
RFC 7 - Extended Parameter Types

I have a fully-functioning prototype implementation across these branches:

• https://github.com/mwiebe/openjd-model-for-python/tree/expr
• https://github.com/mwiebe/openjd-sessions-for-python/tree/expr
• https://github.com/mwiebe/openjd-cli/tree/expr

This branch contains edits to show how things look when using the extension:

• https://github.com/mwiebe/deadline-cloud-samples/tree/expr

This branch in my fork of OpenCue is an experiment implementing a pyoutline backend for OpenJD:

• https://github.com/mwiebe/OpenCue/tree/openjd-prototype

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[crowecawcaw]

I'm curious how you'd compare the approach of adding new syntax to using external tools for features that OpenJD doesn't cover yet. A couple ways external tools could be integrated:

• Using a high level language like Python to generate the job templates, like what Deadline Cloud submitters often do
• Adding some sort of hook or preprocessor option to job templates that allows an arbitrary process / custom tool to handle expressions or logic

I'd think at a high level, the new syntax allows more expressive OpenJD templates to still be portable since we wouldn't need an external tool (like Python) to preprocess the job template. Also it'd let us build more features off of OpenJD like conditionally hiding job submitter UI elements. But the cost is adding complexity to OpenJD that alternative implementations need to support both for parsing job templates and for interacting with them (e.g. building an OpenJD-based submitter).

With agents getting better, the complexity cost is probably lower than it was a year ago. I'm also curious about your experience adding support to OpenCue.

The new expression syntax seems like the right approach to me. I'm asking because I'm sure you've thought about the tradeoffs and I'd like to understand them more.

[mwiebe]

I'm curious how you'd compare the approach of adding new syntax to using external tools for features that OpenJD doesn't cover yet. A couple ways external tools could be integrated:

• Using a high level language like Python to generate the job templates, like what Deadline Cloud submitters often do

When we first made the spec, we also started the pattern in Deadline Cloud submitters because we weren't able to fit in enough features to represent every variation the submitters need. That pattern uses code to generate a template that has mostly has a general template with parameters for the specific submission, but hardcodes some parts of it. The result is that the template isn't then a general reusable entity - it is just for the specific submission. When we can express everything in the template and have the submitter just set parameters, the result is available to read and re-apply to other scenes directly.

• Adding some sort of hook or preprocessor option to job templates that allows an arbitrary process / custom tool to handle expressions or logic

This approach adds constraints on the environment you're submitting a job template from, that an extension to the spec does not. Think of job submissions from the desktop vs from a browser, with and without a file system, etc. Expressions inside the template don't have to be understood at the submission context, but a preprocessor forces the submission context to have the ability to run the preprocessor commands.

I'd think at a high level, the new syntax allows more expressive OpenJD templates to still be portable since we wouldn't need an external tool (like Python) to preprocess the job template. Also it'd let us build more features off of OpenJD like conditionally hiding job submitter UI elements. But the cost is adding complexity to OpenJD that alternative implementations need to support both for parsing job templates and for interacting with them (e.g. building an OpenJD-based submitter).

Yes, I think this expression language hits the design tenets well.

1. Portable: the language is defined independent of the submission context or implementation details of a scheduler.
2. Expressive: It adopts familiar and powerful Python syntax and starts with a broad function library.
3. Human readable and writable: Python is familiar, and I've been experimenting with it myself to help validate this.
4. Tooling parseable: The openjd.expr library interface exposes the symbol references, called functions, and result type for a parsed expression, letting tools understand variable dependencies within the template and whether the template accesses path mapping.

With agents getting better, the complexity cost is probably lower than it was a year ago. I'm also curious about your experience adding support to OpenCue.

The OpenCue support I wrote is a very incomplete prototype - it is adding openjd support to the pyoutline library, which is for defining jobs to submit to OpenCue. I think I got the basic structure right, but it needs a lot of validation before it would be ready to use. The opposite, of running an OpenJD job in OpenCue, is not something I've looked at yet.

The new expression syntax seems like the right approach to me. I'm asking because I'm sure you've thought about the tradeoffs and I'd like to understand them more.

Thanks!