custom keys/parameters in a Yaml file

[tiltX]

Hi,
I'm currently looking into OpenJobDescription for the first time and I wasn't aware that it's much more than just a Yaml formatted file to store render jobs. This looks quite cool, especially the fact that it leaves out the actual scheduling on the render farm but provides all the scaffolding and specs.

My question is: How can I add custom keys or parameters to the job templates that might be beneficial for in-house use? Adding custom keys to the Yaml file or telling the openjd cli tool about a custom extension will cause an error. For example, let's assume I want to add an allow/deny list to a step definition (like what Deadline has). I think it could be done with HostRequirements but what if I wanted to add a "allowedOn" and "neverRenderOn" keys to a step.

I've looked at openjd.model.decode_job_template() which seems to be hard-coded to a certain version of the specification and 2 optional extensions that are part of the official specification (as of v2023_09). Would I have to derive my own specification classes from that spec? Add a new extension and modify the model classes to support it?

Standards like CSS rules or HTTP have mechanisms to gracefully ignore vendor-specific extensions, for example HTTP headers can be prefixed with "X-Vendor-". Is something like that planned for OpenJobDescription? Or is me thinking about this a sign that I'm not understanding what OpenJobDescription is supposed to achieve? :-)

[mwiebe]

Hi tiltX,

Thanks for trying out OpenJobDescription! What you're trying to do is exactly the kind of thing we want to happen - both to work out improved ways that anyone can customize things, and to get ideas like these into the main spec.

First, the change you're suggesting as an example sounds like a great adjustment to the main spec. Adding an allowlist and a denylist based on hostname and/or IP address sounds useful in general, so making an official way to state that in OpenJD seems like a great way to proceed. Our RFC README documents a process you can follow to contribute if you're interested. Since this change is pretty minimal, I would expect an RFC document to be pretty short and non-controversial. Have a peek at #95 for one we have in progress now.

Regarding different ways to extend OpenJD, you make a good point about graceful handling of vendor-specific extensions. RFC 0002 was the first RFC added by someone outside Amazon, and used Python's __future__, CloudFormation AWS::LanguageExtensions, and the OpenGL extension mechanism as inspiration. It didn't look at the idea of gracefully ignoring new properties, but that could be the subject of a new RFC. I suspect there are subtle things to consider - if you put an allowlist in a step's host requirements, is it fine for the scheduler to silently ignore that property and schedule it on all hosts? How do we apply these kinds of extensions and make the result as easy to understand and predictable as possible?

[tiltX]

Thanks for the pointers. As I've answered to @jusiskin below, I don't have an actual project at hand that needs OpenJobDescription. I've read up on it and thought about if I could use it to write a program that can manage our farm (currently using Deadline 10 and happy with it). An allow/deny list was one thing that came to mind but now I realize that there's no job priority in the spec either. And no metadata fields like: submitted when, by whom and from which machine.

These are a lot of things that might be very specific to a developer's needs and could clutter the spec. If I were to use the spec just as a guide for my own data structure I would forfeit the cool cli tools and the Python module that you made. So that's why I asked about vendor extensions and how I would add them without doing RFCs (I think most studios who want to use OpenJobDescription for in-house apps might need their own data in those yaml files in one way or another).

[jusiskin]

Hey @tiltX,

I'd also like to clarify your use-case a bit more.

To connect a few dots here, Deadline 10's machine limits feature (docs) allowed users to specify allow/deny-lists of workers for each job. By default, workers were identified based on their hostname.

If you are looking for exact feature-parity here, OpenJobDescription does not yet support any sort of deny-list and an RFC would be the path forward to including this in the spec.

The HostRequirements only allows specifying what host capabilities are required by the job and must be met. A render management system could use this feature to allow-list by hostname or IP address (up to a maximum of 50 currently). Would an allow-list of 50 or fewer hostnames/IPs meet your current use-case? If so, then the features of the current OpenJobDescription specification may be sufficient.

In addition, if you are able to allow-list based on a different attribute or property of the worker host (e.g. label/tag) then the existing HostRequirements specification could work.

[tiltX]

Thanks for pointing out the HostRequirements section. Yeah, that could probably be used as a allow/deny list. There is no concrete use-case yet, I was playing with the idea of a custom render scheduling system when I found out about OpenJobDescription (we're currently using Deadline 10 which works great for us at the moment).

[mwiebe]

You might also find the blog post How to add Open Job Description in your render pipeline helpful, it shows some details of a proof-of-concept we created running OpenJD jobs in a Deadline 10 farm.