Idea for RFC to represent task-task dependencies, between individual tasks of steps

[mwiebe]

Render farms often support dependencies between two steps that divide per frame instead of being across all the the frames together. Open Job Description could support this by extending the representation of a step dependency. This idea requires #79 to be able to express many patterns by writing expressions.

Here are a few examples to try showing how a general syntax could work for a variety of different cases.

Example: scene export -> frame render.

Suppose you have a scene in Windows-only software, that supports export to a renderer that runs on Linux. You can structure this as a two-step job. Generally, when a frame is finished exporting, it should render immediately, not wait until all the frames are exported.

...
parameterDefinitions:
  - name: Frames
    type: STRING
    default: 1-10

steps:
- name: ExportPerFrame
  parameterSpace:
    taskParameterDefinitions:
      - name: Frame
        type: INT
        range: "{{Param.Frames}}"
  script:
    ...
- name: RenderFrame
  dependencies:
    - dependsOn: ExportPerFrame
      # A task-task dependency specifies task parameters from the ExportPerFrame parameter space
      dependsOnSubspace:
        - name: Frame
          # Each Frame in RenderFrame depends on the same Frame in ExportPerFrame
          range: "{{Task.Param.Frame}}"
  parameterSpace:
    taskParameterDefinitions:
      - name: Frame
        type: INT
        range: "{{Param.Frames}}"
  script:
    ...

Example: simulation with uniform time steps -> render every Nth as an image

Suppose you have a simulation of a world defining its physics and perhaps agents within it. For each timestep, you take the state of the world at the previous timestep, and advance it. You can represent this as a chain of tasks, each one depending on the previous one. You could compose this with a per-timestep dependency

...
parameterDefinitions:
- name: FrameCount
  type: INT
  default: 100
  minValue: 1
- name: TimeStepsPerFrame
  type: INT
  default: 10
  minValue: 1

steps:
- name: SimulateTimeStep
  parameterSpace:
    taskParameterDefinitions:
      - name: TimeStepNumber
        type: INT
        # There are FrameCount * TimeStepsPerFrame total number of time steps
        range: "0-{{Param.FrameCount * TimeStepsPerFrame  - 1}}"
  dependencies:
  # This step depends on itself
  - dependsOn: SimulateStep
    dependsOnSubspace:
      - name: StepNumber
        # Each task depends on the task for the previous TimeStep. Out of bounds values are discarded, so
        # TimeStepNumber 0 has no dependency.
        range: ["{{Task.Param.TimeStepNumber - 1}}"]
  script:
    ...
- name: RenderFrame
  dependencies:
    - dependsOn: SimulateTimeStep
      dependsOnSubspace:
        - name: TimeStepNumber
          # Each Frame in RenderFrame depends on the time step corresponding to the frame
          range: ["{{Task.Param.Frame * Param.TimeStepsPerFrame}}"]
  parameterSpace:
    taskParameterDefinitions:
      - name: Frame
        type: INT
        range: "0-{{Param.FrameCount - 1}}"
  script:
    ...

Example: Directed acyclic graph (DAG) uniform processing of a dataset.

Suppose you want to process your data set in a DAG, running the same processing at each node of that graph. You can structure this as a single-step job, with a self-dependency that holds a graph as a dependency list. There might be a better way to represent this, but this example needs an ARRAY[ARRAY[INT]] (or LIST[LIST[INT]], however #79 ends up) job parameter data type to accept the adjacency list as a parameter.

...
parameterDefinitions:
- name: GraphAdjList
  type: LIST[LIST[INT]]
  # Default adjacency list for this graph:
  # N0 -> N1 -> N4
  #   \-> N3   \
  #     \--------> N2
  default: [[], [0], [0, 1], [0], [1]]

steps:
- name: ProcessGraphNode
  parameterSpace:
    taskParameterDefinitions:
      - name: NodeIndex
        type: INT
        # There's one task for each graph node
        range: "0-{{len(Param.GraphAdjList)}}"
  dependencies:
  # This step depends on itself
  - dependsOn: SimulateStep
    dependsOnSubspace:
      - name: ProcessGraphNode
        # Each graph node depends on the list of nodes in its adjacency list entry
        range: "{{Param.GraphAdjList[Task.Param.NodeIndex]}}"
  script:
    ...