Design first version of framework

[knoepfel]

DUNE US S&C R&D item 062, 063, and 066

The framework design must include:

• Enumeration of core framework systems
• Definition of the interaction between core framework and IO backends
  The design sufficient for the F1 milestone is listed in the following comments:
  • Comment on terminology, preserving data, and providing data
  • Comment on API for data exchange
• The conceptual model of the framework workflow, which cover things like:
  1. What does the user specify? The data to be produced, the input data, and the algorithms?
  2. Form various textual descriptions proposing ways of doing this.
  3. What is the framework configuration schema? How does the user specify what work needs to be done?

The proposal for documenting the design (with @greenc-FNAL as documentation czar):

• The document must be version-controlled
• We should favor human-readable diagram specifications
• Let's use Sphinx for automatically-generated reference (API) documentation
• Let's use Sphinx to produce both user documentation and design documents
• Ideally, we'd use something that is amenable to an eventual publication-quality document

The purpose of the first design would be:

1. Place the products of the F1 milestones into a coherent whole: F1 provides for core functionalities that can be demonstrated in isolation. The initial framework design puts them together to define their boundaries and interdependencies.
2. The F2 milestones provide for a) concrete implementations of specific codes that build upon F1 and b) workflows through the framework. The design needs to identify how these would fit in. (The former likely have a direct placement through F1.)
3. Set the scope of the framework and the boundaries where user input (configuration, input data, algorithms) are provided, external libraries are utilized, and other parts of the software stack (that are not provided by the framework, e.g. I/O, logging, monitoring) fit in and how the framework expects to interact with them.
4. Define the flow of data products and (meta-)information, in particular to what/how input is provided and what needs to be passed on to which externals.

The initial design may impact the F1/F2 demonstrators and v.v., so this is expected to be an iterative process. Workflows in the design should be clear/detailed enough to communicate expectations to users and providers of externals.

The design can start when the first F1 demonstrator exists and any newly completed F1 demonstrators can be added iteratively until convergence. As such, these are not prerequisites to starting the work, but are for completion. A full demonstrator of the design isn't expected until after F2, with workflows in place.

The design document should be human-readable and contain:

• Diagram showing the interdependencies between user, framework, externals
• Flow of inputs, data products, and (meta-)information
• Detailed workflow examples that can be made into demonstrators

Subsystems:

• Required: all F1
• Nice to have: all for F2

[knoepfel]

Responsible developer: @knoepfel

[knoepfel]

Related to:

• Define interaction between core framework and IO backends #54

[knoepfel]

For the purposes of the F1 milestone, this discussion is closed. The job-configuration schema is not part of that initial design, but further explorations of that schema will be documented in discussion #61 as part of the F2 milestone.