Add `uses:` keyword for YAML composition

[diegolovison]

Currently, to compose multiple configuration files or override specific state variables from a base configuration, the user must provide all files as command-line arguments in a specific order.

For example, if I want semeru25.yaml to take priority over (and extend) semeru.yaml, I currently have to execute the command like this:

jbang qDup@hyperfoil -b /tmp semeru25.yaml semeru.yaml

This approach has a few downsides:

1. Command Complexity: As configurations grow, the CLI command becomes verbose and prone to ordering errors.
2. Implicit Dependencies: The entry point file (semeru25.yaml) does not explicitly state that it relies on semeru.yaml. A user must "just know" to include the base file in the CLI arguments for the script to function correctly.

Describe the solution you'd like

I would like to introduce a uses: keyword (or similar, like import or include) within the YAML definition itself. This allows a specific YAML file to explicitly define its base dependencies.

When qDup processes a file with uses:, it should load the referenced file(s) first, and then overlay the current file's configuration on top of it. This ensures the current file maintains priority (overrides) for variables and state.

Proposed Syntax Example

File: semeru.yaml (The Base)

scripts:
  my-script:
    - sh: echo "I am the base script"
STATE:
  VERSION: "1.0"

File: semeru25.yaml (The Implementation)

# Proposed new keyword
uses: semeru.yaml

# This file's definitions should take priority/merge with the used file
STATE:
  VERSION: "2.5" 

Expected Behavior

Running the command with just the single entry file:

jbang qDup@hyperfoil -b /tmp semeru25.yaml

should automatically detect the uses directive, load semeru.yaml in the background, and result in a merged configuration where VERSION is 2.5.

Additional Context

The uses keyword should support a list of files to allow for multiple layers of composition.

Precedence Rule:
The processing order should follow the list sequence, where the last file in the uses list takes precedence over the first. The current file (the one defining uses) takes priority over everything in the list.

Example:

uses:
  - base.yaml
  - middle.yaml

In this scenario:

1. base.yaml is loaded.
2. middle.yaml is loaded and overrides base.yaml.
3. The current file is loaded and overrides middle.yaml.

This feature moves dependency management from "execution time" (CLI args) to "definition time" (YAML), making scripts more self-contained, portable, and easier to share.

[willr3]

Explicit dependency management is something we considered but avoided.

It sounds like you want explicit file dependencies because your automation tasks depend on multiple yaml files and you are putting them all in the command line invocation.
qDup has ways to pass in multiple files without explicitly enumerating them.

Before we get into the design ramifications of introducing a uses concept let's consider your stated downsides to the current user experience:

1. Command Complexity: As configurations grow, the CLI command becomes verbose and prone to ordering errors.

qDup accepts a folder path as an argument and will load all yaml in that folder (using default ordering). This feature can solve the "verbose command" by putting all the helper scripts in a folder and invoking qDup with two script arguments:

qDup.jar ... main.yaml ./helpers/

We have a folder with 30+ helper files that get passed into our performance tests behind the main.yaml file. Any overrides are provided in the main.yaml and qDup runs with only two file arguments.

This could work for your use case unless the yaml files have multiple levels of overrides.
If semeru25.yaml overrides semeru.yaml which then overrides another chain of yaml then we will have to manually manage the order of qDup arguments.
Multiple tiers of overrides is possible but not something I would recommend and I honestly think it is good that we do not make it easier to perform such an anti-pattern.
Our normal use case is to control execution with state variables and pass them into the invocation with -S key=value arguments.

2. Implicit Dependencies: The entry point file (semeru25.yaml) does not explicitly state that it relies on semeru.yaml

You are correct. The yaml files are a user convenience and not something qDup uses internally. qDup loads all files into a single configuration object (following the existing conflict resolution) and performs all verification against the unified configuration.
qDup is not concerned with where an entity is defined, only that it exists when needed.

Ramifications of explicit file dependencies (not an exhaustive list):

1. Adding the name of a file into qDup's syntax will make filenames and file paths a dependency of script execution but the file names are un-related to script execution.

2. Explicit file dependency introduces ambiguity around how file dependencies interacts with overrides from earlier files.
   If semeru25.yaml explicitly uses semeru.yaml which defines an install script but a prior file already defined an install script which one is semeru25 going to invoke? qDup currently has a single policy for overrides and this feature would significantly complicate that policy. Introducing complexity to the runtime behaviour of qDup to decrease the number of command line arguments does not feel like a properly explored design decision.

3. Explicit file dependencies cannot capture all dependencies.
   qDup supports state variable reference in multiple contexts that will change the execution plan based on runtime state. For example:

- script: ${{runtimeVariable}}

If semeru25.yaml used the above command in a script then how would it interact with uses? Currently users can dynamically invoke any defined script at runtime. Does uses limit the scope of dynamic invocation? That would potentially break existing automation that rely on the current global script name lookup. Instead uses would act as a compile time hint but could not be an enforceable limit and thus would not act to ensure a file is fully executable based on the content of the file. Users would still need to properly manage the runtime file dependencies and the benefits of uses are really only a shorthand to limit the number of command line arguments.

We could spend more time exploring the detrimental ramifications of uses but instead lets consider a parallel usage model: java development.

In java we write a class file with imports for external class dependencies (or use fully qualified class names) but it is up to the person executing the code to pass in those necessary class files on the classpath.
Nothing in the java code specifies where those class files need to be located or the necessary file name (jar name). External tools (e.g. maven) are used to manage the version dependencies and resolve class files to simplify the user experience of setting up the classpath.
qDup has a similar model. qDup files do not have an explicit imports section for each external reference because qDup can identify those at runtime so we do not force the user to perform work that qDup can perform on their behalf.
qDup reports any missing dependencies at startup the same way java reports ClassNotFound during code startup and it is up to the user to decide how to resolve those missing dependencies.

What else could we do?

It sounds like you want to simpilfy the commandline execution of qDup. My first thought would be to consider if the command line needs to be growing in complexity. Do the files need to be loaded in an explicit order or is it sufficient to ensure a particular file is first and then load the rest by putting them in a folder?

If the files have multiple tiers of overrides then perhaps the override behavior is an indication that some scripts or states should be merged and controlled with state variables rather than relying on multiple, conditional overrides?

If you feel you are truly stuck with a complex order of scripts then I think we should rely on an external dependency management similar to how users rely on maven. A shell script or file name convention that sets the correct order could help.

If you still feel the uses concept is a necessity then we can more fully explore the ramifications and potential pitfalls but at the moment I think the expected benefit does not justify the introduced model complexity for both the user's mental model and the internal qDup execution model. There are likely other ways we can simplify the command line without an additional to the qDup file syntax.

[diegolovison]

My feature request is still relevant because it addresses the usability and encapsulation problems:

• Hidden Dependency: Without uses:, semeru25.yaml looks standalone, but it actually breaks if semeru.yaml isn't manually included in the command.

• Relying on users to remember the exact CLI order (child parent) is error-prone compared to uses: parent inside the file.

This feature is already implemented by the CLI, by taking the order precedence.
This feature request only adds the uses keyword which is optional.

Other users are asking for this kind of dependency as well: see #173