Should Swell Tier 2 tests run Tier 1 tests?

[Dooruk]

Tier 2 is designed to build JEDI every day and test different suites, and update the stable build that Swell uses if all of the tests run successfully. However, the way it's currently designed, Tier 1 and Tier 2 run different suites. So, only some suites get tested daily while others are only tested manually or when a PR is under review/merged.

Would it be possible to:

1. Define the list of suites to tested with Tier 1? So Github runners can loop over the list rather than replicating the same YAML code. This list can exist outside of the workflows folder (?) so one can propose changes (i.e., add another tier 1 test) and wouldn't have to change the workflow YAMLs.

2. Make the following part of the Tier 2 to run tests with tier 1 suites.

CI-workflows/.github/workflows/swell-tier2_application_discover.yml

Lines 99 to 101 in 90ca9d0

	# ----------------------------------------
	# STEP2: RUN TESTING SUITES WITH NEW BUILD
	# ----------------------------------------

FYI, @rtodling.

[jardizzo]

It's possible to modify the workflows to accept arguments. In this manner, the dispatching workflow could supply the suite/experiment to execute. You would need to have one dispatch workflow for each experiment. Let me know if I'm properly understanding the proposed capability.

[Dooruk]

Instead of repeating this block of code for each suite, we would have a list of Swell suites to run (3dvar, 3dvar_atmos, hofx etc.) and this following codeblock would be looping over that list, instead of 3dvar we would template it:

CI-workflows/.github/workflows/swell-tier1_application_discover.yml

Lines 139 to 188 in 90ca9d0

	# Run 3dvar workflow
	# -----------------
	swell-tier_1-3dvar:
	runs-on: nccs-discover
	timeout-minutes: 600
	needs: swell-tier_1-setup
	steps:
	- name: run-swell-3dvar
	run: |
	CI_WORKSPACE=/discover/nobackup/gmao_ci/swell/tier1/${GITHUB_RUN_ID}
	SUITE_NAME=3dvar
	CI_WORKSPACE_JOB=/discover/nobackup/gmao_ci/swell/tier1/${GITHUB_RUN_ID}/${SUITE_NAME}
	EXPERIMENT_ID=swell-${SUITE_NAME}-${GITHUB_RUN_ID}
	mkdir -p $CI_WORKSPACE_JOB
	source /discover/nobackup/gmao_ci/swell/tier1/${GITHUB_RUN_ID}/modules
	# Get python version
	PYVER=`python --version | awk '{print $2}' | awk -F. '{print $1"."$2}'`
	export PATH=$CI_WORKSPACE/swell/bin:$PATH
	export PYTHONPATH=${PYTHONPATH}:$CI_WORKSPACE/swell/lib/python$PYVER/site-packages
	echo "experiment_id: $EXPERIMENT_ID" > $CI_WORKSPACE_JOB/${SUITE_NAME}-override.yaml
	echo "experiment_root: $CI_WORKSPACE_JOB" >> $CI_WORKSPACE_JOB/${SUITE_NAME}-override.yaml
	rm -r -f $HOME/cylc-run/${EXPERIMENT_ID}-suite
	cd $CI_WORKSPACE_JOB
	swell create ${SUITE_NAME} -m defaults -p nccs_discover -o $CI_WORKSPACE_JOB/${SUITE_NAME}-override.yaml
	swell launch $CI_WORKSPACE_JOB/${EXPERIMENT_ID}/${EXPERIMENT_ID}-suite --no-detach --log_path $CI_WORKSPACE_JOB/${EXPERIMENT_ID}
	# Move experiment directory on failure
	swell-tier_1-3dvar-failure:
	runs-on: nccs-discover
	timeout-minutes: 30
	needs: swell-tier_1-3dvar
	if: failure()
	steps:
	- name: Fail hold for 3dvar
	run: |
	SUITE_NAME=3dvar
	CI_WORKSPACE_JOB=/discover/nobackup/gmao_ci/swell/tier1/${GITHUB_RUN_ID}/${SUITE_NAME}
	mv $CI_WORKSPACE_JOB ${CI_WORKSPACE_JOB}_FAILED

I just don't know if it would work within the confines of assigning single runner for each workflow. Also there are keys and values that would need to be templated.