[patch] Add gitops initial user creation automation FVT test

[tomklapiscak]

Description

Adds an automated test to the fvt-preparer task to drive and verify the new automation for creating initial users in SaaS deployments.

See https://pages.github.ibm.com/maximoappsuite/saas/walkthrough/#automated-creation-of-initial-users for details.

The test runs twice: once in the core phase and once again in the apps phase (to drive the logic that grants application roles and assigns Manage Security Groups to the initial users):

• The initial_users secret is primed with 2 primary and 2 secondary users with randomised emails
• The ibm-create-initial-users Job that reads the secret and creates the users is deleted so ArgoCD resyncs it, causing it to be rerun
• The test passes if the postsyncjobs app that contains the Job becomes healthy again after rerunning the Job

The results of the tests are registered with the FVT dashboard under ibm-mas-gitops/initial-users-automation ${LAUNCHER_ID}-${MAS_INSTANCE_ID}

This SaaS-specific test is performed in the pipeline itself (rather than in the fvt-suites) since the ArgoCD sync window established for the duration of the fvt-suites run would prevent ArgoCD from resyncing the initial-users Job after we delete it (which is the only sensible way to retrigger it).

The script is designed to not block the whole FVT run if something goes wrong.

Related PRs

ibm-mas/python-devops#66
ibm-mas/gitops#278
https://github.ibm.com/maximoappsuite/saas-tekton/pull/127

Testing

Verified running in situ against noble4: dev-gitops-mas-fvt-preparer run:

# - Automated Initial User Tests
# -----------------------------------
Logging into AWS SecretsManager ...
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     **************** shared-credentials-file    
secret_key     **************** shared-credentials-file    
    region                us-east-2              env    ['AWS_REGION', 'AWS_DEFAULT_REGION']
argo:argocd_login : Logging into ArgoCD ...
argo:argocd_login : ARGOCD_URL=[openshift-gitops-server-openshift-gitops.apps.noble4.cp.fyre.ibm.com](http://openshift-gitops-server-openshift-gitops.apps.noble4.cp.fyre.ibm.com/) ARGOCD_USERNAME=*** ARGOCD_PASSWORD=****<snip> ...
'admin:login' logged in successfully
Context '[openshift-gitops-server-openshift-gitops.apps.noble4.cp.fyre.ibm.com](http://openshift-gitops-server-openshift-gitops.apps.noble4.cp.fyre.ibm.com/)' updated
argo:argocd_login : return_code=0
argo:argocd_login : ArgoCD login success
argo:check_argo_app_healthy : Checking health status for postsyncjobs.noble4.tgk01 up to 20 times
Health Status is Healthy
Creating fyre-noble4-dev/noble4/tgk01/initial_users secret
Secret Manager: Updating fyre-noble4-dev/noble4/tgk01/initial_users with tags [{"Key": "source", "Value": "gitops-mas-fvt-preparer"}, {"Key": "account", "Value": "fyre-noble4-dev"}, {"Key": "cluster", "Value": "noble4"}]
- Secret fyre-noble4-dev/noble4/tgk01/initial_users updating
{
    "ARN": "arn:aws:secretsmanager:us-east-2:435377327995:secret:fyre-noble4-dev/noble4/tgk01/initial_users-2RhU1H",
    "Name": "fyre-noble4-dev/noble4/tgk01/initial_users",
    "VersionId": "dca25c85-a1d5-49ff-9c88-8a790a78c405"
}
- Secret fyre-noble4-dev/noble4/tgk01/initial_users updated
Deleting the following ibm-create-initial-user Jobs in mas-tgk01-postsyncjobs namespace:
ibm-create-initial-users-3918966094
ibm-create-initial-users-3918966094
job.batch "ibm-create-initial-users-3918966094" deleted
Forcing postsyncjobs.noble4.tgk01 to resync
Force Application postsyncjobs.noble4.tgk01 to Sync ...
time="2025-05-13T11:27:14Z" level=fatal msg="rpc error: code = FailedPrecondition desc = another operation is already in progress"
ArgoCD response for Force Application postsyncjobs.noble4.tgk01 to Sync: 20
argo:check_argo_app_healthy : Checking health status for postsyncjobs.noble4.tgk01 up to 20 times
argo:check_argo_app_healthy : Health Status is Missing, Waiting 30s before checking status again - 1
argo:check_argo_app_healthy : Checking health status for postsyncjobs.noble4.tgk01 up to 20 times
argo:check_argo_app_healthy : Health Status is Missing, Waiting 30s before checking status again - 2
argo:check_argo_app_healthy : Checking health status for postsyncjobs.noble4.tgk01 up to 20 times
argo:check_argo_app_healthy : Health Status is Progressing, Waiting 30s before checking status again - 3
argo:check_argo_app_healthy : Checking health status for postsyncjobs.noble4.tgk01 up to 20 times
argo:check_argo_app_healthy : Health Status is Progressing, Waiting 30s before checking status again - 4
argo:check_argo_app_healthy : Checking health status for postsyncjobs.noble4.tgk01 up to 20 times
argo:check_argo_app_healthy : Health Status is Progressing, Waiting 30s before checking status again - 5
argo:check_argo_app_healthy : Checking health status for postsyncjobs.noble4.tgk01 up to 20 times
argo:check_argo_app_healthy : Health Status is Progressing, Waiting 30s before checking status again - 6
argo:check_argo_app_healthy : Checking health status for postsyncjobs.noble4.tgk01 up to 20 times
argo:check_argo_app_healthy : Health Status is Progressing, Waiting 30s before checking status again - 7
argo:check_argo_app_healthy : Checking health status for postsyncjobs.noble4.tgk01 up to 20 times
argo:check_argo_app_healthy : Health Status is Progressing, Waiting 30s before checking status again - 8
argo:check_argo_app_healthy : Checking health status for postsyncjobs.noble4.tgk01 up to 20 times
argo:check_argo_app_healthy : Health Status is Progressing, Waiting 30s before checking status again - 9
argo:check_argo_app_healthy : Checking health status for postsyncjobs.noble4.tgk01 up to 20 times
argo:check_argo_app_healthy : Health Status is Progressing, Waiting 30s before checking status again - 10
argo:check_argo_app_healthy : Checking health status for postsyncjobs.noble4.tgk01 up to 20 times
Health Status is Healthy
Run [save-junit-to-mongo.py](http://save-junit-to-mongo.py/)
MongoDb integration enabled (v2 data model)
Instance ID ............ tgk01
Product ID ............. ibm-mas-gitops
Build .................. 2764
Suite .................. initial-users-automation-core-tgk01
Channel ID ............. n/a
CLI Version ............ 13.20.0
mas_devops Version ..... unknown
Run ID ................. tgk01:2764
Result ID .............. tgk01:2764:ibm-mas-gitops:initial-users-automation-core-tgk01
Pipeline results saved to MongoDb (v2 data model)

Also run manually from laptop to test failure cases: