Slurm batch system can submit jobs multiple times if `sbatch` manages to get the job in but fails

[adamnovak]

The Slurm batch system has logic around retrying sbatch submission commands when they fail.

However, @glennhickey had a job submission fail with exit status 1 and sbatch: error: Batch job submission failed: Socket timed out on send/recv operation, but the Slurm leader still actually received and ran the job. (We did not save standard output for that job submission, so we don't know whether we received the ID assigned to the submitted job, in addition to the error message on standard error.)

When Toil retried the job submission to Slurm, it actually resulted in a duplicate instance of the job running at the same time, which then, as far as I can tell, resulted in bad state in the job store as the two jobs fought over whose writes to promise files ought to win. We don't have anything in the job store logic to make promise-fulfillment writes atomic with updates to job state; we rely on knowing that a job failed and re-running it to successful completion to get a coherent view of all the promise files it needed to update.

We need to figure out how to get exactly one copy of a job submitted to Slurm, and to know its Slurm ID, even if we get disconnected from the Slurm leader in the middle of the submission process. It's not clear if sbatch guarantees that the job ID will be on standard output if and only if the Slurm leader accepted the job. We might need code to go sniff the jobs in queue or running to make sure none of them look suspiciously like a job we failed to submit. Or we might need to do something to the sbatch command to make sure it is rejected if the job already exists.

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1812

[adamnovak]

On the Slurm mailing list, it was suggested that --dependency=singleton might help here; it prevents multiple jobs with the same job name and user from running at a time.

I think if I disambiguate names between multiple workflow runs somehow, and make sure the case of a job scheduling and finding out it has already been completed is handled (which I think it is; I think it fails on the worker and then the leader sees it is gone), I might not need a bunch of code to ask Slurm if a job was really submitted or not.

[adamnovak]

For succinct workflow disambiguation, now that we have Toil history and its database, we should have access to a unique, increasing number for each Toil workflow within a user. So we don't have to use the workflow UUID that I think we have.

[adamnovak]

It was also suggested that it might be possible to address this using Slurm's WCKeys feature, which lets users attach a "Workload Characterization Key" to a job. https://github.com/WFU-HPC/OOD-MultitenantApps was put forward as a model.

It's not entirely clear to me how this would work (would we use one WCKey per job that's globally unique? Would we use one per workload?) and WCKeys don't seem to be meant for there to be a large number of them.

[adamnovak]

More fun facts about WCKeys and how to encode a lot in a job name are in "page 38-60 of https://github.com/WFU-HPC/OOD-MultitenantApps/blob/main/presentation.pdf".

We might be able to use one WCKey per workflow, to let the workflow efficiently fetch its jobs, but --dependency=singleton might work better.