Avoid duplicating PATH#5425
Merged
adamnovak merged 6 commits intoDataBiosphere:masterfrom Feb 4, 2026
Merged
Conversation
When using the singleMachine batch system, `environment` contains the saved contents of `os.environ`. This code then effectively concatenates the original PATH saved in `environment` to the PATH in `os.environ`, which is equal. In the end, os.environ thus gets extended with its original contents. In my case, my PATH was quite long. After toil concatenated it several times to `os.environ`, `os.environ` contained 7 times the original PATH. Later, an "Argument list too long" error occured while executing a simple `stat` call. If `environment["PATH"]` already contains `os.environ["PATH"]` (which is also true of `os.environ["PATH"]` is empty or if it equals `environment["PATH]`), the concatenation is not needed, so we can skip it and avoid the "Argument list too long" error.
adamnovak
requested changes
Feb 3, 2026
Member
adamnovak
left a comment
There was a problem hiding this comment.
I think we can fix this, but I am going to try and reimplement the code change because what's here looks breakable.
src/toil/worker.py
Outdated
| # Handle path specially. Sometimes e.g. leader may not include | ||
| # /bin, but the Toil appliance needs it. | ||
| if i in os.environ and os.environ[i] != "": | ||
| if i in os.environ and os.environ[i] not in environment[i]: |
Member
There was a problem hiding this comment.
I don't think this is quite the right code for this; if the worker has PATH=/bin and the workflow is trying to apply PATH=/usr/bin:/bin:/home/username/bin, then it won't make any changes to the path.
Probably what we really want is a real union of the directory sets, with the new ones first.
adamnovak
approved these changes
Feb 3, 2026
Member
|
I've pulled this in for testing in our |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When using the singleMachine batch system,
environmentcontains the saved contents ofos.environ.This code then effectively concatenates the original PATH saved in
environmentto the PATH inos.environ, which is equal. In the end, os.environ thus gets extended with its original contents.In my case, my PATH was quite long. After toil concatenated it several times to
os.environ,os.environcontained 7 times the original PATH. Later, an "Argument list too long" error occured while executing a simplestatcall.If
environment["PATH"]already containsos.environ["PATH"](which is also true ofos.environ["PATH"]is empty or if it equalsenvironment["PATH]), the concatenation is not needed, so we can skip it and avoid the "Argument list too long" error.Changelog Entry
To be copied to the draft changelog by merger:
Reviewer Checklist
issues/XXXX-fix-the-thingin the Toil repo, or from an external repo.camelCasethat want to be insnake_case.docs/running/{cliOptions,cwl,wdl}.rstMerger Checklist