Special values of max parallel downloads #163
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…y installed data from a previously inactive data node. - Support special exceptions to the sdt.conf variable (in the [download] section) max_parallel_download_per_datanode, the maximum number of parallel downloads for any datanode. A new [download] variable, max_parallel_download_special_datanodes, allows different variables for specified data_nodes. The full data_node name does not have to be specified; a substring will do. For example: max_parallel_download_special_datanodes=crd-esgf-drc:3, tropmet:1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sometimes, notably for CCCma, a data node enforces a maximum number of simultaneous connections, which differs from what we would want for most data nodes. To transfer data from such a node, we need a special node-dependent value of max_parallel_download. That is what this patch enables. To use it, set max_parallel_download_special_datanodes in the download section of the configuration file. The value is a comma-separated list of special datanodes and values (separated with a colon) such as:
max_parallel_download_special_datanodes = crd-esgf-drc:3, tropmet:1
Note that the full datanode url or name is not required; you only need a substring.