Test SKU customisations #50
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Define the directory hierarchy for cloud specific tools, scripts resources and tests and encode them into variables for common usage. The structure we want to use for static files follows this template: /opt/hpc/<cloud-vendor>/bin/ # one-off binaries and scripts /opt/hpc/<cloud-vendor>/lib/... # resources and libraries /opt/hpc/<cloud-vendor>/tools/... # standalone tools /opt/hpc/<cloud-vendor>/tests/ # test scripts for local/CI testing For runtime files (e.g. configuration files set up by boot services), we will store them in: /var/hpc/<cloud-vendor>/.... These common directories will be defined by the following set of variables: __hpc_<cloud>_resource_dir # /opt/hpc/<cloud-vendor>/ __hpc_<cloud>_tools_dir # /opt/hpc/<cloud-vendor>/tools/ __hpc_<cloud>_tests_dir # /opt/hpc/<cloud-vendor>/tests/ __hpc_<cloud>_runtime_dir # /var/hpc/<cloud-vendor>/ At the moment we only support Azure, so only those variables are defined. Signed-off-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <dchinner@redhat.com>
These scripts tune the hardware and software according to the type of Azure VM the image is running on. They run at machine startup and source the relevant information according to the machine type that is detected. Signed-off-by: Dave Chinner <dchinner@redhat.com>
When testing the setup script with unknown SKUs, it was found that this fails to clean the configuration files properly: .... Testing some_unknown_sku_for_testing No SKU customization for some_unknown_sku_for_testing Unknown SKU Failed: some_unknown_sku_for_testing: /etc/nccl.conf not empty $ Fix this up by removing the various sku specific files on unknown SKU types. Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewer's GuideAdds Azure HPC SKU customisation support with scripts, topology files, and tests, wiring them into the Ansible role behind a new hpc_sku_customisation toggle and installing a systemd service to apply SKU-specific NCCL/topology tweaks at boot. Sequence diagram for SKU customisation at boot via systemd servicesequenceDiagram
participant Systemd
participant SkuCustomisationService
participant SetupScript
participant AzureIMDS
participant SkuCustomisationHandler
participant NvidiaFabricManager
participant NvidiaDCGM
participant NCCL
Systemd->>SkuCustomisationService: start sku_customisation.service
SkuCustomisationService->>SetupScript: exec setup_sku_customisations.sh
SetupScript->>SetupScript: init NCCL_CONF / topology runtime dirs
alt sku mocked
SetupScript->>SetupScript: read env __MOCK_SKU as sku
else sku from IMDS
loop up to 5 retries
SetupScript->>AzureIMDS: GET /metadata/instance vmSize
AzureIMDS-->>SetupScript: vmSize or empty
end
SetupScript->>SetupScript: tolower(sku)
end
alt known SKU pattern
SetupScript->>SkuCustomisationHandler: run ncv4.sh | ndv4.sh | ndv5.sh | ndv2.sh | ncv5.sh | ndv6.sh | hbv4.sh
SkuCustomisationHandler->>SetupScript: configure TOPOLOGY_FILE / TOPOLOGY_GRAPH
opt NCCL tuning
SkuCustomisationHandler->>SetupScript: append NCCL_* to NCCL_CONF
end
opt fabric manager control
SkuCustomisationHandler->>NvidiaFabricManager: systemctl enable/start
NvidiaFabricManager-->>SkuCustomisationHandler: is-active status
end
opt NVLink workaround
SkuCustomisationHandler->>NvidiaDCGM: stop nvidia-dcgm.service
SkuCustomisationHandler->>NvidiaDCGM: reload NVIDIA kernel modules
SkuCustomisationHandler->>NvidiaDCGM: start nvidia-dcgm.service
end
SetupScript->>SetupScript: add NCCL_TOPO_FILE / NCCL_GRAPH_FILE to NCCL_CONF
else unknown SKU
SetupScript->>SetupScript: remove TOPOLOGY_FILE / TOPOLOGY_GRAPH / NCCL_CONF
end
SetupScript-->>SkuCustomisationService: exit status
SkuCustomisationService-->>Systemd: service result
Systemd-->>NCCL: NCCL_CONF ready for MPI jobs at runtime
Sequence diagram for manual SKU setup test using mocked SKUsequenceDiagram
actor Admin
participant TestScript
participant SetupScript
participant SkuCustomisationHandler
participant NvidiaFabricManager
Admin->>TestScript: sudo test-sku-setup.sh --manual
TestScript->>TestScript: select test SKU
loop for each test_sku
TestScript->>TestScript: export __MOCK_SKU=test_sku
TestScript->>SetupScript: run setup_sku_customisations.sh
SetupScript->>SetupScript: detect __MOCK_SKU and set sku
SetupScript->>SkuCustomisationHandler: dispatch per SKU
opt SKU requires fabric manager
SkuCustomisationHandler->>NvidiaFabricManager: enable/start
NvidiaFabricManager-->>SkuCustomisationHandler: status
end
SetupScript-->>TestScript: exit code
alt exit code 0
TestScript->>Admin: "Test Passed: test_sku"
else failure
TestScript->>Admin: log failure details
end
end
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- The systemd unit name used in the test script (
systemctl is-active --quiet sku-customisations) does not match the installed service name (sku_customisation.service); aligning these (and choosing a single spelling/format) will avoid false negatives in service state checks. - The IMDS SKU lookup and retry logic is duplicated between
setup_sku_customisations.shandtest-sku-setup.sh; consider factoring this into a common helper or at least a shared function to keep the behavior consistent and easier to change. - Several SKU customisation scripts (e.g. NDv4/NDv5) contain very similar
nvidia-fabricmanagerenable/start/error-handling blocks; extracting this into a common helper or function would reduce repetition and the risk of divergent behavior between SKUs.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The systemd unit name used in the test script (`systemctl is-active --quiet sku-customisations`) does not match the installed service name (`sku_customisation.service`); aligning these (and choosing a single spelling/format) will avoid false negatives in service state checks.
- The IMDS SKU lookup and retry logic is duplicated between `setup_sku_customisations.sh` and `test-sku-setup.sh`; consider factoring this into a common helper or at least a shared function to keep the behavior consistent and easier to change.
- Several SKU customisation scripts (e.g. NDv4/NDv5) contain very similar `nvidia-fabricmanager` enable/start/error-handling blocks; extracting this into a common helper or function would reduce repetition and the risk of divergent behavior between SKUs.
## Individual Comments
### Comment 1
<location> `README.md:198` </location>
<code_context>
+Whether to install the hardware tuning files for different Azure VM types (SKUs).
+
+This will install definitions for optimal hardware configurations for the different types of high performance VMs that are typically used for HPC workloads in the Azure environment.
+These include Infiniband and GPU/NVLink and NCCL customisations, as well as any workarounds for specific hardware problems that may be needed.
+
+Default: `true`
</code_context>
<issue_to_address>
**suggestion (typo):** Use the standard spelling "InfiniBand" for the interconnect name.
Use the vendor-standard capitalization "InfiniBand" to match industry and Azure documentation conventions.
```suggestion
These include InfiniBand and GPU/NVLink and NCCL customisations, as well as any workarounds for specific hardware problems that may be needed.
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Add a script to enable both manual and automated testing of the Azure SKU customisation scripts. When running the tests manually, it will exercise all the different supported SKU types via mocking and checking that appropriate links are installed. It will not check that the customisation service is active and running as manual mode is expected to used on dev machines that are unsupported SKU types. Manual testing like this may throw some warnings or errors because hardware is not directly supported. For example, testing on a VM type that does not have GPUs that are supported by the fabric manager will result in warnings that the service failed to start: $ sudo /opt/hpc/azure/tests/test-sku-setup.sh --manual Testing standard_nc96ads_a100_v4 Test Passed: standard_nc96ads_a100_v4 Testing standard_nd40rs_v2 Test Passed: standard_nd40rs_v2 Testing standard_nd96asr_v4 Job for nvidia-fabricmanager.service failed because the control process exited with error code. See "systemctl status nvidia-fabricmanager.service" and "journalctl -xeu nvidia-fabricmanager.service" for details. NVIDIA Fabric Manager Inactive! Test Passed: standard_nd96asr_v4 Testing standard_hb176rs_v4 Test Passed: standard_hb176rs_v4 Testing standard_nc80adis_h100_v5 Check NVLink status after reloading NVIDIA kernel modules... NVLink is Active. Test Passed: standard_nc80adis_h100_v5 Testing standard_nd96isr_h200_v5 Job for nvidia-fabricmanager.service failed because the control process exited with error code. See "systemctl status nvidia-fabricmanager.service" and "journalctl -xeu nvidia-fabricmanager.service" for details. NVIDIA Fabric Manager Inactive! Test Passed: standard_nd96isr_h200_v5 $ Such warnings are fine. When not in manual mode, the test expects that it is running on a supported SKU VM (e.g. in the CI system) and will query the current the SKU type. If the SKU is unsupported, it will check that no files are currently installed. It will fail in that case: $ sudo /opt/hpc/azure/tests/test-sku-setup.sh Unknown SKU Failed: Standard_NC8as_T4_v3: /etc/nccl.conf not empty $ If the SKU is supported, it will check that appropriate files are installed and the service is running. Signed-off-by: Dave Chinner <dchinner@redhat.com>
dgchinner
force-pushed
the
test-sku-customisations
branch
from
6201e98 to
abb3d95
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Enhancement: Test script for the SKU customisations feature in PR #49
This is dependent on the changes in RP #48 and PR #49, the commits a duplicated in the branch the PR is generated from.
Manual (mocked SKU) testing:
Testing the SKU installed correctly and the service is running on a given VM running the built image (e.g. via a CI system):
Issue Tracker Tickets (Jira or BZ if any): RHELHPC-126
Summary by Sourcery
Add Azure-specific SKU customisation support for NCCL/topology tuning on HPC VMs and provide tooling to manage and validate these configurations.
New Features:
Enhancements:
Tests: