Enhanced Offline Mode #4479
Enhanced Offline Mode #4479
Conversation
github-actions
bot
added
documentation
Greptile OverviewGreptile SummaryThis PR implements comprehensive offline training support by adding an asset resolution system that transparently redirects Nucleus/S3 URLs to local storage. The implementation uses a two-layer strategy: monkey patching Isaac Lab spawn configs ( Key Changes:
Implementation Quality: Issues Found:
Testing: Confidence Score: 4/5
Important Files Changed
Sequence DiagramsequenceDiagram
participant User
participant TrainScript as Training Script
participant Resolver as OfflineAssetResolver
participant ConfigHooks as Monkey Patched Configs
participant FileSystem as Local Filesystem
participant Nucleus as Nucleus Server
Note over User,Nucleus: One-time Setup Phase
User->>TrainScript: Run download_assets.py
TrainScript->>Nucleus: Request asset list
Nucleus-->>TrainScript: Return file structure
TrainScript->>Nucleus: Download assets
Nucleus-->>FileSystem: Write to offline_assets/
Note over User,Nucleus: Offline Training Phase
User->>TrainScript: Run train.py --offline
TrainScript->>Resolver: setup_offline_mode()
Resolver->>Resolver: enable_offline_mode()
Resolver->>ConfigHooks: install_path_hooks()
Note over ConfigHooks: Monkey patch UsdFileCfg,<br/>GroundPlaneCfg, PreviewSurfaceCfg
TrainScript->>Resolver: patch_config_for_offline_mode(env_cfg)
Resolver->>Resolver: Patch robot, terrain, sky_light, visualizers
Note over TrainScript,FileSystem: Asset Loading
TrainScript->>ConfigHooks: Instantiate UsdFileCfg(usd_path="nucleus://...")
ConfigHooks->>Resolver: resolve_asset_path(nucleus_url)
Resolver->>Resolver: _extract_relative_path()
Note over Resolver: Extract "Robots/Go2/go2.usd"<br/>from versioned Nucleus URL
Resolver->>FileSystem: Check if offline_assets/Robots/Go2/go2.usd exists
alt Asset exists locally
FileSystem-->>Resolver: File found
Resolver-->>ConfigHooks: Return local path
ConfigHooks-->>TrainScript: Use local asset
else Asset not found
FileSystem-->>Resolver: File not found
Resolver-->>ConfigHooks: Return original Nucleus URL
ConfigHooks->>Nucleus: Fallback to Nucleus server
Nucleus-->>TrainScript: Stream asset
end
TrainScript->>TrainScript: Train environment
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| print(f"[OfflineAssetResolver] ✓ Using offline: {path_to_convert}") | ||
| return offline_path | ||
| else: | ||
| print(f"[OfflineAssetResolver] ⚠️ Not found locally: {path_to_convert}") | ||
| print("[OfflineAssetResolver] Falling back to Nucleus") |
There was a problem hiding this comment.
Printing to stdout on every asset resolution could create excessive output when processing environments with many assets. Consider using a logging level or adding a verbose flag.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
UPDATE: Addressed the following issues found:
|
Mayankm96
left a comment
Mayankm96
left a comment
There was a problem hiding this comment.
Thank you for the PR!
Still have to review in detail but at a high-level, I feel the concept of offline assets should be a global thing that lives through the AppLauncher. Right now, it only gets propogated inside the RL training scripts and not other tutorials/demos, which might be hindering.
Mayankm96
force-pushed
the
main
branch
2 times, most recently
from
2ef7fc8 to
f3061a4
Compare
clayton-littlejohn
requested review from
hhansen-bdai and
kellyguo11
as code owners
You're so right! Changes have been made and drastically reduced repetitive code. Now works alongside various tutorials and demos. ✔️ |
2 participants
Enhanced Offline Training Support for Isaac Lab
Description
This PR implements an enhanced comprehensive offline training system that enables training Isaac Lab environments, tutorials, and demos without internet connectivity. The system automatically redirects asset paths from Nucleus/S3 servers to local storage while maintaining the same directory structure, requiring zero configuration changes to existing environments.
Problem
Current Problem:
Training Isaac Lab environments requires constant internet connectivity to load assets from Nucleus/S3 servers. This creates several critical issues:
Solution
One-time asset download with automatic path resolution for seamless offline training. Simply add
--offlineflag to any training command.Key Features:
--offlinefor various training environments, tutorials, and demos--offline-permissivefallback to Nucleus for asset not found inoffline_assetsoffline_assetplug and play mirroring Nucleus structureType of change
scripts/offline_setup)Changes Made
New Files
Core System:
source/isaaclab/isaaclab/utils/asset_resolver.py- Path resolution engine with singleton pattern, monkey patching, and regex-based URL matchingAsset Management:
scripts/offline_setup/download_assets.py- Asset downloader with size calculation and progress trackingscripts/offline_setup/README.md- Complete setup and usage guideModified Files
Training Scripts (All RL Libraries):
AppLauncher. (Propagates globally across RL training scripts and other tutorials/demos)Core Integration:
source/isaaclab/isaaclab/utils/__init__.py- Added wildcard import for offline resolverHow It Works
Two-Layer Resolution Strategy
Layer 1 - Monkey Patching (90% coverage):
Patches Isaac Lab spawn configs (
UsdFileCfg,GroundPlaneCfg,PreviewSurfaceCfg) to automatically resolve paths when instantiated.Layer 2 - Config Patching (10% coverage):
Explicitly patches already-loaded configs (robot, terrain, visualizers) to catch pre-loaded paths.
Path Resolution Pipeline
Handles versioned URLs (
/Assets/Isaac/5.1/...), non-versioned URLs, and environment variables.Usage
Quick Start
1. Download essential assets (one-time,
all~60 GB)Assets download to the
~/IsaacLab/offline_assetsdirectory:cd ~/IsaacLabAlternative Note: Category fields can be specified separately
Successful Downloads
2. Train completely offline with any robot via the
--offlineflag (also works with/play)Supported for:
rl_games,rsl_rl,sb3,skrl, andsim2transferRun various demos and tutorials with
--offlineflagStrict mode (default) - fails immediately if asset not found locally
Permissive mode - warns and falls back to Nucleus if asset not found
Environment variables work too
Missing Assets Awareness
Note: For offline training, assets that cannot be found in
offline_assetswill attempted to be fetched from the Nucleus Server when using the--offline-permissiveflag.Before vs After
Before (Hardcoded approach):
After (Generic system):
Testing
Manual Testing Completed
✅ Asset Download
✅ Offline Training - All RL Libraries
✅ Various Demos and Tutorials:
✅ Multi-Robot Testing
✅ Complete Offline Verification
--offline-permissivespecified)Test Output
Screenshots
Asset Download Progress
Training Output
Breaking Changes
None. This is a purely additive feature:
--offlineflagDependencies
Required:
tqdm- For download progress bars (add to requirements:pip install tqdm)No new dependencies for core functionality - Uses existing Isaac Lab infrastructure.
Documentation
scripts/offline_setup/README.mdChecklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists thereAdditional Notes
Performance Impact
Future Enhancements
Tested On
sha f3061a464c2cded093db021233fadea63663631c)Ready for review! This implementation enables airgapped training, improves reliability, and reduces code duplication and localized complexity while maintaining full backward compatibility.