[WIP] Add Visualizers and Scene Data Providers

[matthewtrepte]

Description

Initial impl to add visualizers and scene data providers with omni physics backend.
Also has a few updates for compatibility Isaac Sim 6.0.

visualizers

• OVVisualizer
• NewtonVisualizer
• RerunVisualizer

scene data providers

• OVSceneDataProvider (scene data from omniverse physX backend)
• NewtonSceneDataProvier (scene data from newton warp physics backend)

Type of change

• New feature (non-breaking change which adds functionality)

Screenshots

Please attach before and after screenshots of the change if applicable.

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have updated the changelog and the corresponding version in the extension's config/extension.toml file
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[greptile-apps]

Greptile Overview

Greptile Summary

This PR introduces a modular visualization framework for Isaac Lab that abstracts different visualizer backends (Newton OpenGL, Rerun, Omniverse) behind a unified interface. The implementation adds scene data providers to adapt physics state between backends (PhysX ↔ Newton) and integrates visualizers into the simulation lifecycle.

Key Changes

• Visualizer abstraction: New base Visualizer class with lifecycle methods (initialize, step, close) and three implementations: NewtonVisualizer (OpenGL), RerunVisualizer (web), and OVVisualizer (native Omniverse)
• Scene data providers: Adapts physics data between backends - OVSceneDataProvider converts PhysX state to Newton format, NewtonSceneDataProvider provides native Newton data
• Simulation integration: Visualizers auto-initialize on play() and step with physics via SimulationContext.step_visualizers()
• Backend detection: Scene data providers conditionally build expensive adapted data only when needed by active visualizers

Issues Found

• Debug code left in production: commented ipdb breakpoints and print statements in ov_scene_data_provider.py:124-129 and manager_based_env.py:145-151
• Fabric cloning forcibly disabled in interactive_scene.py:124 without clear justification - needs verification this doesn't break existing workflows
• WIP markers indicate incomplete implementation

Confidence Score: 3/5

• PR has solid architecture but contains WIP code with debug artifacts and disabled features requiring cleanup before merge
• Architecture is well-designed with good separation of concerns, but the WIP nature is evident with debug code, commented breakpoints, and forcibly disabled Fabric cloning that could affect production use
• Pay close attention to interactive_scene.py (Fabric cloning disabled), ov_scene_data_provider.py (debug code, quaternion conversion logic), and manager_based_env.py (debug prints)

Important Files Changed

Filename	Overview
source/isaaclab/isaaclab/visualizers/newton_visualizer.py	Newton OpenGL visualizer implementation with custom training/rendering controls
source/isaaclab/isaaclab/sim/scene_data_providers/ov_scene_data_provider.py	Complex PhysX-to-Newton data adapter with quaternion conversion logic and view management
source/isaaclab/isaaclab/sim/simulation_context.py	Integration of visualizers into simulation lifecycle with auto-initialization
source/isaaclab/isaaclab/scene/interactive_scene.py	Debug prints and Fabric cloning disabled, publishes scene metadata to settings
source/isaaclab/isaaclab/envs/manager_based_env.py	Debug print added for env prim count verification

Sequence Diagram

sequenceDiagram
    participant User
    participant SimContext as SimulationContext
    participant SceneDP as SceneDataProvider
    participant OVProvider as OVSceneDataProvider
    participant NewtonProvider as NewtonSceneDataProvider
    participant Visualizer as Visualizer (Newton/Rerun/OV)
    participant PhysX as PhysX Backend
    participant Newton as Newton Backend

    User->>SimContext: play() / start simulation
    SimContext->>SimContext: initialize_visualizers()
    SimContext->>SceneDP: new SceneDataProvider(backend, visualizer_cfgs)
    
    alt Physics Backend = "omni"
        SceneDP->>OVProvider: new OVSceneDataProvider(visualizer_cfgs, stage, sim_ctx)
        OVProvider->>OVProvider: _build_newton_model_from_usd() if newton/rerun visualizers
        OVProvider->>PhysX: create_rigid_body_view(), create_articulation_view()
    else Physics Backend = "newton"
        SceneDP->>NewtonProvider: new NewtonSceneDataProvider(visualizer_cfgs)
        NewtonProvider->>Newton: access NewtonManager._model, _state_0
    end

    SimContext->>Visualizer: create_visualizer() from config
    SimContext->>Visualizer: initialize(scene_data)
    Visualizer->>SceneDP: get_newton_model() / get_usd_stage()
    SceneDP-->>Visualizer: model/state/stage data
    Visualizer->>Visualizer: setup viewer (ViewerGL/ViewerRerun/OV)

    loop Every simulation step
        User->>SimContext: step()
        SimContext->>PhysX: physics step (if PhysX backend)
        SimContext->>Newton: physics step (if Newton backend)
        SimContext->>SimContext: step_visualizers(dt)
        SimContext->>SceneDP: update()
        
        alt Physics Backend = "omni" and needs Newton data
            OVProvider->>PhysX: get_world_poses(), get_velocities()
            OVProvider->>OVProvider: sync PhysX transforms to Newton state
            OVProvider->>OVProvider: convert quaternions (wxyz->xyzw)
        end
        
        SimContext->>Visualizer: step(dt)
        Visualizer->>SceneDP: get_newton_state() or get_usd_stage()
        SceneDP-->>Visualizer: updated state/stage
        Visualizer->>Visualizer: render frame (ViewerGL/ViewerRerun/OV)
    end

    User->>SimContext: stop() / clear_instance()
    SimContext->>SimContext: close_visualizers()
    SimContext->>Visualizer: close()
    Visualizer->>Visualizer: cleanup resources

Loading

[greptile-apps]

_{4 files reviewed, 4 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Debug print and commented debugger breakpoint left in production code

Suggested change

	from isaaclab.sim.utils import find_matching_prim_paths
	env_prim_paths = find_matching_prim_paths("/World/envs/env_.*", stage=self.scene.stage)
	print(
	"[SceneDebug] env prims after InteractiveScene: "
	f"num_envs_setting={self.cfg.scene.num_envs}, env_prims={len(env_prim_paths)}"
	)

[greptile-apps]

Commented debugger breakpoint left in code

[greptile-apps]

Debug print statement left in production code

[greptile-apps]

Fabric cloning disabled globally - verify this doesn't break existing functionality