DROID Sim Evaluation

This repository contains scripts for evaluating DROID policies in a simple ISAAC Sim environment.

Here is an example rollout of a pi0-FAST-DROID policy:

Scene 1

Scene 2

Scene 3

The simulation is tuned to work zero-shot with DROID policies trained on the real-world DROID dataset, so no separate simulation data is required.

Note: The current simulator works best for policies trained with joint position action space (and not joint velocity control). We provide examples for evaluating pi0-FAST-DROID policies trained with joint position control below.

Installation

Clone the repo

git clone --recurse-submodules git@github.com:arhanjain/sim-evals.git
cd sim-evals

Install uv (see: https://github.com/astral-sh/uv#installation)

For example (Linux/macOS):

curl -LsSf https://astral.sh/uv/install.sh | sh

Create and activate virtual environment

uv sync
source .venv/bin/activate

Quick Start

First, make sure you download the simulation assets into the root of this directory

uvx hf download owhan/DROID-sim-environments --repo-type dataset --local-dir assets

Then, in a separate terminal, launch the policy server on localhost:8000. For example, to launch a pi0-FAST-DROID policy (with joint position control), checkout openpi and use the polaris configs

XLA_PYTHON_CLIENT_MEM_FRACTION=0.5 uv run scripts/serve_policy.py policy:checkpoint --policy.config=pi05_droid_jointpos_polaris --policy.dir=gs://openpi-assets/checkpoints/pi05_droid_jointpos

Note: We set XLA_PYTHON_CLIENT_MEM_FRACTION=0.5 to avoid JAX hogging all the GPU memory (incase Isaac Sim is using the same GPU).

Finally, run the evaluation script:

python run_eval.py --episodes [INT] --scene [INT] --headless

Minimal Example

env_cfg.set_scene(scene) # pass scene integer
env = gym.make("DROID", cfg=env_cfg)

obs, _ = env.reset()
obs, _ = env.reset() # need second render cycle to get correctly loaded materials
client = # Your policy of choice

max_steps = env.env.max_episode_length
for _ in tqdm(range(max_steps), desc=f"Episode"):
    action = client.infer(obs, INSTRUCTION) # calling inference on your policy
    action = torch.tensor(ret["action"])[None]
    obs, _, term, trunc, _ = env.step(action)
    if term or trunc:
        break
env.close()