PainDiffusion: Learning to Express Pain

🦾🔥 IROS 2025 Best Entertainment and Amusement Paper Award 🦾🔥

---

Introduction

We introduce a generative model intended for robotic facial expression. It can generate expressions according to a signal of pain stimuli, continuously, without divergence. You can install and run it easily with the instructions provided here.

Q&A:

• Is it big? -> It is small (5.4M parameters).
• How can we know it is not divergent? -> We let it run indefinitely. We compute metrics on longer sequences (on the test set).

Code release: Due to the privacy policies of the BioVid Database, we can only release the checkpoints, training code, and inference code. To minimize our effort, we are releasing the training and preprocessing code as is for reference purposes. We have only tested and verified the inference code, which includes a Gradio-based demo.

Installation

Prerequisites

Install Inferno for the EMOCA decoder. Follow the instructions here and download the necessary models for facial reconstruction here. We have slightly modified the original code to generate useful latent variables for the face reconstruction app and to support the render_from_exp.py script. Or you can just simply follow the instructions below.

Note: Installing pytorch3d might present compatibility issues due to mismatched versions of CUDA, PyTorch, and pytorch3d. Please confirm your version of CUDA and the version that PyTorch compiled with.

Instructions

# Create and activate a new Conda environment
conda create python=3.10 -n paindiff 
conda activate paindiff
pip install -r requirements.txt

# Install the required packages and pytorch3d
# Ensure the CUDA version matches your PyTorch installation and system configuration: https://pytorch.org/get-started/locally/
conda install -c "nvidia/label/cuda-12.1.1" cuda-toolkit ninja cmake  # use the right CUDA version that you saw when run the requirement installation
ln -s "$CONDA_PREFIX/lib" "$CONDA_PREFIX/lib64"  # to avoid error "/usr/bin/ld: cannot find -lcudart"
conda env config vars set CUDA_HOME=$CONDA_PREFIX  # for compilation

# Reactivate the environment for new env config
conda deactivate
conda activate paindiff

# Install pytorch3d with cuda support
FORCE_CUDA=1 pip install git+https://github.com/facebookresearch/pytorch3d.git@stable

# Clone and set up the Inferno package
git clone https://github.com/damtien444/inferno inferno_package
cd inferno_package/
bash pull_submodules.sh
pip install -e .

# Download the pretrained EMOCA model for PainDiffusion
cd inferno_apps/FaceReconstruction
bash download_assets.sh

# Return to the repo's root dir
cd ../../..

Running the Mesh Demo

Thanks to the Hugging Face Model Hub, you can run PainDiffusion with a single command.

python online_run.py

Access at http://127.0.0.1:7860/ in your web browser.

gradio_demo.webm

Driving GaussianAvatars

For a better, realistic avatar, we use PainDiffusion to drive Gaussian Avatars. Follow the steps in this repository to do that.

drive_gaussianavatars.webm

---

Acknowledgments

We thank the previous author for their open-source code. This project is heavily based on the excellent implementations of diffusion models from:

• modular-diffusion
• denoising-diffusion-pytorch
• k-diffusion

Citation

@misc{dam2025paindiffusionlearningexpresspain,
      title={PainDiffusion: Learning to Express Pain}, 
      author={Quang Tien Dam and Tri Tung Nguyen Nguyen and Yuki Endo and Dinh Tuan Tran and Joo-Ho Lee},
      year={2025},
      url={https://arxiv.org/abs/2409.11635}, 
}

Work in progress: Pain Expression Editing Software

Pain facial expression is not an easy data to collect. To provide an easier way to augment data of pain expression, the following project aims to edit the existing expression sequence to follow the desired characteristic through textual or acting video guidance. Please see the current internal demo of the software:

master-editing-ui.1.mp4

IROS 2025 Best Entertainment and Amusement Paper Award

We are honored to be nominated as the Best Paper Award Finalist (40/1991 ~ top 2%).

We are delighted that the paper won the IROS Best Entertainment and Amusement Paper Award. This is an award in the emotional, entertainment, and social robotics category at IROS 2025, with 5 finalists within its scope.