Research Project: Code, Data Analysis and Modelling comparing Workload Classification and Prediction from Facial Pose, Physiological, and Eye-Tracking Measures.
This project analyzes how different measures of human behavior and physiology change during tasks with different levels of mental workload (Low, Medium, High).
We process data from:
- Facial movements - How your face moves and expressions change
- Heart activity - Heart rate patterns and variability
- Skin conductance - Electrical activity on the skin (related to stress/arousal)
- Eye movements - Where you look and how your pupils respond
- Task performance - How well you complete the tasks
All of this data is then used to train machine learning models that can automatically detect workload levels.
This repository contains analysis pipelines for workload classification using multiple physiological and behavioral measures collected during Multi-Attribute Task Battery (MATB) experiments:
- Facial Pose Analysis - Extract behavioral features from facial landmarks
- ECG Analysis - Heart rate variability and cardiac measures
- GSR Analysis - Galvanic skin response and electrodermal activity
- Eye-Tracking Analysis - Gaze patterns and pupil responses
- MATB Performance - Task accuracy and reaction times
- Multi-Modal Modeling - Machine learning classification and prediction
┌─────────────────────────────────────────────────────────────────┐
│ Raw Data Collection │
│ (Pose, ECG, GSR, Eye-Tracking, Task Performance during MATB) │
└────────────┬────────────────────────────────────────────────────┘
│
├──> Pose Pipeline ──────────> Facial Features
├──> ECG Pipeline ───────────> HRV Features
├──> GSR Pipeline ───────────> EDA Features
├──> Eye Pipeline ───────────> Gaze/Pupil Features
└──> Performance Pipeline ──> Task Metrics
│
▼
┌───────────────────────┐
│ Feature Extraction │
│ (60s windows, 50% │
│ overlap) │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ Statistical Analysis │
│ (Mixed-effects │
│ models) │
└───────────────────────┘
│
▼
┌───────────────────────┐
│ Machine Learning │
│ (Random Forest │
│ classification) │
└───────────────────────┘
│
▼
┌───────────────────────┐
│ Workload Prediction │
│ (Low/Medium/High) │
└───────────────────────┘
- Anaconda or Miniconda installed on your system
- Download from: https://www.anaconda.com/products/distribution
- Or Miniconda: https://docs.conda.io/en/latest/miniconda.html
- Git (to clone this repository)
- R (for advanced statistical analysis - will be installed via conda)
# Clone the repository
git clone [REPOSITORY_URL]
# Navigate to the project directory
cd MATB-Workload-Classification-Facial-Pose-Data# Create a new environment named 'matb-analysis' with Python 3.11
conda create --name matb-analysis python=3.11
# Activate the environment (do this every time you work on the project)
conda activate matb-analysis# Install Python packages
pip install -r requirements.txt
# Install R and R packages via conda-forge (for statistical analysis)
conda install -c conda-forge r-base rpy2
# Install required R packages for mixed-effects modeling
R -e "install.packages(c('lmerTest', 'emmeans', 'pbkrtest'), repos='https://cloud.r-project.org')"# Test Python packages
python -c "import numpy, pandas, matplotlib, seaborn, scipy; print('All Python packages installed!')"
# Test R integration
python -c "import rpy2.robjects as ro; print('R integration working!')"MATB-Workload-Classification-Facial-Pose-Data/
├── README.md # This file - project overview and setup
├── requirements.txt # Python package dependencies
│
├── pose/ # FACIAL POSE ANALYSIS (COMPLETE)
│ ├── pose_processing_pipeline.py # Main processing pipeline
│ ├── pose_processing_visualisation.ipynb # Interactive analysis notebook
│ ├── stats_figures.ipynb # Statistical analysis and plots
│ ├── utils/ # Modular analysis utilities
│ │ ├── config.py # Configuration settings
│ │ ├── io_utils.py # File I/O operations
│ │ ├── preprocessing_utils.py # Data preprocessing
│ │ ├── signal_utils.py # Signal processing
│ │ ├── normalize_utils.py # Coordinate normalization
│ │ ├── features_utils.py # Feature extraction
│ │ ├── window_utils.py # Windowing and segmentation
│ │ └── stats_utils.py # Statistical analysis
│ ├── data/ # Data directory (user-provided)
│ │ ├── pose_data/ # Raw OpenPose CSV files
│ │ └── processed/ # Processed outputs (generated)
│ └── figs/ # Generated statistical plots
│
├── ecg/ # Heart rate analysis
├── gsr/ # Galvanic skin response
├── eye_tracking/ # Gaze analysis
├── MATB_performance/ # Task performance metrics
├── modeling/ # ML classification
└── stats_utils/ # Statistical analysis utilities
The facial pose analysis pipeline includes:
- 8-step automated processing from raw OpenPose data to analysis-ready features
- 3 normalization approaches: Original, Procrustes Global, Procrustes Participant
- Smart skip logic - automatically detects and skips completed steps
- Robust error handling and progress tracking
- Behavioral metrics: Head rotation, blink patterns, mouth movements, head displacement
- Motion statistics: Velocity, acceleration, RMS for all features
- Statistical analysis: Mixed-effects models with R integration
- Publication-ready plots with significance testing
# Navigate to pose directory
cd Pose
# Run the complete processing pipeline
python process_pose_data.py
# Visualise the analysis pipline using interactive Jupyter notebook
jupyter notebook pose_processing_visualisation.ipynbEach analysis module contains its own detailed README:
- Pose Analysis - Facial landmark processing and feature extraction
- ECG Analysis - Heart rate variability analysis
- GSR Analysis - Galvanic skin response processing
- Eye-Tracking Analysis - Gaze pattern analysis
- MATB Performance - Task performance metrics
- Modeling - Machine learning classification
- Stats Utilities - Statistical analysis utilities
- Pose Processing & Visualization - Complete pipeline walkthrough with visualizations
- Pose Statistical Analysis - Publication-ready statistical plots
- ECG Analysis - ECG processing and HRV analysis
- GSR Analysis - GSR signal processing and features
- Eye Tracking Analysis - Eye tracking metrics and visualization
- Performance Analysis - MATB task performance
- Modeling Visualization - Model results and comparisons
Data files are organized within each module's data/ directory. The project expects:
data/
└── participant_info.csv # Participant metadata and condition mapping
data/
└── pose_data/
├── 3101_01_pose.csv # Participant 3101, Session 1
├── 3101_02_pose.csv # Participant 3101, Session 2
├── 3101_03_pose.csv # Participant 3101, Session 3
└── ... # Additional participants
data/
└── eyelink_data/
├── 3105_session01.csv # Participant 3105, Session 1
└── ...
data/
└── ecg_data/
├── 3105_ecg_session01.csv # ECG waveform
├── 3105_summary_session01.csv # Summary data
└── ...
data/
└── gsr_data/
├── 3208_session01.csv # Participant 3208, Session 1
└── ...
The complete dataset can be downloaded from the Open Science Framework (OSF) repository:
After downloading, place the data files in the appropriate directories as shown in the structure above.
If you use this code in your research, please cite the associated publication.
This project is licensed for academic and research use.