Real-Time ASL Recognition System with ESP32 Data Glove

This project implements a real-time American Sign Language (ASL) recognition system using a sensor-based data glove and Machine Learning. It captures hand gestures via an ESP32 microcontroller, processes the data, and classifies the sign using a trained Neural Network. The detected sign is then converted to speech using Text-to-Speech (TTS).

🚀 Features

• Real-time Recognition: Instantly translates sensor data into ASL signs.
• Smart Data Glove: Utilizes an ESP32 with multiple sensors (Flex, Gyroscope, Accelerometer) to capture precise hand movements.
• Deep Learning Model: Powered by a TensorFlow/Keras model (asl_model_best.h5) trained on the ASL Sensor Dataglove Dataset.
• Text-to-Speech (TTS): Audible feedback using pyttsx3 for improved accessibility.
• Robust Preprocessing: Uses StandardScaler to normalize incoming sensor data for accurate predictions.

🛠️ Hardware Requirements

• ESP32 Microcontroller
• Data Glove equipped with:
  • 5 Flex Sensors (for finger bending)
  • IMU (Inertial Measurement Unit) providing:
    • Quaternions (Orientation)
    • Gyroscope (Angular velocity)
    • Accelerometer (Linear acceleration)

💻 Software Requirements

• Python 3.7+
• TensorFlow / Keras
• NumPy
• Joblib
• PySerial
• Pyttsx3 (for TTS)

📦 Installation

1. Clone the Repository (if applicable) or download the project files.

2. Install Dependencies: Open a terminal/command prompt and run:

   pip install tensorflow numpy joblib pyserial pyttsx3

⚙️ Configuration

Before running the system, you must configure the Serial Port connection to your ESP32.

1. Connect your ESP32 to your computer via USB.
2. Find the COM port number (e.g., COM3, COM9, /dev/ttyUSB0):
   • Windows: Device Manager -> Ports (COM & LPT).
   • Linux/Mac: Run ls /dev/tty* in terminal.
3. Open realtime_asl.py in a text editor.
4. Update Line 15 with your specific COM port:

   SERIAL_PORT = "COM9"  # Change to your port, e.g., "COM3"

5. Ensure BAUD_RATE matches your ESP32 firmware setting (default: 115200).

▶️ Usage

1. Ensure the ESP32 is connected and transmitting data.
2. Run the main script:

   python realtime_asl.py

3. System Status:
   • [INFO] Listening to ESP32...: Connection successful, waiting for data.
   • Detected Sign: A: Shows the currently recognized sign.
   • Audio: The system will speak the detected sign.

📂 Project Structure

• realtime_asl.py: Main execution script. Handles serial communication, prediction, and TTS.
• asl_model_best.h5: Pre-trained Deep Learning model for sign classification.
• scaler.pkl: StandardScaler object for normalizing input data (must match training scaling).
• label_encoder.pkl: Encoder to translate model output indices back to Text labels (e.g., 0 -> 'A').
• ASL-Sensor-Dataglove-Dataset/: Directory containing the dataset used for training.

🧩 Data Format

The system expects a CSV-formatted string from the serial port with 23 values: Timestamp, User_ID, Flex_1...Flex_5, Quat_W...Quat_Z, Gyro_X...Gyro_Z, Acc_X...Acc_Z, AccBody_X...AccBody_Z, AccWorld_X...AccWorld_Z

The script ignores the first 2 values (Timestamp, User_ID) and uses the remaining 21 features for prediction.

🤝 Troubleshooting

• SerialException / Port not found:
  • Check your USB connection.
  • Verify the correct COM port is set in the code.
  • Close any other applications (like Arduino IDE serial monitor) using the port.
• Incorrect Predictions:
  • Ensure the sensors are calibrated.
  • Verify the order of sensors in your ESP32 code matches the Python script expectations.

👤 Author

• Subhra Sameer Dash

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.