LLM-lightshow

This project came about due to a hackathon idea where we wanted to create a lightshow synchronized to music using an ESP32 microcontroller. The goal was to leverage large language models (LLMs) to generate light patterns based on the audio input. This was inspired by the increasing capabilities of LLMs in understanding and generating content. Running an LLM on an ESP32 is challenging due to its limited resources, so we focused on optimizing the model and using efficient libraries. Moreover, we integrated FFT (Fast Fourier Transform) to analyze the audio signals in real-time, allowing the light patterns to respond dynamically to the music.

The idea is that each segment of audio can be processed to extract frequency components, which can then be mapped to specific light patterns generated by the LLM. This combination of LLMs and FFT on a microcontroller opens up new possibilities for interactive and immersive experiences in small form factors. The final model was able to generate at an inference speed of 40 tokens per second on the ESP32, making it feasible for real-time applications like a music-synced lightshow.

Another challenging aspect was getting the audio input and fft processing to work efficiently on the ESP32. We utilized optimized libraries and techniques to ensure that the audio analysis did not bottleneck the performance of the LLM inference. Essentially, we utilized the ESP32S3's dual core architecture to run FFT processing and audio streaming to DAC module on separate cores, while the LLM inference ran on a separate ESP32S2 board. This parallel processing is what enabled us to get song + preprocessed FFT data to the LLM at a sufficient speed to generate real-time light patterns.

For the model training, we created a custom dataset that paired audio features (like FFT data) with desired light patterns. This dataset was used to train from scratch a smaller LLM, ensuring it could generate relevant light patterns based on the audio input. The training scripts and dataset preparation code are included in the train_scripts directory. The folder includes a lot of files from the llama2.c repository, as we built upon that codebase to create our custom model.

In summary, Our LLM-lightshow project demonstrates the potential of combining large language models with real-time audio processing on resource-constrained devices like the ESP32. By leveraging FFT for audio analysis and optimizing the LLM for inference speed, we were able to create an engaging lightshow experience that responds dynamically to music. While the model could definitely do better at recognizing the patterns of a song, it was a fun and educational project that showcased the capabilities of modern AI and embedded systems.

Project Structure

• esp32-llm/: Contains the ESP32 code for running the LLM inference.
• embed-fft/: Contains the ESP32 code for audio input and FFT processing.
• train_scripts/: Contains scripts and datasets used for training the custom LLM model.

Getting Started

To get started with the LLM-lightshow project, follow these steps:

1. Clone the repository to your local machine.
2. Set up the ESP32 development environment (e.g., ESP-IDF).
3. Flash the embed-fft code to one ESP32 board for audio processing.
4. Flash the esp32-llm code to another ESP32 board for LLM inference.
5. Connect the two ESP32 boards via serial communication.
6. Run the system and enjoy the music-synced lightshow!

Requirements

• 2 ESP32S3 microcontrollers
• ESP-IDF development environment
• Python 3.x for training scripts
• DAC module for audio output
• SD card reader
• Speakers and LED lights for the lightshow
• Shift register (74HC595) for controlling multiple LEDs

Schematics

References

• Llama2.c Repository
• ESP32-LLM Repository

Demo Video