Fix/audio pipe 2 #261
Draft
Fix/audio pipe 2 #261
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling. - Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth. - Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.
- Updated `services.py` to allow service restart with an option to recreate containers, addressing WSL2 bind mount issues. - Added new chat configuration management functions in `system_controller.py` for loading, saving, and validating chat prompts. - Introduced `ChatSettings` component in the web UI for admin users to manage chat configurations easily. - Updated API service methods in `api.ts` to support chat configuration endpoints. - Integrated chat settings into the system management page for better accessibility.
…ion logging - Updated `start.sh` to improve shutdown handling by explicitly killing the backend process if running. - Modified `chat_service.py` to enhance logging for loading chat system prompts, providing clearer feedback on configuration usage. - Added a new `chat` field in `model_registry.py` for better chat service configuration management. - Updated vector store query parameters in `vector_stores.py` for improved clarity and functionality. - Enhanced the chat component in the web UI to conditionally auto-scroll based on message sending status.
…management - Introduced a new plugin architecture to allow for extensibility in the Chronicle application. - Added Home Assistant plugin for controlling devices via natural language commands triggered by wake words. - Implemented plugin configuration management endpoints in the API for loading, saving, and validating plugin settings. - Enhanced the web UI with a dedicated Plugins page for managing plugin configurations. - Updated Docker Compose files to include Tailscale integration for remote service access. - Refactored existing services to support plugin interactions during conversation and memory processing. - Improved error handling and logging for plugin initialization and execution processes.
- Updated .gitignore to include plugins.yml for security reasons. - Modified start.sh to allow passing additional arguments during service startup. - Refactored wizard.py to support new HF_TOKEN configuration prompts and improved handling of wake words in plugin settings. - Introduced a new setup_hf_token_if_needed function to streamline Hugging Face token management. - Enhanced the GitHub Actions workflow to create plugins.yml from a template, ensuring proper configuration setup. - Added detailed comments and documentation in the plugins.yml.template for better user guidance on Home Assistant integration.
…word processing - Added asynchronous Redis support in ClientManager for tracking client-user relationships. - Introduced `initialize_redis_for_client_manager` to set up Redis for cross-container mapping. - Updated `create_client_state` to use asynchronous tracking for client-user relationships. - Enhanced wake word processing in PluginRouter with normalization and command extraction. - Refactored DeepgramStreamingConsumer to utilize async Redis lookups for user ID retrieval. - Set TTL on Redis streams during client state cleanup for better resource management.
- Disabled the batch Deepgram worker in favor of the streaming worker to prevent race conditions. - Updated text normalization in wake word processing to replace punctuation with spaces, preserving word boundaries. - Enhanced regex pattern for wake word matching to allow optional punctuation and whitespace after the last part. - Improved logging in DeepgramStreamingConsumer for better visibility of message processing and error handling.
- Implemented retrieval of the original chat prompt before saving a custom prompt to ensure test isolation. - Added restoration of the original prompt after the test to prevent interference with subsequent tests. - Enhanced the test documentation for clarity on the purpose of these changes.
- Simplified test execution commands in CLAUDE.md and quickstart.md for better usability. - Added instructions for running tests from the project root and clarified the process for executing the complete Robot Framework test suite. - Introduced a new Docker service for the Deepgram streaming worker in docker-compose-test.yml to improve testing capabilities. - Updated system_admin_tests.robot to use a defined default prompt for restoration, enhancing test reliability and clarity.
- Updated `run-test.sh` and `run-robot-tests.sh` to improve cleanup processes, including handling permission issues with Docker. - Introduced a new function `mark_session_complete` in `session_controller.py` to ensure atomic updates for session completion status. - Refactored WebSocket and conversation job handling to utilize the new session completion function, enhancing reliability. - Updated `start-workers.sh` to enable the batch Deepgram worker alongside the streaming worker for improved transcription capabilities. - Enhanced test scripts to verify the status of Deepgram workers and ensure proper cleanup of test containers.
…ocess handling - Replaced the bash-based `start-workers.sh` script with a Python-based worker orchestrator for better process management and health monitoring. - Updated `docker-compose.yml` to configure the new orchestrator and adjust worker definitions, including the addition of audio persistence and stream workers. - Enhanced the Dockerfile to remove the old startup script and ensure the orchestrator is executable. - Introduced new modules for orchestrator configuration, health monitoring, process management, and worker registry to streamline worker lifecycle management. - Improved environment variable handling for worker configuration and health checks.
- Deleted the `run-test.sh` script, which was used for local test execution. - Updated Docker configurations to replace the `start-workers.sh` script with `worker_orchestrator.py` for improved worker management. - Enhanced health monitoring and process management in the orchestrator to ensure better reliability and logging. - Adjusted deployment configurations to reflect the new orchestrator setup.
- Introduced a new method `_handle_registration_loss` to manage RQ worker registration loss, replicating the behavior of the previous bash script. - Implemented a cooldown period to prevent frequent restarts during network issues. - Added logging for bulk restart actions and their outcomes to enhance monitoring and debugging capabilities. - Created a `_restart_all_rq_workers` method to facilitate the bulk restart of RQ workers, ensuring they re-register with Redis upon startup.
…tion - Introduced a new Test Event Plugin to log all plugin events to an SQLite database for integration testing. - Updated the plugin system to utilize event subscriptions instead of access levels, allowing for more flexible event handling. - Refactored the PluginRouter to dispatch events based on subscriptions, improving the event-driven architecture. - Enhanced Docker configurations to support development and testing environments with appropriate dependencies. - Added comprehensive integration tests to verify the functionality of the event dispatch system and plugin interactions. - Updated documentation and test configurations to reflect the new event-based plugin structure.
- Updated `docker-compose-test.yml` to include a test command for services, enabling a dedicated test mode. - Modified `start.sh` to support a `--test` flag, allowing the FastAPI backend to run with test-specific configurations. - Adjusted worker commands to utilize the `--group test` option in test mode for improved orchestration and management.
- Updated `run-robot-tests.sh` to enhance the verification of the Deepgram batch worker process, ensuring non-numeric characters are removed from the check. - Modified `plugin_tests.robot` to use a more explicit method for checking the length of subscriptions and added a skip condition for unavailable audio files. - Adjusted `plugin_event_tests.robot` to load the test audio file from a variable, improving test data management. - Refactored `plugin_keywords.robot` to utilize clearer length checks for subscriptions and event parts, enhancing readability and maintainability.
- Updated WebSocket endpoints to use a unified format with codec parameters (`/ws?codec=pcm` and `/ws?codec=opus`) for audio streaming, replacing the previous `/ws_pcm` and `/ws_omi` endpoints. - Enhanced documentation to reflect the new endpoint structure and clarify audio processing capabilities. - Removed deprecated audio cropping functionality and related configurations to streamline the audio processing workflow. - Updated various components and scripts to align with the new endpoint structure, ensuring consistent usage across the application.
- Updated `docker-compose-test.yml` to introduce low speech detection thresholds for testing, improving the accuracy of speech detection during tests. - Added new test-only API routes in `test_routes.py` for clearing and retrieving plugin events, ensuring a clean state between tests. - Refactored existing test scripts to utilize the new API endpoints for event management, enhancing test reliability and clarity. - Improved logging and error handling in various components to facilitate debugging during test execution. - Adjusted environment variable handling in test setup scripts to streamline configuration and improve flexibility.
…stence worker configuration - Introduced a comprehensive documentation file detailing the audio pipeline architecture, covering data flow, processing stages, and key components. - Enhanced the audio persistence worker setup by implementing multiple concurrent workers to improve audio processing efficiency. - Adjusted sleep intervals in the audio streaming persistence job for better responsiveness and event loop yielding. - Updated test script to run the full suite of integration tests from the specified directory, ensuring thorough testing coverage.
Contributor
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Comment |
Collaborator
Author
|
@coderabbitai review boss |
Contributor
|
@AnkushMalaker: I'll review the changes in this PR right away! ✅ Actions performedReview triggered.
|
- Introduced `setup-test-containers.sh` for streamlined startup of test containers, including health checks and environment variable loading. - Added `teardown-test-containers.sh` for simplified container shutdown, with options to remove volumes. - Enhanced user feedback with color-coded messages for better visibility during test setup and teardown processes.
|
| Metric | Count |
|---|---|
| ✅ Passed | 108 |
| ❌ Failed | 3 |
| 📊 Total | 111 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
- Adjusted worker count expectations in the Worker Count Validation Test to reflect an increase from 7 to 9 workers, accounting for additional audio persistence workers. - Enhanced the WebSocket Disconnect Conversation End Reason Test by adding steps to maintain audio streaming during disconnection, ensuring accurate simulation of network dropout scenarios. - Improved comments for clarity and added critical notes regarding inactivity timeout handling.
|
| Metric | Count |
|---|---|
| ✅ Passed | 109 |
| ❌ Failed | 2 |
| 📊 Total | 111 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.