Fix Radarr rename, GenStage issues, and update dependencies #10
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The Radarr RenameFiles API command requires both movieId and files parameters in the JSON payload. Previously only files was sent, causing the rename operation to fail. This matches the Sonarr implementation which correctly includes seriesId. Verified against Radarr source code (RenameFilesCommand class).
Fixed 'GenStage producer has discarded events from buffer' warnings by correcting the producer pattern in all three Broadway pipelines. **Root Cause:** Producers were emitting events in handle_info(:poll) without demand from downstream consumers. With max_demand=1 and concurrency=1, when a processor is busy (e.g., encoding in progress), it can't accept new events. The periodic :poll kept pushing events anyway, causing GenStage to discard them. **Solution:** Producers now only emit events in response to demand via handle_demand. The :poll handler only schedules the next poll - it doesn't push events. This follows the correct GenStage pattern: - Demand arrives → check for work → return events - Periodic poll → just reschedule → return empty list **Files Changed:** - lib/reencodarr/encoder/broadway/producer.ex - lib/reencodarr/crf_searcher/broadway/producer.ex - lib/reencodarr/analyzer/broadway/producer.ex All tests pass.
Videos were showing in 'next in queue' even when currently being encoded because the encoder wasn't transitioning videos to :encoding state. **Root Cause:** The encoder Broadway pipeline was broadcasting :encoding_started events but never actually transitioning the video's state from :crf_searched to :encoding. This caused the video to remain in the encoding queue query results. **Solution:** Added state transition to :encoding when process_vmaf_encoding starts, matching the pattern used in CRF searcher which transitions to :crf_searching. **Impact:** - Encoding queue now correctly excludes currently encoding videos - 'Next in queue' shows actual next video, not current one - State machine flow is complete: crf_searched → encoding → encoded This fixes the issue where the dashboard showed currently processing items as 'next in queue' for the encoder. Also added VideoStateMachine and Repo aliases to satisfy credo.
Updated 17 packages: - credo 1.7.12 → 1.7.13 - dialyxir 1.4.6 → 1.4.7 - ecto 3.13.2 → 3.13.4 - ecto_sqlite3 0.21.0 → 0.22.0 - erlex 0.2.7 → 0.2.8 - expo 1.1.0 → 1.1.1 - exqlite 0.33.0 → 0.33.1 - gettext 1.0.0 → 1.0.1 - lazy_html 0.1.7 → 0.1.8 - meck 1.0.0 → 1.1.0 - phoenix_html 4.2.1 → 4.3.0 - phoenix_live_view 1.1.11 → 1.1.17 - phoenix_pubsub 2.1.3 → 2.2.0 - swoosh 1.19.5 → 1.19.8 - tailwind 0.4.0 → 0.4.1 - thousand_island 1.4.1 → 1.4.2 - websock_adapter 0.5.8 → 0.5.9
The previous fix to prevent GenStage buffer overflow inadvertently broke the wake-up mechanism for Broadway pipelines. When Broadway finished all work and became idle, the :poll handler would see new work but couldn't emit events (to avoid buffer overflow). Solution: Track pending_demand in producer state and emit events during :poll if there's pending demand. This allows: - Broadway to express demand when it's ready - Producers to fulfill that demand immediately when work becomes available - No buffer overflow from pushing events without demand Changes: - Add pending_demand: 0 to initial state in all three producers - Accumulate demand in handle_demand/2 - Dispatch events if pending_demand > 0 during :poll - Track remaining demand after dispatching events
There was a problem hiding this comment.
Pull Request Overview
This pull request updates dependencies and refactors Broadway producer state management to properly track pending demand across all three processing pipelines (Analyzer, CRF Searcher, and Encoder).
- Updates Elixir from 1.14 to 1.19 and upgrades 20+ Hex dependencies to their latest versions
- Refactors all three Broadway producers to track
pending_demandin state, preventing lost work when demand arrives during polling intervals - Adds explicit state transition to
encodingstate at the start of encoding to ensure videos are marked as in-progress immediately
Reviewed Changes
Copilot reviewed 10 out of 12 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| mix.exs | Bumps Elixir version requirement from ~> 1.14 to ~> 1.19 |
| mix.lock | Updates 20+ dependencies including credo, ecto, phoenix_live_view, and others |
| flake.nix | Switches from custom Elixir 1.19.0-rc.0 build to stable elixir_1_19, removes redundant pinentry package |
| flake.lock | Updates nixpkgs lockfile to newer commit |
| lib/reencodarr/analyzer/broadway/producer.ex | Refactors to track pending_demand in state and consolidates dispatch logic |
| lib/reencodarr/crf_searcher/broadway/producer.ex | Refactors to track pending_demand in state and consolidates dispatch logic |
| lib/reencodarr/encoder/broadway/producer.ex | Refactors to track pending_demand in state and consolidates dispatch logic |
| lib/reencodarr/encoder/broadway.ex | Adds explicit state transition to encoding when processing starts |
| lib/reencodarr/services/radarr.ex | Adds missing movieId field to RenameFiles command payload |
| lib/reencodarr/sync.ex | Reorganizes alias declarations for better readability |
| lib/reencodarr/media/video_queries.ex | Adds documentation clarifying exclusion of crf_searching videos |
| test/reencodarr/media/video_upsert_test.exs | Consolidates alias declarations into single grouped statement |
Switch from Alpine container to erlef/setup-beam for better version control and add comprehensive caching to dramatically speed up CI runs. Changes: - Use erlef/setup-beam@v1 for Elixir 1.19.2 and OTP 28.0 (strict versions) - Separate cache for Rust toolchain and ab-av1 binary - Cache ab-av1 binary independently (takes ~5-10 min to compile) - Cache Mix deps and _build together with OTP/Elixir version keys - Add PLT cache for future Dialyzer support - Skip deps install/compile on cache hit - Switch from Alpine to ubuntu-latest runner Cache strategy: - Rust toolchain: Cached separately from binaries - ab-av1 binary: Cached independently, only rebuild when cache misses - Mix: Include OTP/Elixir versions in key to invalidate on version changes - Build: Hash lib/ and config/ files to detect code changes This should reduce CI time from ~15+ minutes to ~2-3 minutes on cache hits.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes several issues and updates the project dependencies:
Changes
Radarr Integration (
cb60806)rename_files/2to includemovieIdparameter in API payloadmovieIdandfilesfor RenameFiles commandGenStage Producer Fixes (
d9c23cc,f43fa70):pollhandler (was causing buffer overflow)pending_demandtracking to allow poll to wake up Broadway:pollState Machine Improvements (
92535cb):encodingstate when encoding starts:crf_searchingstate when CRF search startsDependency Updates (
c47f4be)Updated 17 packages:
Elixir/OTP Update
Testing
Technical Details
The stuck pipeline issue was subtle:
:pollwithout demand → buffer overflow:poll→ prevented buffer overflow but broke wake-up:pollonly if demand exists → works perfectlyBroadway now properly wakes up when: