Skip to content

feat: DailyPaper SSE streaming, LLM filter, email push, persisted config #24

@jerry609

Description

@jerry609

Summary

The /api/research/paperscool/daily endpoint times out (502 after 5 min) when LLM analysis or Judge is enabled because the frontend proxy buffers the entire response. This issue tracks converting the DailyPaper pipeline to SSE streaming with real-time progress, adding post-judge paper filtering, email push notifications, and persisting all config settings.

Changes

Backend

  • SSE streaming for /daily: When LLM or Judge is enabled, the endpoint returns text/event-stream with per-phase progress events (search → build → LLM → judge → filter → save → notify → result). Fast path (no LLM/Judge) remains sync JSON.
  • Post-judge paper filtering: After judge scoring, papers with recommendation "skip" or "skim" are removed. Only "must_read" and "worth_reading" papers are kept. Complete judge logs are preserved in the filter.log field.
  • Email push: New notify_email_to field on DailyPaperRequest allows the frontend to pass recipient email directly, overriding env config.
  • Judge cap raised: judge_max_items_per_query limit raised from 20 to 200.

Frontend

  • SSE-aware proxy: daily/route.ts detects upstream content-type and pipes SSE streams through without buffering.
  • StreamProgressCard: Real-time progress display showing current phase, progress bar, elapsed time, and detailed log (including per-paper judge scores and filter decisions).
  • runDailyPaperStream(): Replaces sync runDailyPaper() — reads SSE events and progressively updates the store (papers appear one-by-one as judge scores arrive).
  • Persisted config: All feature toggles (LLM, Judge, sources, etc.) are persisted in zustand store — no more re-checking on page refresh. All features default to enabled.
  • Email notification UI: Email address input + enable checkbox in settings sheet.

Bug fixes

  • Fixed @dataclass inheritance in errors.py (subclass defaults not applied)
  • Fixed ensure_execution_result treating raw dicts as result dicts in executable.py

Testing

  • 215 unit tests pass, 0 failures
  • 3 new end-to-end tests for the SSE pipeline (filter, full pipeline, sync fallback)
  • Next.js build passes

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions