Skip to content

Comments

feat(engine): add query result retention policy (#123)#125

Draft
shefeek-jinnah wants to merge 2 commits intomainfrom
shefeek/query_result_retention
Draft

feat(engine): add query result retention policy (#123)#125
shefeek-jinnah wants to merge 2 commits intomainfrom
shefeek/query_result_retention

Conversation

@shefeek-jinnah
Copy link
Contributor

@shefeek-jinnah shefeek-jinnah commented Feb 19, 2026

Summary

  • Add background worker that periodically deletes expired query results (ready/failed) older than a configurable retention period
  • Schedule associated Parquet files for deletion when results are cleaned up
  • Configurable via result_retention_days (default: 7, set to 0 to disable)

Details

Complements the existing stale result cleanup worker (which marks stuck pending/processing results as failed) with a second phase that garbage-collects terminal results after a retention window.

  • Config: RUNTIMEDB_ENGINE__RESULT_RETENTION_DAYS env var or engine.result_retention_days in config
  • Worker: Runs every hour, deletes terminal results older than the cutoff and schedules their Parquet directories for deletion
  • Catalog: delete_expired_results implemented for Postgres (DELETE...RETURNING) and SQLite (SELECT + DELETE)
  • Lifecycle: Worker handle stored on RuntimeEngine, gracefully shut down during engine termination

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant