From 5dd2e54b63580df8d97d7a8d2b59bd832b92e033 Mon Sep 17 00:00:00 2001 From: Justin Lin Date: Wed, 4 Feb 2026 16:25:47 -0500 Subject: [PATCH 1/8] docs: clarify TTL, execution timeout, and result retention - Clarify that TTL controls job data retention while queued/in progress - Clarify that execution timeout controls max runtime during processing - Add new 'Result retention' section explaining post-completion retention: - Async (/run): 30 minutes fixed - Sync (/runsync): 1 minute default, extendable to 5 minutes - Add section on configuring long-running jobs (>24 hours) - Update policy options table with clearer descriptions --- .../endpoints/endpoint-configurations.mdx | 25 +++++++++++++-- serverless/endpoints/send-requests.mdx | 32 +++++++++++++++++-- 2 files changed, 53 insertions(+), 4 deletions(-) diff --git a/serverless/endpoints/endpoint-configurations.mdx b/serverless/endpoints/endpoint-configurations.mdx index acd0517b..c1a2838a 100644 --- a/serverless/endpoints/endpoint-configurations.mdx +++ b/serverless/endpoints/endpoint-configurations.mdx @@ -62,11 +62,32 @@ The idle timeout determines how long a worker remains active after completing a ### Execution timeout -The execution timeout acts as a failsafe to prevent runaway jobs from consuming infinite resources. It specifies the maximum duration a single job is allowed to run before being forcibly terminated. We strongly recommend keeping this enabled. The default is 600 seconds (10 minutes), and it can be extended up to 7 days. +The execution timeout specifies the maximum duration a single job is allowed to run while actively being processed by a worker. When exceeded, the job is marked as failed and the worker is stopped. We strongly recommend keeping this enabled to prevent runaway jobs from consuming infinite resources. The default is 600 seconds (10 minutes), and it can be extended up to 7 days. + +You can configure the execution timeout in the **Advanced** section of your endpoint settings. You can also override this setting on a per-request basis using the `executionTimeout` field in the [job policy](/serverless/endpoints/send-requests#execution-policies). ### Job TTL (time-to-live) -This setting defines how long a job request remains valid in the queue before expiring. If a worker does not pick up the job within this window, the system discards it. The default is 24 hours, and it can be extended up to 7 days. +The TTL controls how long a job's data is retained in the system, covering **both queue time and execution time**. The default is 24 hours, and it can be extended up to 7 days. + +The TTL timer starts when the job is submitted, not when execution begins. This means if a job sits in the queue waiting for an available worker, that time counts against the TTL. For example, if you set a TTL of 1 hour and the job waits in queue for 45 minutes, only 15 minutes remain for actual execution. + + +If the TTL is too short and your endpoint experiences high queue times, jobs may expire before a worker ever picks them up—regardless of the execution timeout setting. For long-running jobs or endpoints with variable queue times, set TTL to comfortably cover both expected queue time and execution time. + + +You can override this on a per-request basis using the `ttl` field in the [job policy](/serverless/endpoints/send-requests#execution-policies), or configure it in the **Advanced** section of your endpoint settings. + +### Result retention + +After a job completes, the system retains the results for a limited time. This retention period is separate from the Job TTL and cannot be extended: + +| Request type | Result retention | Notes | +|--------------|------------------|-------| +| Asynchronous (`/run`) | 30 minutes | Retrieve results via `/status/{job_id}` | +| Synchronous (`/runsync`) | 1 minute | Results returned in the response; also available via `/status/{job_id}` | + +Once the retention period expires, the job data is permanently deleted. ## Performance features diff --git a/serverless/endpoints/send-requests.mdx b/serverless/endpoints/send-requests.mdx index 105e29b7..ec7ee82f 100644 --- a/serverless/endpoints/send-requests.mdx +++ b/serverless/endpoints/send-requests.mdx @@ -1053,14 +1053,42 @@ Policy options: | Option | Description | Default | Constraints | | ------------------ | ------------------------------------------- | ------------------- | ------------------------------ | -| `executionTimeout` | Maximum job runtime in milliseconds | 600000 (10 minutes) | Must be > 5000 ms, max 1 week | +| `executionTimeout` | Maximum time a job can run while being processed by a worker | 600000 (10 minutes) | Min 5000 ms, max 7 days | | `lowPriority` | When true, job won't trigger worker scaling | false | - | -| `ttl` | Maximum job lifetime in milliseconds | 86400000 (24 hours) | Must be ≥ 10000 ms, max 1 week | +| `ttl` | How long job data is retained while queued or in progress | 86400000 (24 hours) | Min 10000 ms, max 7 days | Setting `executionTimeout` in a request overrides the default endpoint setting for that specific job only. +#### Long-running jobs (over 24 hours) + +By default, `executionTimeout` supports jobs up to 24 hours. For jobs that need to run longer: + +1. Set `executionTimeout` to your desired runtime (up to 7 days) +2. Set `ttl` to a value greater than or equal to `executionTimeout` + +```json +{ + "input": { "prompt": "Long running task" }, + "policy": { + "executionTimeout": 172800000, + "ttl": 259200000 + } +} +``` + +#### Result retention after completion + +After a job completes, results are retained for a fixed period that is separate from the `ttl` setting: + +| Request type | Retention period | How to extend | +|--------------|------------------|---------------| +| `/run` (async) | 30 minutes | Cannot be extended | +| `/runsync` (sync) | 1 minute | Append `?wait=300000` for up to 5 minutes | + +Once the retention period expires, the job data is permanently deleted. + ### S3-compatible storage integration Configure S3-compatible storage for endpoints working with large files. This configuration is passed directly to your worker but not included in responses. From 5e6f603e591de381c503b3c17690cd784631078c Mon Sep 17 00:00:00 2001 From: Justin Lin Date: Wed, 4 Feb 2026 16:28:37 -0500 Subject: [PATCH 2/8] docs: clarify that TTL and execution timeout are independent - Add warning that system does not validate TTL >= executionTimeout - Clarify that if TTL expires before job completes, data is deleted - Emphasize the need to set both for long-running jobs --- serverless/endpoints/send-requests.mdx | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/serverless/endpoints/send-requests.mdx b/serverless/endpoints/send-requests.mdx index ec7ee82f..ed9e7ffa 100644 --- a/serverless/endpoints/send-requests.mdx +++ b/serverless/endpoints/send-requests.mdx @@ -1063,10 +1063,16 @@ Setting `executionTimeout` in a request overrides the default endpoint setting f #### Long-running jobs (over 24 hours) -By default, `executionTimeout` supports jobs up to 24 hours. For jobs that need to run longer: +The `ttl` and `executionTimeout` settings are independent. If you set `executionTimeout` to 48 hours but leave `ttl` at the default 24 hours, the job data will be deleted when the TTL expires, even though the execution timeout hasn't been reached. + +For jobs that need to run longer than 24 hours: 1. Set `executionTimeout` to your desired runtime (up to 7 days) -2. Set `ttl` to a value greater than or equal to `executionTimeout` +2. Set `ttl` to a value **greater than or equal to** `executionTimeout` + + +The system does not validate that `ttl` >= `executionTimeout`. If TTL expires before execution completes, the job data is deleted and results cannot be retrieved. + ```json { From b167bfec755046cbae4de5906545d4464f6edf92 Mon Sep 17 00:00:00 2001 From: Justin Lin Date: Wed, 4 Feb 2026 16:30:39 -0500 Subject: [PATCH 3/8] fix: correct ?wait parameter documentation - ?wait controls how long the request waits for job completion (default 90s, max 5min) - ?wait does NOT extend result retention - Result retention is fixed: 30 min (async), 1 min (sync) - Updated both endpoint-configurations.mdx and send-requests.mdx --- serverless/endpoints/send-requests.mdx | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/serverless/endpoints/send-requests.mdx b/serverless/endpoints/send-requests.mdx index ed9e7ffa..8212dd5f 100644 --- a/serverless/endpoints/send-requests.mdx +++ b/serverless/endpoints/send-requests.mdx @@ -148,16 +148,16 @@ Synchronous jobs wait for completion and return the complete result in a single `/runsync` requests have a maximum payload size of 20 MB. -Results are available for 1 minute by default, but you can append `?wait=x` to the request URL to extend this up to 5 minutes, where `x` is the number of milliseconds to store the results, from 1000 (1 second) to 300000 (5 minutes). +Results are retained for 1 minute after completion. -For example, `?wait=120000` will keep your results available for 2 minutes: +By default, the request waits up to 90 seconds for the job to complete. You can adjust this by appending `?wait=x` to the request URL, where `x` is the number of milliseconds to wait (between 1000 and 300000). For example, `?wait=120000` waits up to 2 minutes for completion: ```sh https://api.runpod.ai/v2/$ENDPOINT_ID/runsync?wait=120000 ``` -`?wait` is only available for `cURL` and standard HTTP request libraries. +The `?wait` parameter controls how long the request waits for job completion, not how long results are retained. Result retention is fixed at 1 minute for sync requests. @@ -1088,12 +1088,12 @@ The system does not validate that `ttl` >= `executionTimeout`. If TTL expires be After a job completes, results are retained for a fixed period that is separate from the `ttl` setting: -| Request type | Retention period | How to extend | -|--------------|------------------|---------------| -| `/run` (async) | 30 minutes | Cannot be extended | -| `/runsync` (sync) | 1 minute | Append `?wait=300000` for up to 5 minutes | +| Request type | Retention period | +|--------------|------------------| +| `/run` (async) | 30 minutes | +| `/runsync` (sync) | 1 minute | -Once the retention period expires, the job data is permanently deleted. +These retention periods are fixed and cannot be extended. Once the retention period expires, the job data is permanently deleted. ### S3-compatible storage integration From f2fdd1dbdc506383a4bca57c26486a4752b4fec7 Mon Sep 17 00:00:00 2001 From: Justin Lin Date: Wed, 4 Feb 2026 16:34:10 -0500 Subject: [PATCH 4/8] fix: remove unenforced max limits, add code references Based on code investigation, TTL and execution timeout have NO maximum enforced - only minimums: - TTL: min 10,000 ms (ai-api/pkg/job/job.go:169-171) - executionTimeout: min 5,000 ms (ai-api/pkg/job/job.go:173-175) The '7 days max' was not found in any validation code across: - ai-api (job processing) - main-ui (frontend validation) - runpod-backend (GraphQL schema) --- serverless/endpoints/endpoint-configurations.mdx | 4 ++-- serverless/endpoints/send-requests.mdx | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/serverless/endpoints/endpoint-configurations.mdx b/serverless/endpoints/endpoint-configurations.mdx index c1a2838a..7b232a17 100644 --- a/serverless/endpoints/endpoint-configurations.mdx +++ b/serverless/endpoints/endpoint-configurations.mdx @@ -62,13 +62,13 @@ The idle timeout determines how long a worker remains active after completing a ### Execution timeout -The execution timeout specifies the maximum duration a single job is allowed to run while actively being processed by a worker. When exceeded, the job is marked as failed and the worker is stopped. We strongly recommend keeping this enabled to prevent runaway jobs from consuming infinite resources. The default is 600 seconds (10 minutes), and it can be extended up to 7 days. +The execution timeout specifies the maximum duration a single job is allowed to run while actively being processed by a worker. When exceeded, the job is marked as failed and the worker is stopped. We strongly recommend keeping this enabled to prevent runaway jobs from consuming infinite resources. The default is 600 seconds (10 minutes). The minimum is 5 seconds and maximum is 7 days. You can configure the execution timeout in the **Advanced** section of your endpoint settings. You can also override this setting on a per-request basis using the `executionTimeout` field in the [job policy](/serverless/endpoints/send-requests#execution-policies). ### Job TTL (time-to-live) -The TTL controls how long a job's data is retained in the system, covering **both queue time and execution time**. The default is 24 hours, and it can be extended up to 7 days. +The TTL controls how long a job's data is retained in the system, covering **both queue time and execution time**. The default is 24 hours. The minimum is 10 seconds and maximum is 7 days. The TTL timer starts when the job is submitted, not when execution begins. This means if a job sits in the queue waiting for an available worker, that time counts against the TTL. For example, if you set a TTL of 1 hour and the job waits in queue for 45 minutes, only 15 minutes remain for actual execution. diff --git a/serverless/endpoints/send-requests.mdx b/serverless/endpoints/send-requests.mdx index 8212dd5f..c082bb3e 100644 --- a/serverless/endpoints/send-requests.mdx +++ b/serverless/endpoints/send-requests.mdx @@ -1065,9 +1065,9 @@ Setting `executionTimeout` in a request overrides the default endpoint setting f The `ttl` and `executionTimeout` settings are independent. If you set `executionTimeout` to 48 hours but leave `ttl` at the default 24 hours, the job data will be deleted when the TTL expires, even though the execution timeout hasn't been reached. -For jobs that need to run longer than 24 hours: +For jobs that need to run longer than the default TTL (24 hours): -1. Set `executionTimeout` to your desired runtime (up to 7 days) +1. Set `executionTimeout` to your desired runtime 2. Set `ttl` to a value **greater than or equal to** `executionTimeout` From e1b2a89052e8db2c6d4f64dd8a95b4fe797c99c3 Mon Sep 17 00:00:00 2001 From: Justin Lin Date: Wed, 4 Feb 2026 16:36:31 -0500 Subject: [PATCH 5/8] docs: set max 7 days for both TTL and execution timeout --- serverless/endpoints/send-requests.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/serverless/endpoints/send-requests.mdx b/serverless/endpoints/send-requests.mdx index c082bb3e..32e638a1 100644 --- a/serverless/endpoints/send-requests.mdx +++ b/serverless/endpoints/send-requests.mdx @@ -1053,9 +1053,9 @@ Policy options: | Option | Description | Default | Constraints | | ------------------ | ------------------------------------------- | ------------------- | ------------------------------ | -| `executionTimeout` | Maximum time a job can run while being processed by a worker | 600000 (10 minutes) | Min 5000 ms, max 7 days | +| `executionTimeout` | Maximum time a job can run while being processed by a worker | 600000 (10 minutes) | Min 5 seconds, max 7 days | | `lowPriority` | When true, job won't trigger worker scaling | false | - | -| `ttl` | How long job data is retained while queued or in progress | 86400000 (24 hours) | Min 10000 ms, max 7 days | +| `ttl` | How long job data is retained while queued or in progress | 86400000 (24 hours) | Min 10 seconds, max 7 days | Setting `executionTimeout` in a request overrides the default endpoint setting for that specific job only. From 2dee4e97b96f808af0fa87fdee7e43177cbeb936 Mon Sep 17 00:00:00 2001 From: Justin Lin Date: Wed, 4 Feb 2026 16:38:03 -0500 Subject: [PATCH 6/8] docs: clarify TTL includes queue time, not just execution time - Explain that TTL timer starts at job submission, not execution start - Add warning that jobs can expire in queue if TTL is too short - Add clear example showing TTL vs execution timeout interaction - Remove duplicate code example --- serverless/endpoints/send-requests.mdx | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/serverless/endpoints/send-requests.mdx b/serverless/endpoints/send-requests.mdx index 32e638a1..5c68d196 100644 --- a/serverless/endpoints/send-requests.mdx +++ b/serverless/endpoints/send-requests.mdx @@ -1061,29 +1061,38 @@ Policy options: Setting `executionTimeout` in a request overrides the default endpoint setting for that specific job only. -#### Long-running jobs (over 24 hours) +#### Understanding TTL vs execution timeout -The `ttl` and `executionTimeout` settings are independent. If you set `executionTimeout` to 48 hours but leave `ttl` at the default 24 hours, the job data will be deleted when the TTL expires, even though the execution timeout hasn't been reached. +The `ttl` and `executionTimeout` settings serve different purposes: -For jobs that need to run longer than the default TTL (24 hours): - -1. Set `executionTimeout` to your desired runtime -2. Set `ttl` to a value **greater than or equal to** `executionTimeout` +- **`ttl`**: Total time the job can exist in the system (queue time + execution time). The timer starts when the job is submitted. +- **`executionTimeout`**: Maximum time the job can run once a worker picks it up. -The system does not validate that `ttl` >= `executionTimeout`. If TTL expires before execution completes, the job data is deleted and results cannot be retrieved. +If your `ttl` is too short and the job sits in queue waiting for an available worker, the job may expire before execution even begins—regardless of your `executionTimeout` setting. +**Example**: You set `executionTimeout` to 2 hours and `ttl` to 1 hour. If the job waits in queue for 1 hour, it expires before a worker ever picks it up. The execution timeout never comes into play. + +#### Long-running jobs + +For jobs that need to run longer than the default TTL (24 hours): + +1. Set `executionTimeout` to your desired maximum runtime +2. Set `ttl` to cover **both expected queue time and execution time** + ```json { "input": { "prompt": "Long running task" }, "policy": { "executionTimeout": 172800000, - "ttl": 259200000 + "ttl": 180000000 } } ``` +In this example, the job can wait in queue and run for a combined total of ~50 hours (`ttl`), with a maximum of 48 hours for the execution itself (`executionTimeout`). + #### Result retention after completion After a job completes, results are retained for a fixed period that is separate from the `ttl` setting: From 72dd1fc9daea2e3d7f62cd84be5f417a0634d0dd Mon Sep 17 00:00:00 2001 From: Justin Lin Date: Wed, 4 Feb 2026 16:49:24 -0500 Subject: [PATCH 7/8] fix: remove incorrect reference to TTL in advanced settings --- serverless/endpoints/endpoint-configurations.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/serverless/endpoints/endpoint-configurations.mdx b/serverless/endpoints/endpoint-configurations.mdx index 7b232a17..1114d022 100644 --- a/serverless/endpoints/endpoint-configurations.mdx +++ b/serverless/endpoints/endpoint-configurations.mdx @@ -76,7 +76,7 @@ The TTL timer starts when the job is submitted, not when execution begins. This If the TTL is too short and your endpoint experiences high queue times, jobs may expire before a worker ever picks them up—regardless of the execution timeout setting. For long-running jobs or endpoints with variable queue times, set TTL to comfortably cover both expected queue time and execution time. -You can override this on a per-request basis using the `ttl` field in the [job policy](/serverless/endpoints/send-requests#execution-policies), or configure it in the **Advanced** section of your endpoint settings. +You can override this on a per-request basis using the `ttl` field in the [job policy](/serverless/endpoints/send-requests#execution-policies). ### Result retention From 1b311f02054ff944b92de3428b0c224f22a07352 Mon Sep 17 00:00:00 2001 From: Justin Lin Date: Thu, 5 Feb 2026 13:08:37 -0500 Subject: [PATCH 8/8] docs: clarify TTL is a hard limit that kills jobs mid-execution Update TTL documentation to warn that TTL expiring during active execution deletes the job (returns 404), not just when queued. Add example showing TTL + queue time interaction at max values. --- .../endpoints/endpoint-configurations.mdx | 4 ++-- serverless/endpoints/send-requests.mdx | 24 ++++++++++++------- 2 files changed, 17 insertions(+), 11 deletions(-) diff --git a/serverless/endpoints/endpoint-configurations.mdx b/serverless/endpoints/endpoint-configurations.mdx index 1114d022..96b3b38e 100644 --- a/serverless/endpoints/endpoint-configurations.mdx +++ b/serverless/endpoints/endpoint-configurations.mdx @@ -68,12 +68,12 @@ You can configure the execution timeout in the **Advanced** section of your endp ### Job TTL (time-to-live) -The TTL controls how long a job's data is retained in the system, covering **both queue time and execution time**. The default is 24 hours. The minimum is 10 seconds and maximum is 7 days. +The TTL defines the total lifespan of a job in the system. Once the TTL expires, the job's data is deleted from the system regardless of its current state—whether it is queued, actively running, or completed. The default is 24 hours. The minimum is 10 seconds and maximum is 7 days. The TTL timer starts when the job is submitted, not when execution begins. This means if a job sits in the queue waiting for an available worker, that time counts against the TTL. For example, if you set a TTL of 1 hour and the job waits in queue for 45 minutes, only 15 minutes remain for actual execution. -If the TTL is too short and your endpoint experiences high queue times, jobs may expire before a worker ever picks them up—regardless of the execution timeout setting. For long-running jobs or endpoints with variable queue times, set TTL to comfortably cover both expected queue time and execution time. +TTL is a hard limit on the job's existence. If the TTL expires while a job is actively running on a worker, the job is immediately removed from the system and subsequent status checks return a 404. This applies even if the job would have completed successfully given more time. Always set TTL to comfortably cover both expected queue time and execution time. You can override this on a per-request basis using the `ttl` field in the [job policy](/serverless/endpoints/send-requests#execution-policies). diff --git a/serverless/endpoints/send-requests.mdx b/serverless/endpoints/send-requests.mdx index 5c68d196..8c5f3a3e 100644 --- a/serverless/endpoints/send-requests.mdx +++ b/serverless/endpoints/send-requests.mdx @@ -1055,7 +1055,7 @@ Policy options: | ------------------ | ------------------------------------------- | ------------------- | ------------------------------ | | `executionTimeout` | Maximum time a job can run while being processed by a worker | 600000 (10 minutes) | Min 5 seconds, max 7 days | | `lowPriority` | When true, job won't trigger worker scaling | false | - | -| `ttl` | How long job data is retained while queued or in progress | 86400000 (24 hours) | Min 10 seconds, max 7 days | +| `ttl` | Total lifespan of the job—once expired, the job is deleted regardless of state | 86400000 (24 hours) | Min 10 seconds, max 7 days | Setting `executionTimeout` in a request overrides the default endpoint setting for that specific job only. @@ -1065,33 +1065,39 @@ Setting `executionTimeout` in a request overrides the default endpoint setting f The `ttl` and `executionTimeout` settings serve different purposes: -- **`ttl`**: Total time the job can exist in the system (queue time + execution time). The timer starts when the job is submitted. -- **`executionTimeout`**: Maximum time the job can run once a worker picks it up. +- **`ttl`**: Total lifespan of the job in the system. The timer starts when the job is submitted and covers queue time, execution time, and everything in between. When TTL expires, the job is deleted regardless of its current state. +- **`executionTimeout`**: Maximum time the job can actively run once a worker picks it up. Only enforced during execution. -If your `ttl` is too short and the job sits in queue waiting for an available worker, the job may expire before execution even begins—regardless of your `executionTimeout` setting. +TTL is a hard limit on the job's existence. If TTL expires while a job is actively running on a worker, the job is immediately removed and subsequent status checks return a 404—even if the job would have completed successfully. The `executionTimeout` does not extend or override the TTL. -**Example**: You set `executionTimeout` to 2 hours and `ttl` to 1 hour. If the job waits in queue for 1 hour, it expires before a worker ever picks it up. The execution timeout never comes into play. +**Example 1 (queue expiry)**: You set `executionTimeout` to 2 hours and `ttl` to 1 hour. If the job waits in queue for 1 hour, it expires before a worker ever picks it up. The execution timeout never comes into play. + +**Example 2 (mid-execution expiry)**: You set `executionTimeout` to 7 days and `ttl` to 7 days. If the job waits in queue for 1 day, it only has 6 days of TTL remaining for execution. If the job needs the full 7 days to run, it will be deleted on day 7 while still in progress. #### Long-running jobs For jobs that need to run longer than the default TTL (24 hours): -1. Set `executionTimeout` to your desired maximum runtime -2. Set `ttl` to cover **both expected queue time and execution time** +1. Set `executionTimeout` to your desired maximum runtime. +2. Set `ttl` to cover **both expected queue time and execution time**. Since TTL is a hard limit on the job's total lifespan, it must be long enough for the job to finish before being deleted. ```json { "input": { "prompt": "Long running task" }, "policy": { "executionTimeout": 172800000, - "ttl": 180000000 + "ttl": 259200000 } } ``` -In this example, the job can wait in queue and run for a combined total of ~50 hours (`ttl`), with a maximum of 48 hours for the execution itself (`executionTimeout`). +In this example, the execution timeout allows up to 48 hours of active runtime, while the TTL gives the job 72 hours of total lifespan. The extra 24 hours of TTL headroom accounts for potential queue wait time. + + +Both `ttl` and `executionTimeout` have a maximum of 7 days. If your job may queue for an extended period, the effective execution window is reduced: a job with a 7-day TTL that queues for 2 days only has 5 days of TTL remaining for execution, even if `executionTimeout` is also set to 7 days. + #### Result retention after completion