Skip to content

Support per-message timeouts in RunTaskWithMultiprocessing #505

@trevor-e

Description

@trevor-e

(moving our Slack convo here)

We had a situation where one message took a very long time to process and essentially got stuck in our queue. To solve for this, we would like the ability to impose a hard timeout for message processing. If a message fails to process within the timeout period, it should be sent to the configured DLQ.

I tried solving this a few different ways in our subclass of RunTaskWithMultiprocessing as well as a custom processing strategy but they were all pretty hacky/duplicative and it felt best to support directly within Arroyo. Also worth noting, the work our message processing is doing (heavy C++ use) completely hogs the GIL from what I can tell, so traditional signals and thread interrupts don't seem to work. From what I understand our use-case of Arroyo is unique (low-volume long-running jobs) but being able to set a timeout seems useful for others as well.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions