Skip to content

Investigate whether we can use auto-commit in arroyo #414

@untitaker

Description

@untitaker

rdkafka has two options related to auto-committing:

  • enable.auto.commit -- commit staged offsets automatically
  • enable.auto.commit.store -- auto-stage offsets that have been read

we could potentially get rid of the internal offset stage that arroyo keeps track of, by storing the committed offsets in rdkafka and using enable.auto.commit to write them out periodically. this way we would not have to explicitly debounce commits.

arroyo still has a lot of code around retrying commits, but theoretically we could do away with that and only explicitly commit (with retries and everything else) in the revocation callbacks.

i'm not confident this would make the code simpler as it feels like we'd still need to store offsets in arroyo for other purposes + now we're interacting with rdkafka committing in two ways (auto vs in revocation)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions