-
-
Notifications
You must be signed in to change notification settings - Fork 4
Open
Description
rdkafka has two options related to auto-committing:
enable.auto.commit-- commit staged offsets automaticallyenable.auto.commit.store-- auto-stage offsets that have been read
we could potentially get rid of the internal offset stage that arroyo keeps track of, by storing the committed offsets in rdkafka and using enable.auto.commit to write them out periodically. this way we would not have to explicitly debounce commits.
arroyo still has a lot of code around retrying commits, but theoretically we could do away with that and only explicitly commit (with retries and everything else) in the revocation callbacks.
i'm not confident this would make the code simpler as it feels like we'd still need to store offsets in arroyo for other purposes + now we're interacting with rdkafka committing in two ways (auto vs in revocation)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels