Skip to content

Conversation

@TheSafo
Copy link
Contributor

@TheSafo TheSafo commented Dec 30, 2025

What does this PR do?

Enables configuring byte-based throttlers in generators to be based per block (e.g. 10 blocks per second instead of 100 MB/s).

Impl. details:

  • Adds a wrapper for generators around lading throttle that's based around throttling blocks at a time (which aligns with how the throttler is currently used)
  • Adds some config tests to verify that both the new format and old format is still supported

Motivation

When using payload generators where blocks are intentionally not the same size/close to maximum bytes, it can be helpful to reason about throughput per block. This allows us to do things like:

  • Define a payload where each block is 1 second worth of a system's logs
  • Set the throttle rate at 5 blocks per second
    to simulate generating logs at 5x standard playback speed.

Related issues

A follow up pr (#1683) adds some flushing logic to BufWriter based generators that enables more regular flushing. A separate PR will introduce a new payload format to utilize block-based throttling.

Copy link
Contributor Author

TheSafo commented Dec 30, 2025

@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch 3 times, most recently from c4199fc to 6ed7dca Compare December 31, 2025 16:20
@TheSafo TheSafo changed the title blocks [WIP] Update generator throttler to support block-based strategy Dec 31, 2025
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch 11 times, most recently from a5dc28f to 10ba2f9 Compare January 5, 2026 14:50
@TheSafo TheSafo changed the base branch from main to graphite-base/1675 January 5, 2026 16:00
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from 10ba2f9 to d724a4c Compare January 5, 2026 16:00
@TheSafo TheSafo changed the base branch from graphite-base/1675 to jsaf/throttle-error-change January 5, 2026 16:00
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch 4 times, most recently from 74c2f05 to 9ae8ce6 Compare January 6, 2026 18:11
@TheSafo TheSafo marked this pull request as ready for review January 6, 2026 19:15
@TheSafo TheSafo requested a review from a team as a code owner January 6, 2026 19:15
@TheSafo TheSafo marked this pull request as draft January 6, 2026 19:25
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch 2 times, most recently from 3c1df18 to 01c4cd3 Compare January 6, 2026 21:03
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from 01c4cd3 to 648642c Compare January 6, 2026 21:16
@TheSafo TheSafo changed the base branch from jsaf/throttle-error-change to graphite-base/1675 January 6, 2026 21:46
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from 648642c to a636e4c Compare January 6, 2026 21:46
@TheSafo TheSafo changed the base branch from graphite-base/1675 to jsaf/add-config-tests January 6, 2026 21:46
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from a636e4c to 661d654 Compare January 6, 2026 21:58
@TheSafo TheSafo force-pushed the jsaf/add-config-tests branch from af8d207 to cf2d888 Compare January 6, 2026 22:13
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch 3 times, most recently from e1d8abc to e73bb21 Compare January 6, 2026 22:31
@TheSafo TheSafo changed the title [WIP] Update generator throttler to support block-based strategy Update generator throttler to support block-based strategy Jan 7, 2026
@TheSafo TheSafo mentioned this pull request Jan 7, 2026
@TheSafo TheSafo force-pushed the jsaf/add-config-tests branch from cf2d888 to e564637 Compare January 15, 2026 21:34
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from e73bb21 to 57bce08 Compare January 15, 2026 21:34
@TheSafo TheSafo force-pushed the jsaf/add-config-tests branch from e564637 to a1be97c Compare January 15, 2026 21:35
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from 57bce08 to a47d241 Compare January 15, 2026 21:35
@TheSafo TheSafo marked this pull request as ready for review January 16, 2026 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants