HTTP sink: add gauge metrics for packet payload sizes #1718
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces an experimental issue tracker for use by agentic code tools. While I have found it effective on personal projects it is an open question how well this will work for a multi-person effort. If we don't like it we can yank it. Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
### What does this PR do? This commit updates payloadtool with a tracking allocator, showing: * total allocations * total deallocations * total bytes allocated * peak bytes live This information is exposed on stderr if `--memory-stats` is provided to payloadtool. The goal is to enable more convenient optimization flows for lading developers.
Runs of cargo bench require a release build. This adds a lot of time to local ci/validate loops even if one is not working on optimizations. Remove the check from ci/validate in this commit, retained in CI. Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Apply allocation avoidance and buffer reuse patterns: - Change all String fields to &'static str (12 allocations per member → 0) - Use serde_json::to_writer() instead of to_string() - Reuse buffer across iterations with clear() - Update test to use serde_json::Value for deserialization **Micro-benchmark** (cargo criterion): - +66% throughput (40% time reduction) **Macro-benchmark** (hyperfine with payloadtool): - +18% faster (4.92s → 4.15s mean)
### What does this PR do? This commit introduces skills for optimization and validation of the same patterned on my experimental https://github.com/blt/datadog-skills. My hope is that this will be useful to core lading engineers but also helpful for contribution efforts.
### What does this PR do?
Reduce allocations by reusing buffers instead of creating new Strings
per iteration in both framed and unframed to_bytes methods.
Technique: buffer-reuse
- Unframed: Use a reusable Vec<u8> buffer instead of format!() per iteration
- Framed: Accumulate into single Vec<u8> content buffer + reusable member
buffer, instead of Vec<String>
Micro-benchmark results (cargo bench with critcmp):
- 1 MiB: 1.03x faster (3%)
- 10 MiB: 1.09x faster (9%)
- 100 MiB: 1.20x faster (20%)
- 1 GiB: 1.26x faster (26%)
Macro-benchmark (payloadtool):
- Memory: -12.9% (27.28 MiB -> 23.75 MiB)
- Allocations: -8.6% (59,034 -> 53,960)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* opt: eliminate allocations in syslog serialization Hypothesis: Reduce allocations by 60-80% per log line Technique: Reusable buffer + static str references Changes: - Member struct: hostname/app_name now &'static str (not String) - Removed into_string() method that created temp String - to_bytes: use reusable Vec<u8> buffer with write! macro - Same pattern as JSON (PR #1692) and Splunk HEC (PR #1694) Expected improvements: - Eliminates 3 allocations per iteration (hostname, app_name, format!) - Reduces memory usage for buffer reuse - Similar to recent successful optimizations Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Record syslog optimization hunt results Added database entries tracking the successful optimization: - 28% faster micro-benchmarks (avg 479 -> 668 MiB/s) - 14.5% faster macro-benchmarks (8.3ms -> 7.1ms) - 49.3% reduction in allocations (67,688 -> 34,331) - 35.8% reduction in total memory allocated (6.17 -> 3.96 MiB) These database files track completed optimization attempts to prevent duplication in future hunts. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix clippy warnings: remove unnecessary explicit derefs Clippy flagged explicit_auto_deref on lines 79-80. Auto-deref handles &'static str conversion from choose() result. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Arpafaucon
merged commit 06c3be0
into
greg/http_payload_size_metrics
52 of 53 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Superseded by #1719