Fix rate limiting to evenly space API calls (fixes #28)#29
Open
vipinkataria2209 wants to merge 2 commits intoUSEPA:mainfrom
Open
Fix rate limiting to evenly space API calls (fixes #28)#29vipinkataria2209 wants to merge 2 commits intoUSEPA:mainfrom
vipinkataria2209 wants to merge 2 commits intoUSEPA:mainfrom
Conversation
Fix rate limiting to evenly space API calls (fixes USEPA#28) Replace blocking rate limiter with token bucket implementation that evenly spaces API calls over time instead of allowing bursts followed by long waits. This significantly improves performance for multi-year data requests by reducing total wait time. - Implement TokenBucketRateLimiter class to space calls evenly - Remove @sleep_and_retry and @limits decorators from __aqs method - Add rate limiter acquire() call at start of each API request - Maintains 10 calls per minute limit while improving efficiency This change makes pyaqsapi's rate limiting behavior similar to RAQSAPI's httr2::req_throttle, addressing the performance gap reported in issue USEPA#28.
Collaborator
|
@vipinkataria2209 thank you for your submission, unfortunately any ratelimiting that is implemented needs to satisfy the rules as stated on the EPA AQS DataMart API's website. I realized this while considering how I should re-implement this ratelimit. Unfortunately I am going to have to change RAQSAPI's rate limiting functionality to match this as well. I am in talks with the DataMart team to see if we can get something faster than what they are requiring. There may be some changes to the API coming in the near future that will allow us to do implement something similar to what you are proposing but for now we have to follow the guidelines as stated on the Datamart API page. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR replaces the blocking rate limiter with a token bucket implementation that evenly spaces API calls over time, significantly improving performance and reliability for multi-year data requests.
Fixes #28
Problem
The previous rate limiting implementation using
@limitsand@sleep_and_retrydecorators from theratelimitpackage caused inefficient behavior:httr2::req_throttleimplementationSolution
Implemented a
TokenBucketRateLimiterclass that:Key Features
Smart spacing:
Predictable timing: Makes completion time easier to estimate
Server protection: Prevents bursts that can overwhelm the API server
Feature parity: Matches RAQSAPI's
httr2::req_throttlebehaviorChanges
TokenBucketRateLimiterclass tohelperfunctions.py@sleep_and_retryand@limitsdecorators from__aqs()methodtests/test_rate_limiter.pytests/test_helperfunctions.pyTesting
Impact
For 74 API calls (74 years of data):
Key improvements: