Fleshing out OpenAIModel #6
Conversation
The self.model field doesn't exist on my end, so hoping this change fixes it?
| max_tokens=1, | ||
| logprobs=5, | ||
| ) | ||
| for i in range(self.max_api_retries): |
There was a problem hiding this comment.
We should handle this with the openai client: https://github.com/openai/openai-python/blob/main/src/openai/_client.py#L74
There was a problem hiding this comment.
I didn't see a backoff strategy in the openai client, and it looks like they recommend using other libraries (or implementing from scratch) to do retries with backoff here:
https://cookbook.openai.com/examples/how_to_handle_rate_limits#retrying-with-exponential-backoff
Would you be down for using the tenacity or backoff libraries they mention in that cookbook link, or even keeping the current implementation?
There was a problem hiding this comment.
I've used and like the tenacity library -- I thought after the refactor of the openai python client library, they recommended not using external libraries for retrying. Do you know whether the cookbook is outdated, or how outdated?
There was a problem hiding this comment.
I looked around a bit more, and looks like the same rate limit guide is in the API documentation itself:
https://platform.openai.com/docs/guides/rate-limits?context=tier-free
The final call is yours on whether to keep a manual backoff strategy or just increase max_retries in the openai client, it should be fine either way!
2 participants
Two fixes: