Context Management for Session in Case of Context Length Breach #116
Description
Confirm this is a new feature request
- I've checked the current issues, and there's no record of this feature request
Describe the feature
Problem:
After a certain number of messages, the context length of the LLM may be exceeded. This scenario is currently unhandled.
Proposed Implementations:
- Simple Sliding Window Approach:
- Continuously remove the oldest messages to make room for new ones as the context approaches its limit.
- Smart Context Management with Vector Matching:
- Implement a vector-based retrieval system to inject only the most relevant older messages (beyond a certain threshold) into the context.
Additional Context
No response