Skip to content

Instrumentation#270

Draft
gewenyu99 wants to merge 3 commits intomainfrom
loop
Draft

Instrumentation#270
gewenyu99 wants to merge 3 commits intomainfrom
loop

Conversation

@gewenyu99
Copy link
Contributor

Created a tracker that is called through:

tracker?.onMessage(message);

and runs parallel to the agent to benchmark things like:

  • Input and output size
  • Context size
  • Log compactions
  • Turns per phase (that's what I'm calling each of the prompts we have in llm-prompts
  • Time per turn
  • Cost

What's cool is that we can use a similar pattern to build all kinds of plugins and instrumentation, so I might generalize this in the future to an interface...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant