Configs and updates to gaia eval #304
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Based on your review schedule, I'll hold off on reviewing this PR until it's marked as ready for review. If you'd like me to take a look now, comment
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description by Korbit AI
What change is being made?
Update Gaia evaluation configs and flow to use Apriel 1.5 LLM, switch to function-call style guidance, extend max_turns settings, and wire LLM endpoint configuration via environment variable; rename action class, adjust Gaia Gym/Benchmark wiring, and add supporting experiment/docs.
Why are these changes being made?
Switch to an Apriel-based evaluation setup with function-call style guidance to improve tool use and reliability; expose LLM base URL via environment for flexibility; extend max_turns to better control evaluation length and align env creation with new config fields; and update docs to guide setup.