Skip to content

gather_evidence tool response filtering irrelevant Context, with updated threshold of 0#1106

Merged
jamesbraza merged 4 commits intomainfrom
validating-context-3
Sep 26, 2025
Merged

gather_evidence tool response filtering irrelevant Context, with updated threshold of 0#1106
jamesbraza merged 4 commits intomainfrom
validating-context-3

Conversation

@jamesbraza
Copy link
Copy Markdown
Collaborator

We were seeing gather_evidence responses like this:

Added 22 pieces of evidence, 0 of which were relevant. Best evidence(s):

1. 

2. 

3. 

4. 

5. 

Status: Paper Count=2 | Relevant Papers=0 | Current Evidence=0 | Current Cost=$0.0086"

There's more to the story here, but this PR at least:

  • Documents when we'd have an empty Context.context
  • Updates the relevant evidence threshold from 5 to 0
  • Filters the gather_evidence tool response for zero-scored Context

@jamesbraza jamesbraza self-assigned this Sep 25, 2025
Copilot AI review requested due to automatic review settings September 25, 2025 22:43
@jamesbraza jamesbraza added the bug Something isn't working label Sep 25, 2025
@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. documentation Improvements or additions to documentation enhancement New feature or request labels Sep 25, 2025
@dosubot
Copy link
Copy Markdown

dosubot bot commented Sep 25, 2025

Related Documentation

Checked 1 published document(s). No updates required.

How did I do? Any feedback?  Join Discord

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR improves the gather_evidence tool by filtering out irrelevant context entries and updating the relevance threshold from 5 to 0. The changes address issues where the tool was reporting evidence that wasn't actually useful.

  • Updated relevance scoring threshold from 5 to 0 to better identify useful evidence
  • Enhanced context filtering to exclude zero-scored (irrelevant) contexts from tool responses
  • Added documentation and validation for empty context strings

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
src/paperqa/agents/tools.py Updated evidence gathering logic to filter contexts with score > 0 and removed relevance counting
src/paperqa/types.py Added string constraints and documentation for empty context handling
src/paperqa/core.py Fixed conditional logic for parser usage
tests/test_paperqa.py Added type annotation and updated assertion to handle whitespace
tests/test_agents.py Adjusted test assertion to allow tolerance for whitespace in chunks

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment on lines -279 to -280
try:
result_data = parser(context) if parser else {}
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This little refactor removes the extra if parser else {}, making the code slightly less stateful

Copy link
Copy Markdown
Collaborator

@whitead whitead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work getting all these cleaned-up

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Sep 26, 2025
@jamesbraza jamesbraza merged commit 0effe2e into main Sep 26, 2025
5 checks passed
@jamesbraza jamesbraza deleted the validating-context-3 branch September 26, 2025 05:08
@dosubot
Copy link
Copy Markdown

dosubot bot commented Sep 26, 2025

Documentation Updates

Checked 1 published document(s). No updates required.

How did I do? Any feedback?  Join Discord

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request lgtm This PR has been approved by a maintainer size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants