Skip to content

Generalizing tests for smarter LLMs #1149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jamesbraza merged 9 commits into from
Oct 21, 2025
Merged

Generalizing tests for smarter LLMs #1149

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
444a276
Used more clear keys in test_partitioning_fn_docs
jamesbraza Oct 20, 2025
9791ede
Added session.contexts assertion to avoid generator-based assertion f…
jamesbraza Oct 20, 2025
6e73102
Clarifying settings.answer.evidence_k rationale
jamesbraza Oct 20, 2025
53e0ba5
Generalized test_partitioning_fn_docs query to work for smarter models
jamesbraza Oct 20, 2025
58bf747
Generalized test_settings_default_instantiation to work for Anthropic…
jamesbraza Oct 20, 2025
b89e81f
Allowing 1.0mm in test_querying_tables assertion
jamesbraza Oct 20, 2025
9f5daa0
Removed duplicate 'is is' from questions
jamesbraza Oct 20, 2025
ffc07b6
Loosened test_propagate_options since removal of 'is is' affected res…
jamesbraza Oct 20, 2025
8c04cb4
Using first 30-chars instead of first 20-chars in substring check, to…
jamesbraza Oct 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Removed duplicate 'is is' from questions
  • Loading branch information
@jamesbraza
jamesbraza committed Oct 20, 2025
commit 9f5daa0e4efcfd4d6e7b414a918000e7a1110fbf
2 changes: 1 addition & 1 deletion tests/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ def agent_stub_session() -> PQASession:
# > are already imported: paperqa
from paperqa.types import PQASession

return PQASession(question="What is is a self-explanatory model?")
return PQASession(question="What is a self-explanatory model?")


@pytest.fixture
Expand Down
4 changes: 2 additions & 2 deletions tests/test_agents.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -516,7 +516,7 @@ async def test_propagate_options(agent_test_settings: Settings) -> None:
agent_test_settings.answer.evidence_skip_summary = True

response = await agent_query(
query="What is is a self-explanatory model?",
query="What is a self-explanatory model?",
settings=agent_test_settings,
agent_type=FAKE_AGENT_TYPE,
)
Expand Down Expand Up @@ -622,7 +622,7 @@ def files_filter(f) -> bool:

agent_test_settings.agent.callbacks = callbacks

session = PQASession(question="What is is a self-explanatory model?")
session = PQASession(question="What is a self-explanatory model?")
env_state = EnvironmentState(docs=Docs(), session=session)
built_index = await get_directory_index(settings=agent_test_settings)
assert await built_index.count, "Index build did not work"
Expand Down