Fixing dockey/doc_id mismatch when no metadata is found

[jamesbraza]

Summary

• Fixes the flaky test_get_directory_index[check-md-query] CI failure caused by a dockey/doc_id mismatch when both Crossref and Semantic Scholar fail to return metadata for "Gravity Hill"
• When upgrade_doc_to_doc_details falls back (no metadata found) and the dockey was auto-generated from content_hash, "doc_id" is now included in fields_to_overwrite_from_metadata so the Pydantic validator can sync dockey with the newly computed doc_id
• User-provided dockey values (e.g. dockey="test" in test_docs_lifecycle) are still preserved

Root cause

When both metadata providers fail, the fallback path in DocMetadataClient.upgrade_doc_to_doc_details set fields_to_overwrite_from_metadata = set(). This prevented overwrite_docname_dockey_for_compatibility_w_doc from syncing dockey with doc_id. The dockey stayed as the raw 32-char content_hash while the Pydantic validator recomputed doc_id as a 16-char hash via compute_unique_doc_id(), so neither expected ID in the test matched.

The latent bug was introduced by #1029, which changed the test expectations from raw md5sum() (32-char) to compute_unique_doc_id(None, md5sum(...)) (16-char) without updating the fallback path to keep dockey in sync. CI passed at the time because the bug only surfaces when metadata providers fail — before #1029, the fallback dockey (raw content hash) still matched the expected IDs, so provider failures were harmless. The test started failing now because Crossref is returning errors in CI (likely an expired CROSSREF_API_KEY secret), pushing the test into the broken fallback path.

Test plan

• Verified fix locally: when both providers fail, dockey now equals doc_id (93085aa5ff54865c)
• Verified no regression: test_docs_lifecycle still passes (user-provided dockey="test" is preserved)
• All 47 tests in test_clients.py and test_paperqa.py::test_docs_lifecycle pass
• All pre-commit hooks pass

[dosubot]

Related Documentation

Checked 1 published document(s) in 1 knowledge base(s). No updates required.

^{How did I do? Any feedback?}  

[Copilot]

Pull request overview

This PR fixes a bug where dockey and doc_id could become mismatched when metadata providers fail to return metadata for a document. The issue caused flaky test failures in test_get_directory_index[check-md-query].

Changes:

• Modified the fallback path in upgrade_doc_to_doc_details to include "doc_id" in fields_to_overwrite_from_metadata when dockey was auto-generated from content_hash
• This allows the Pydantic validator to sync dockey with the newly computed doc_id, ensuring both fields use the same 16-character hash

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.