Repository navigation
Conversation
Independent calibration and evaluation need stable provenance and family boundaries. Preserve rendered schema order, reject conflicting duplicate labels, and expose one representative per group so repeated tickets cannot inflate statistical evidence.
Official test families must stay held out when derived variants or duplicate links are present. Record reservations in the verified manifest and keep zero-probability roles reserved.
|
Thank you, keeping families of near-duplicate examples on one side of a split is exactly the discipline that stops calibration looking better than it is. #186 showed that problem in our own notebook today. This is part of a larger series (#151, #152, #159, #164, #165, #167), and it's a lot to review at once. Could you tell me which one piece you'd most like to land first? A single focused PR with a short example is what I can review properly. |
|
Thanks for the clear steer. Of the six, please prioritize #151 first, once I mark it ready again. I have put #151 and #152 back in draft; #159, #164, #165 and #167 will stay draft. The smallest useful slice is the family-preserving split. For example, an English lost-card request and its Arabic translation from the same source family must stay on one side of the train/calibration/policy/test boundary. That is the data condition the later risk claims depend on. I'll tighten #151 around that slice, put the example and local verification up front, and request review only when it is small enough to review properly. No action needed on the rest of the stack now. |
|
Thank you for the clear plan. I've thought about scope: the open-source package is staying focused on fast local inference, so adaptation, training and deployment tooling like this series is a better fit as a standalone project than inside |
Summary
Prepare labelled domain-adaptation data with reproducible boundaries between training, temperature calibration, policy selection and final evaluation. Random row splits can put copies, translations or revisions of the same example on both sides of an evaluation; this opt-in data contract keeps declared and detected families together and preserves the provenance needed to audit the split.
Changes
Testing
Additional validation on the minimum supported Python version: all 21 data-contract tests pass on Windows with Python 3.10.11, CPU Torch 2.14.0, Transformers 5.17.0 and NumPy 2.2.6. The isolated test run used synthetic/local fixtures with Hub access disabled; it establishes compatibility, not model quality.
All checks ran locally without downloading pretrained weights or running training.
Notes for reviewer
This is the data foundation for a larger domain-adaptation/selective-decision workflow. It preserves the SDK, adds no dependency, and leaves inference, training, calibration fitting and GitHub workflows unchanged. It complements #19 and #53. Existing manifests without reservations remain verifiable.
Adapters must identify translations, conversations and semantic near-duplicates that exact matching cannot infer. Language and license metadata require source audits. Hash verification establishes consistency, not authenticity. Fractions are expected proportions; small or skewed splits can lack evidence.
These tests establish data/configuration behavior only. No model-accuracy, calibration-quality or automation-coverage improvement is claimed.