Repository navigation
feat(presets): add ready-to-use presets for typed-decisions workflows - #153
chilkotiKartik wants to merge 3 commits into
Conversation
|
Thank you, presets for the typed-decisions workflows would help new users a lot. One request: the typed-decisions checkpoint was trained on specific question definitions for those workflows, and presets that match them will do much better than ones written fresh. Could you align the instructions and criteria with the dataset's own question set ( |
|
Still holding, and the ask from September has not been picked up, so I want to restate it concretely rather than just leave this open. Presets for the typed-decisions workflows are worth having and I would merge them. The problem is that a preset is a prompt, and the typed-decisions checkpoint was trained on specific question definitions for What would get this merged:
If point 1 turns out to be enough and the numbers come out level, say so with the numbers and I will take it as is. The code shape is fine and it still merges cleanly, so this is only about the content of the three question sets. Note that |
|
Not merging this. The three presets are well formed and the file follows the house style, but a preset ships as a recommendation, so I ran all three against real weights on the cases they are written for, and they do not hold up.
Both states get
Part of this is a known defect, and it is worth knowing which part. Neither fixes it. In the shipped form the real compromise scores below the benign scan, and in the other two they are indistinguishable. So The four mechanical gaps, for completeness, since they are the easy part: What would make this mergeable is the measurement, not the wiring: pick the states each preset is for, score them against the preset, and show the answers are right. If a question cannot be made to work, the useful contribution is the finding rather than the preset. The existing presets each earned their place that way, and Closing, and genuinely worth another go with the measurements attached. |
Summary
Add preset generator functions in
laya/presets.pycorresponding to the fine-tuned typed-decisions workflows recognized bylaya.router:agent_trace_questions()for agent trajectory and observability triage (agent_trace_observability).invoice_processing_questions()for automated invoice and PO matching (invoice_processing).security_incident_questions()for SOC incident triage and threat verification (security_incidents).Changes
agent_trace_questions(),invoice_processing_questions(), andsecurity_incident_questions()inlaya/presets.pywith validated schemas and criteria rubrics.laya/__init__.pyand__all__.tests/test_workflow_presets.pyverifying that generated question schemas match_TYPED_DECISION_WORKFLOWSrouting signatures.