Skip to content

feat(presets): add ready-to-use presets for typed-decisions workflows - #153

Closed
chilkotiKartik wants to merge 3 commits into
NandhaKishorM:mainfrom
chilkotiKartik:feat/typed-decisions-presets
Closed

chilkotiKartik wants to merge 3 commits into
NandhaKishorM:mainfrom
chilkotiKartik:feat/typed-decisions-presets

Conversation

@chilkotiKartik

Copy link
Copy Markdown
Contributor

Summary

Add preset generator functions in laya/presets.py corresponding to the fine-tuned typed-decisions workflows recognized by laya.router:

  • agent_trace_questions() for agent trajectory and observability triage (agent_trace_observability).
  • invoice_processing_questions() for automated invoice and PO matching (invoice_processing).
  • security_incident_questions() for SOC incident triage and threat verification (security_incidents).

Changes

  1. Defined agent_trace_questions(), invoice_processing_questions(), and security_incident_questions() in laya/presets.py with validated schemas and criteria rubrics.
  2. Exported new presets in laya/__init__.py and __all__.
  3. Added tests/test_workflow_presets.py verifying that generated question schemas match _TYPED_DECISION_WORKFLOWS routing signatures.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, presets for the typed-decisions workflows would help new users a lot. One request: the typed-decisions checkpoint was trained on specific question definitions for those workflows, and presets that match them will do much better than ones written fresh. Could you align the instructions and criteria with the dataset's own question set (LocalLLaMA/typed-decisions), and ideally report accuracy for a few states from its test split? Then I'd like to merge it.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Still holding, and the ask from September has not been picked up, so I want to restate it concretely rather than just leave this open.

Presets for the typed-decisions workflows are worth having and I would merge them. The problem is that a preset is a prompt, and the typed-decisions checkpoint was trained on specific question definitions for agent_trace_observability, invoice_processing and security_incidents. Questions written fresh will underperform the ones the model was trained against, and because a preset is the thing a new user reaches for first, a preset that scores worse than hand-written questions is actively misleading about what the checkpoint can do.

What would get this merged:

  1. Align the instructions and criteria with the question set in the LocalLLaMA/typed-decisions dataset, rather than writing them from the workflow names.
  2. Report accuracy on a few states from its test split, for your version against the dataset's own. Even a dozen rows per workflow would settle whether the wording matters, and laya-evals run will do the scoring.

If point 1 turns out to be enough and the numbers come out level, say so with the numbers and I will take it as is.

The code shape is fine and it still merges cleanly, so this is only about the content of the three question sets. Note that laya/presets.py has gained Swedish localization since (#728 and the held #729), so a rebase is worth doing when you come back to it.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Not merging this. The three presets are well formed and the file follows the house style, but a preset ships as a recommendation, so I ran all three against real weights on the cases they are written for, and they do not hold up.

security_incident_questions, on a textbook impossible-travel compromise and on a scheduled vulnerability scan:

"Impossible travel: user alice@corp.com signed in from Lagos 3 minutes after a
 London session, MFA push approved on the second try."
      true_positive=0.000  conf 1.000      <-- a real compromise, answered no at full confidence
      credential_compromise=0.398 conf 0.602
      disposition=close_benign conf 0.545  <-- would close it
      severity=1.227  urgency=1.364

"Nessus scanner 10.0.4.12 performed its scheduled weekly TCP sweep of the staging
 subnet. Owner: infra team, ticket INFRA-221."
      true_positive=0.067  conf 0.933
      disposition=close_benign conf 0.525

Both states get close_benign, and the real compromise is answered true_positive = 0.000 at confidence 1.000. A security triage preset that closes a credential compromise with full confidence is worse than no preset, because the confidence gate cannot catch it either.

invoice_processing_questions inverts on both of its own cases:

invoice matching its PO exactly, 14 line items     duplicate=0.771  <-- false positive
the actual second submission of the same invoice   disposition=reject, matches_order=0.488

agent_trace_questions cannot separate its two cases at all:

agent retried a policy-blocked delete_production_database twice
      action=retry  (the preset's own "abort" is what this is for)
      needs_review=0.451
agent read a ticket, called lookup_order, finished in 2 steps
      needs_review=0.541   <-- higher than the blocked destructive call

Part of this is a known defect, and it is worth knowing which part. true_positive's instruction contains the word "positive", which is #1244: a noul asking whether something is "positive" stops reading the state. I tried the two workarounds on your exact question:

as shipped here           real compromise P(true)=0.0132  benign scan P(true)=0.0650
same, with labels         real compromise P(true)=0.0901  benign scan P(true)=0.3375
reworded, no "positive"   real compromise P(true)=0.2706  benign scan P(true)=0.2928

Neither fixes it. In the shipped form the real compromise scores below the benign scan, and in the other two they are indistinguishable. So true_positive is not rescuable by rewording, and the other questions' problems are not #1244 at all, they are the checkpoints not having the operational semantics these questions assume.

The four mechanical gaps, for completeness, since they are the easy part: tests/test_workflow_presets.py is not in SCRIPT_SUITES in scripts/test_suites.py; docs/questions-and-answers.md's Presets section still says "Five" where the gate now computes eight; the same section has to name each new helper in backticks; and tests/test_packaging.py requires every exported name to appear on a docs page. The gate names all four.

What would make this mergeable is the measurement, not the wiring: pick the states each preset is for, score them against the preset, and show the answers are right. If a question cannot be made to work, the useful contribution is the finding rather than the preset. The existing presets each earned their place that way, and triage_questions is a good model to compare against. The invoice set looks closest to salvageable; the agent-trace needs_review and the security true_positive questions I would drop rather than reword.

Closing, and genuinely worth another go with the measurements attached.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants