diff --git a/README.md b/README.md index 16744f965..2b2a0c408 100644 --- a/README.md +++ b/README.md @@ -429,6 +429,17 @@ as a fast base to specialise, not as a zero-shot decision engine. Runtime on 2xT4 is roughly 4-5 hours for 4 epochs over ~30k questions. +### Worked example: a browser-agent decision head + +[`docs/finetune_browser_agent.md`](docs/finetune_browser_agent.md) records a complete specialisation +on a single 16 GB GPU with no paid API: Laya as the operation/target decider for +[browser-use/jev-ultrafast](https://github.com/browser-use/jev-ultrafast) (same request format as +TypeSafe Jev). Element top-1 among ~45 candidates goes from 0.10 zero-shot to 0.66, real-task +success from 0 % to 62 % at 17-23 ms per step; weights, pipeline code and per-run results are on +the Hub at [cklxx/laya-browser](https://huggingface.co/cklxx/laya-browser). The write-up covers the +data recipe (reverse-generated goals, executed DONE states, Mind2Web, on-policy corrections), the +input-format change that mattered most, and the things that did not work. + --- ## Support the Project diff --git a/docs/finetune_browser_agent.md b/docs/finetune_browser_agent.md new file mode 100644 index 000000000..d18f498c8 --- /dev/null +++ b/docs/finetune_browser_agent.md @@ -0,0 +1,87 @@ +# Fine-tuning Laya as a browser-agent decision head + +A worked, fully reproducible example of specialising Laya for a decision family it cannot do +zero-shot: picking the next browser action (operation + target element) for +[browser-use/jev-ultrafast](https://github.com/browser-use/jev-ultrafast), whose `/v1/systemone` +request format is the same as `Agent.predict(state, questions)`. Everything below ran on one +RTX 4070 Ti SUPER (16 GB) with no paid API; weights, code and per-run results are at +[huggingface.co/cklxx/laya-browser](https://huggingface.co/cklxx/laya-browser). + +## Result + +| | typed-decisions, zero-shot | fine-tuned | +|---|---|---| +| element top-1 on held-out pages (2,734 decisions, ~45 candidates each) | 0.10 (chance) | **0.66** (421M) / 0.63 (322M) | +| operation accuracy (CLICK / TYPE_TEXT / SELECT / DONE) | 0.54 | 0.88–0.89 | +| 16 real browser tasks, 3 runs each | 0 % | **62 %** (322M), 50 % (421M) | +| latency per step (3 questions, 30–65 candidates) | 50–200 ms | 41–50 ms (421M), **17–23 ms** (322M) | + +The live suite is bimodal: 10 tasks pass 3/3 (category / tab / page navigation, checkbox, +`