Skip to content

docs: worked example of fine-tuning Laya as a browser-agent decision head (weights + pipeline on the Hub) - #143

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
cklxx:docs/browser-agent-finetune
Sep 23, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
cklxx:docs/browser-agent-finetune

Conversation

@cklxx

@cklxx cklxx commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Docs only. Adds docs/finetune_browser_agent.md plus a short pointer under the README's Fine-Tuning section.

It records a complete, reproducible specialisation of Laya for a decision family the base checkpoints cannot do zero-shot: choosing the next browser action (operation + target element) for browser-use/jev-ultrafast, whose /v1/systemone request format is identical to Agent.predict(state, questions). Everything ran on one RTX 4070 Ti SUPER with no paid API.

zero-shot (typed-decisions) fine-tuned
element top-1, 2,734 held-out decisions, ~45 candidates 0.10 0.66 (421M) / 0.63 (322M)
operation accuracy 0.54 0.88–0.89
16 real browser tasks × 3 runs 0 % 62 % (322M)
latency per 3-question step 50–200 ms 17–23 ms (322M)

Weights (three Laya-format checkpoint dirs), the full pipeline as a uv project, a verify.py, a demo video and every per-run result JSON are on the Hub: https://huggingface.co/cklxx/laya-browser.

The write-up covers the data recipe (reverse-generated goals, executed DONE states, step-2 negatives, Mind2Web conversion, on-policy corrections), the input-format change that turned out to matter most (elements out of the state, head_max_len 768), and a "what did not work" list (templated DONE goals, history-length shortcut, class imbalance from Mind2Web, text truncation, torch.compile, LLM escalation with 8B/27B teachers).

No code changes; independent of #25.

…head

Adds docs/finetune_browser_agent.md and a pointer under README's Fine-Tuning
section: a fully reproducible specialisation of Laya for jev-ultrafast's
operation/target decisions on one 16 GB GPU with no paid API (element top-1
0.10 -> 0.66 among ~45 candidates, 0 % -> 62 % on 16 real browser tasks,
17-23 ms per step). Records the data recipe, the input-format change that
mattered most, the failed attempts, and links the weights, code and per-run
results on the Hub (cklxx/laya-browser).
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, a worked fine-tuning example with weights and pipeline on the Hub is the kind of thing that helps people see what Laya can be specialised for. Merging.

@NandhaKishorM
NandhaKishorM merged commit 6bbe781 into NandhaKishorM:main Sep 23, 2026
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…head (NandhaKishorM#143)

Adds docs/finetune_browser_agent.md and a pointer under README's Fine-Tuning
section: a fully reproducible specialisation of Laya for jev-ultrafast's
operation/target decisions on one 16 GB GPU with no paid API (element top-1
0.10 -> 0.66 among ~45 candidates, 0 % -> 62 % on 16 real browser tasks,
17-23 ms per step). Records the data recipe, the input-format change that
mattered most, the failed attempts, and links the weights, code and per-run
results on the Hub (cklxx/laya-browser).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants