TL;DR: GPT-2 124M can do tool calling. Two production checkpoints, both reproducible on CPU. Adapter approach (250K trainable) → 50% BFCL v4. Full fine-tune (124M trainable, 68 min CPU) → 92% on 690-item fresh benchmark with zero training contamination.
- Highlights
- Two checkpoints
- Benchmarks
- Reproducing
- Architecture
- Limitations
- Files
- License & citing
- 🇷🇺 Описание на русском
| Model | Params | Train | BFCL v4 OVERALL | Fresh (no leakage) |
|---|---|---|---|---|
| GPT-2 + Adapter (this work) | 125M (250K trained) | ~1.5h CPU | 50.0% | 53.5% |
| GPT-2 + Full FT (this work) | 124M (all trained) | 68 min CPU | ~95% on subsets in train | 92.0% |
| Phi-3-mini Instruct | 3.8B | proprietary | ~40% | — |
| Llama-3.1 8B Instruct | 8B | proprietary | ~62% | — |
| Llama-3.1 70B Instruct | 70B | proprietary | ~70% | — |
| GPT-4-turbo | ~1.7T | proprietary | ~88% | — |
Why this matters: A 125M model, fine-tuned for 68 minutes on a laptop CPU, generalizes to 690 completely novel tool-calling tasks at 92% accuracy. The data has zero overlap with training (function names like catalog_lepidoptera_specimen, assay_lichen_oxygen_uptake, transcribe_cuneiform_tablet — generated by Claude Opus 4.7).
- Frozen GPT-2 124M + 250K-parameter steering adapter at layer 6
- Adds a delta vector to the residual stream after a small bottleneck (768→192→96→768)
- Refusal head (irrelevance detection) intact: 40-45% on BFCL irrelevance
- Use when: memory-constrained, need refusal capability, or want minimal training
- All 124M GPT-2 parameters fine-tuned
- 1500 mixed examples (500 BFCL + 500 Glaive + 500 xLAM held-out slice), 1 epoch
- LR 1e-5, AdamW, grad_accum=4, PAD=512
- Use when: maximum accuracy on novel functions
Source: bfcl_eval/data from gorilla-llm/Berkeley-Function-Calling-Leaderboard
Honest disclosure: BFCL was partly in the training distribution for both checkpoints (first ~125 examples per subset used for training). So BFCL v4 numbers below have leakage.
| subset | adapter | full FT |
|---|---|---|
| simple_python | 80.0% | 100.0% * |
| live_simple | 72.5% | 80.0% |
| simple_java | 47.5% | 80.0% |
| simple_javascript | 52.5% | 95.0% |
| multiple | 30.0% | 37.5% * |
| live_multiple | 20.0% | 20.0% |
| parallel | 72.5% | 80.0% * |
| live_parallel | 62.5% | 81.2% |
| parallel_multiple | 42.5% | 57.5% |
| live_parallel_multiple | 45.8% | 54.2% |
| irrelevance | 40.0% | 15.0% |
| live_irrelevance | 45.0% | 32.5% |
| live_relevance | 37.5% | 18.8% |
| OVERALL | 50.0% | (partial leakage, see fresh bench below) |
* subsets present in training, partial memorization
We generated 690 tool-call test items via 9 parallel Claude Opus 4.7 agents in 9 niche domains (biology, industrial engineering, history/culture, materials science, niche technology + multiple-choice + parallel + irrelevance variants). All function names are novel — no overlap with BFCL/Glaive/xLAM.
| Model | Params | Fresh slice (n=20) | Inference latency |
|---|---|---|---|
| GPT-2 + Full FT (this work, CPU) | 124M | 98.2% (full 450) | ~15s CPU |
| Phi-3-mini Instruct (ollama) | 3.8B | 100.0% | ~10s CPU |
| Qwen3 4B (ollama) | 4B | 95.0% | ~12s CPU |
Reproduce: python src/bench_external_models.py --model phi3:3.8b --n 20 (requires ollama running and the model pulled).
Honest takeaway: Phi-3-mini 3.8B beats us by ~2pp on this 20-item slice (and is much smarter on harder reasoning tasks we don't measure). On simple single-tool-call generalisation we're competitive with a 30× larger model — that's the entire selling point of this repo, not a "we beat Phi-3" claim.
| subset | plain GPT-2 | adapter | full FT | FT+adapter |
|---|---|---|---|---|
| simple (n=450) | 0.0% | 61.6% | 98.2% | 80.2% |
| multiple (n=80) | 0.0% | 2.5% | 62.5% | 55.0% |
| parallel (n=80) | 0.0% | 16.2% | 88.8% | 88.8% |
| irrelevance (n=80) | 100.0% ** | 96.2% | 90.0% | 53.8% |
| OVERALL (n=690) | 11.6% | 53.5% | 92.0% | 75.2% |
** plain GPT-2 "100%" on irrelevance is a metric artefact — the model never outputs a valid function name, so it formally satisfies the refusal check.
git clone https://github.com/barometech/gpt2-tool-call
cd gpt2-tool-call
git lfs pull # fetch gpt2_ft_final.pt (475 MB)
pip install torch==2.4.0+cpu fastapi numpy safetensors tokenizers \
--index-url https://download.pytorch.org/whl/cpu --extra-index-url https://pypi.org/simple
# Base GPT-2 weights are auto-downloaded on first run to weights/base_gpt2/
# (or pre-pull manually:)
huggingface-cli download openai-community/gpt2 --local-dir weights/base_gpt2Quick smoke test (~10 min, 35 prompts per model):
python src/bench_fresh_4way.py --smokeFull bench (4 models × 690 prompts, ~6h CPU):
python src/bench_fresh_4way.py# Fresh bench, 4-way comparison (~6h CPU)
python src/bench_fresh_4way.py
# BFCL v4 on adapter
python src/bench_bfcl_v4.py# Full FT: ~68 min on 4 CPU threads
python src/train_ft.py
# Adapter (h13): ~1.5h CPU
python src/train_adapter.pyimport torch, json
from src.integrated_gpt2_torch import GPT2, encode, decode
m = GPT2()
m.load_state_dict(torch.load("weights/gpt2_ft_final.pt", map_location='cpu'))
m.eval()
spec = {"name": "get_weather", "description": "Get weather for a city",
"parameters": {"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"]}}
prompt = (
f"SYSTEM: You are a helpful assistant with access to the following functions. Use them if required -\n"
f"{json.dumps(spec, indent=2)}\n\n\n"
f"USER: What's the weather in Paris?\n\n\n"
f"ASSISTANT: <functioncall> "
)
ids = encode(prompt)
with torch.no_grad():
for _ in range(40):
logits, _ = m(torch.tensor([ids]))
nxt = int(logits[0, -1].argmax())
ids.append(nxt)
if decode([nxt]) == "}": break
print(decode(ids[len(encode(prompt)):]))
# → {"name":"get_weather","arguments":{"city":"Paris"}}input → GPT-2 wte+wpe → L0..L5 → L6 hidden ─┐
│
├→ adapter (768→192→96)
│ │
│ ├→ classifier heads (action, gate, ...)
│ │
│ └→ W_steer (96→768) delta
│
+alpha·delta (residual feedback)
│
↓
L7..L11 → ln_f → logits
Trainable: adapter only (250K params), GPT-2 frozen.
Why layer 6 (middle)? GPT-2 124M has 12 transformer blocks. Empirically L5–L7 capture the best mix of syntactic features (early) and semantic features (late). L6 is the sweet spot: hidden state already disambiguated the request topic but downstream layers still have enough capacity to compose the JSON output. Probing earlier (L2–L4) under-fits structure; later (L9–L11) over-constrains and loses argument extraction.
W_steer mechanism. A single dense matrix (96→768). It maps a pooled adapter feature back into GPT-2's hidden space. The output (delta) is broadcast over the sequence and added as a residual: h6_steered = h6 + α·delta. Downstream layers see "GPT-2 with a small additive push." W_steer is initialised to zero, so before training the pipeline is identity (same logits as plain GPT-2). Training learns just the push that nudges the output toward a tool-call JSON.
Classifier heads (inherited). Eight small linear heads on the pooled 96-dim feature: action, scope, format, specificity, target, ptr_s, ptr_e, gate. They come from a prior project that classified short commands ("read src/auth.py", "delete this") into action/object slots. We reuse those weights as warm initialisation for the adapter bottleneck (W1, ln1, W2, ln2). The classifier outputs themselves are not used at inference for tool calling — only W_steer matters for generation. The gate head is what gives the adapter useful refusal capability (40-45% on BFCL irrelevance).
Training recipe (train_adapter.py). Initialise W1/ln1/W2/ln2 from adapter_torch_EN_BFCL.npz. Initialise W_steer to zeros. Freeze GPT-2 124M. Train only adapter parameters (250K) for ~1.5h on CPU with: 600 BFCL pairs + 300 Glaive anchor pairs, AdamW lr=3e-5, batch=2, PAD=2048 (Position Interpolation ×2), grad-clip 1.0, 1 epoch. Save the best epoch by name@1 on a held-out slice.
Standard causal LM fine-tune. All 124M params trainable. Loss is cross-entropy on the gold tool-call tokens only (system+user masked with -100).
Training recipe (train_ft.py). No adapter, no PI. Start from raw HF openai-community/gpt2 weights. Unfreeze everything. Mix 500 BFCL + 500 Glaive + 500 xLAM (held-out slice) examples, shuffle. AdamW lr=1e-5, weight_decay=0.01, grad-clip 1.0, batch=1, grad_accum=4 (effective batch=4), PAD=512, 1 epoch. ~68 minutes on 4 CPU threads. Loss drops 1.6 → 0.3 within 100 optimisation steps then plateaus — single epoch is sufficient.
- Context ≤ 1024 tokens (GPT-2 native). Position Interpolation experiments did not preserve quality reliably.
- Multi-step reasoning: GPT-2 124M cannot decompose multi-step research tasks with rich tool sets. It works well for single tool call but breaks on 5-step workflows.
- Ensemble of clones gives 0 gain. We tested 10× workers + 1 orchestrator (all identical FT weights, greedy decode). Result: 100% unanimous voting, same accuracy as single model, 7× slower. Real ensemble requires diversity (different FT runs or sampling temperatures).
- English only. Multilingual not tested.
- Code generation: GPT-2 124M cannot write meaningful code.
src/
integrated_gpt2_torch.py # GPT-2 backbone (pure torch)
steering_v2.py # adapter architecture
long_context_pi_chunk.py # Position Interpolation helper
train_ft.py # full FT script (~68 min)
train_adapter.py # adapter training (~1.5h)
bench_ft.py # bench FT on BFCL v4
bench_fresh_4way.py # bench plain/adapter/ft/ft+adapter on fresh 690
bench_bfcl_v4.py # bench adapter on full BFCL v4
weights/
adapter_h13_bfcl_ep1.pt # 1 MB — adapter (steering + classifier heads) for the FROZEN base
adapter_torch_EN_BFCL.npz # 700 KB — warm-init weights for the adapter bottleneck:
# W1 (768→192), ln1, W2 (192→96), ln2, plus 8 classifier heads.
# Origin: action-classifier from a prior English command-parsing
# project ("read src/auth.py" → action=read, target=file, ...).
# The classifier heads themselves are NOT used at tool-call
# inference — only their pre-trained bottleneck features
# serve as a useful initialisation. Training starts here and
# adds W_steer on top. See train_adapter.py for the wiring.
gpt2_ft_final.pt # 475 MB — full FT model (Git LFS)
BASE_GPT2_README.md # how to fetch base GPT-2 from HF
bench/
fresh_bench_*.json # 9 files, 690 items total — generated by Claude Opus 4.7
results/
h13_bfcl_v4.json # raw BFCL v4 results for adapter
fresh_4way.json # raw 4-way fresh bench results
MIT. Use for research, commercial, anything. No warranty.
If you use this:
@misc{popovich_gpt2_tool_call_2026,
title = {GPT-2 124M Tool Calling via Steering Adapter and Full Fine-tune},
author = {Popovich, Pavel D.},
year = {2026},
note = {Also known as ``Tekhnozhrets'' (Техножнец). GitHub: barometech},
url = {https://github.com/barometech/gpt2-tool-call}
}
Author: Попович Павел Дмитриевич («Техножнец») — Pavel D. Popovich (alias Tekhnozhrets).
Кратко: GPT-2 124M умеет вызывать функции (tool calling). Два production-чекпоинта, оба воспроизводимы на CPU. Adapter (250K обучаемых параметров) → 50% на BFCL v4. Full fine-tune (124M обучаемых, 68 минут на CPU) → 92% на 690-задачном свежем бенчмарке без утечки тренировочных данных.
| Модель | Параметров | Время обучения | BFCL v4 OVERALL | Свежий бенч (без leakage) |
|---|---|---|---|---|
| GPT-2 + Adapter (наше) | 125M (250K обучено) | ~1.5ч CPU | 50.0% | 53.5% |
| GPT-2 + Full FT (наше) | 124M (всё обучено) | 68 мин CPU | ~95% на subset'ах из трейна | 92.0% |
| Phi-3-mini Instruct | 3.8B | проприетарно | ~40% | — |
| Llama-3.1 8B Instruct | 8B | проприетарно | ~62% | — |
| Llama-3.1 70B Instruct | 70B | проприетарно | ~70% | — |
| GPT-4-turbo | ~1.7T | проприетарно | ~88% | — |
1. Adapter (weights/adapter_h13_bfcl_ep1.pt, 1 МБ)
- Замороженный GPT-2 124M + steering adapter 250K параметров на 6-м слое
- Добавляет delta-вектор в residual stream через бутылочное горлышко (768→192→96→768)
- Голова отказа (irrelevance detection) сохранена: 40-45% на BFCL irrelevance
- Использовать когда: мало памяти, нужен отказ от ненужных вызовов, минимум обучения
2. Full FT (weights/gpt2_ft_final.pt, 475 МБ — через Git LFS)
- Все 124M параметров GPT-2 обновлены
- 1500 mixed примеров (500 BFCL + 500 Glaive + 500 xLAM holdout slice), 1 эпоха
- LR 1e-5, AdamW, grad_accum=4, PAD=512
- Использовать когда: нужна максимальная точность на новых функциях
Источник: bfcl_eval/data из gorilla-llm/Berkeley-Function-Calling-Leaderboard
Честная оговорка: BFCL частично был в обучающей выборке обоих чекпоинтов (первые ~125 примеров каждого subset'a использовались для обучения). То есть цифры BFCL v4 ниже с утечкой.
Мы сгенерировали 690 тестовых tool-call задач через 9 параллельных агентов Claude Opus 4.7 в 9 нишевых доменах (биология, промышленность, история/культура, материаловедение, нишевая техника + multiple-choice + parallel + irrelevance). Все имена функций новые — без пересечения с BFCL/Glaive/xLAM.
| Задача | plain GPT-2 | adapter | full FT | FT+adapter |
|---|---|---|---|---|
| simple (n=450) | 0.0% | 61.6% | 98.2% | 80.2% |
| multiple (n=80) | 0.0% | 2.5% | 62.5% | 55.0% |
| parallel (n=80) | 0.0% | 16.2% | 88.8% | 88.8% |
| irrelevance (n=80) | 100.0% ** | 96.2% | 90.0% | 53.8% |
| OVERALL (n=690) | 11.6% | 53.5% | 92.0% | 75.2% |
** plain GPT-2 «100%» на irrelevance — это методологический артефакт: модель никогда не выдаёт валидное имя функции, поэтому формально проходит refusal-check.
git clone https://github.com/barometech/gpt2-tool-call
cd gpt2-tool-call
git lfs pull # подтянуть gpt2_ft_final.pt (475 МБ)
pip install torch==2.4.0+cpu fastapi numpy safetensors tokenizers \
--index-url https://download.pytorch.org/whl/cpu --extra-index-url https://pypi.org/simple
# Скачать базовые веса GPT-2
huggingface-cli download openai-community/gpt2 --local-dir weights/base_gpt2- Контекст ≤ 1024 токенов (нативно у GPT-2). Эксперименты с Position Interpolation не сохранили качество надёжно.
- Многошаговое рассуждение: GPT-2 124M не может декомпозировать многошаговые научные задачи с богатым набором tools. Работает хорошо для одного tool call, ломается на 5-шаговых workflow.
- Ансамбль клонов = ноль выгоды. Тестили 10× workers + 1 orchestrator (одинаковые FT-веса, greedy decode). Результат: 100% единогласное голосование, та же точность что у одной модели, 7× медленнее. Реальному ансамблю нужна разнородность (разные FT-прогоны или sampling temperatures).
- Только английский. Многоязычность не тестировалась.
- Генерация кода: GPT-2 124M не способен писать осмысленный код.
src/ — код
integrated_gpt2_torch.py — GPT-2 backbone (чистый torch)
steering_v2.py — архитектура adapter
long_context_pi_chunk.py — Position Interpolation helper
train_ft.py — full FT (~68 мин)
train_adapter.py — adapter training (~1.5ч)
bench_ft.py — бенч FT на BFCL v4
bench_fresh_4way.py — бенч plain/adapter/ft/ft+adapter на 690 fresh items
bench_bfcl_v4.py — бенч adapter на полном BFCL v4
weights/ — веса
adapter_h13_bfcl_ep1.pt — 1 МБ — adapter (steering + classifier heads)
adapter_torch_EN_BFCL.npz — 700 КБ — стартовые веса классификатора
gpt2_ft_final.pt — 475 МБ — full FT модель (Git LFS)
bench/ — данные свежего бенчмарка
fresh_bench_*.json — 9 файлов, 690 items total — сгенерировано Claude Opus 4.7
results/ — сырые результаты
h13_bfcl_v4.json — BFCL v4 для adapter
fresh_4way.json — 4-way fresh bench
MIT. Используй для research, коммерции, чего угодно. Без гарантий.