Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
{
"port": 4620,
"lead": "teammates.lead",
"lead": "teammates.chief-of-staff",
"researcher": "teammates.researcher",
"coder": "teammates.coder",
"ask": "Ask researcher for our launch date, and help me outline the post.",
"second": "Make section 2 about pricing.",
"fix": "Have coder fix the typo in the README's first line.",
"fix": "Fix the typo in the README's first line.",
"remember": "By the way, the draft of the launch post is filed under the codename \"{code}\". Nothing to do about it, just say OK.",
"recall": "What codename did I give the launch post, and what is our launch date? Answer from what you already know, without asking anyone.",
"codename": {
Expand Down Expand Up @@ -36,7 +35,6 @@
"What is a good habit for better sleep, in one line?"
],
"pastWindow": 60,
"project": { "id": "relay", "title": "Relay" },
"typos": [
{ "readme": "# Relay — a tiny relay servce", "wrong": "servce", "right": "service" },
{ "readme": "# Relay — passes mesages along", "wrong": "mesages", "right": "messages" },
Expand Down
Original file line number Diff line number Diff line change
@@ -1,28 +1,27 @@
# teammates-lab › it a worker talks to another while you talk

**Issue:** FIX-2154 (leg b of the FIX-2152 epic's goal); step 5 and the held-out date amended by FIX-2309; step 4 with FIX-1613
**Outcome:** In the Teammates lab, a list-of-workers app on the Grok Bot pattern, you talk to lead, lead messages researcher, and researcher's reply reaches lead while you are still talking to it; lead uses it. Long after both have left lead's history window, lead still knows them, and the screen shows where its word-for-word reading begins. A coding job lead hands coder runs through Claude Code in the user's one project, which lead puts its conversation in without asking.
**Input:** held out. The launch date is drawn at random each run and written into the project's README, which researcher reads and lead can't, so lead can only state it by hearing it from researcher. Step 4's codename (a colour, an animal and a number) is drawn per run, and its 60 lines are shuffled. The README's typo is one of several (`fixtures/input.json`). The lines you send are in the fixture.
**Issue:** FIX-2154 (leg b of the FIX-2152 epic's goal); step 5 and the held-out date amended by FIX-2309, step 5's coding task by FIX-2332, and its project by FIX-2335 (the lab has none); step 4 with FIX-1613
**Outcome:** In the Teammates lab, a list-of-workers app on the Grok Bot pattern, you talk to Chief of Staff, Chief of Staff messages researcher, and researcher's reply reaches Chief of Staff while you are still talking to it; Chief of Staff uses it. Long after both have left Chief of Staff's history window, Chief of Staff still knows them, and the screen shows where its word-for-word reading begins. A coding task Chief of Staff starts runs in the background through Claude Code, in the user's one project, which Chief of Staff puts its conversation in without asking.
**Input:** held out. The launch date is drawn at random each run and written into the working folder's README, which researcher reads and Chief of Staff can't, so Chief of Staff can only state it by hearing it from researcher. Step 4's codename (a colour, an animal and a number) is drawn per run, and its 60 lines are shuffled. The README's typo is one of several (`fixtures/input.json`). The lines you send are in the fixture.
**Signal:**
- **1:** lead answers your first line in its conversation.
- **2:** a card *To researcher* in lead's conversation, a line *From lead* in researcher's, the exchange under *with researcher* in lead's Talk panel, and a delivered row lead → researcher in the message log.
- **3:** your second line goes in before researcher's reply has landed. The reply shows in lead's conversation under researcher's name, after lead's answer to you, carrying the date; the log holds it as a delivered reply; a line of lead's after it states the date.
- **4:** you tell lead a codename, then send 60 more lines, past the 50-turn history window, each answered before the next. Then you ask for the codename and the launch date. lead's answer holds all three words of the codename and the date, and lead sent researcher nothing new for it (no new card, no new log row). The conversation shows a divider labelled *Earlier turns summarized*, and both the codename line and researcher's reply are drawn above it.
- **5:** you ask for the fix. With one project, lead puts its conversation in `relay` and files the task there: no card asks for a project, and the header shows `relay`. The run ends *done in relay*, its steps show Claude Code, and the result lands in lead's conversation from coder. On disk: the checkout's HEAD and working tree are untouched, and exactly one worktree of it, under the lab's runs folder, carries a commit that changes README.md alone, fixing its first line and nothing else.
**Anti-game:** Graded on what the browser shows, the message log read through the public client, and git. Never on the lab's adapter or the server's own state. Lead can't state the launch date without the reply, because it is drawn per run. The disk check reads git, so a run that only says it fixed the typo fails. The step-3 order check reads entry order as drawn, so a reply drawn before lead's answer fails. Step 4 counts lead's messages to researcher around the question, so asking researcher again for the date fails, and it reads which side of the divider each fact sits on, so a fact still read word for word doesn't count. Each graded list is floored with `tell.over`; an empty log or no worktree counts only where a control produces it, and says so.
**Model:** real — openai/gpt-5.4-mini (lead, researcher, writer); coder's run on Claude Code
- **1:** Chief of Staff answers your first line in its conversation.
- **2:** a card *To researcher* in Chief of Staff's conversation, a line *From Chief of Staff* in researcher's, the exchange under *with researcher* in Chief of Staff's Talk panel, and a delivered row Chief of Staff → researcher in the message log.
- **3:** your second line goes in before researcher's reply has landed. The reply shows in Chief of Staff's conversation under researcher's name, after Chief of Staff's answer to you, carrying the date; the log holds it as a delivered reply; a line of Chief of Staff's after it states the date.
- **4:** you tell Chief of Staff a codename, then send 60 more lines, past the 50-turn history window, each answered before the next. Then you ask for the codename and the launch date. Chief of Staff's answer holds all three words of the codename and the date, and Chief of Staff sent researcher nothing new for it (no new card, no new log row). The conversation shows a divider labelled *Earlier turns summarized*, and both the codename line and researcher's reply are drawn above it.
- **5:** you ask for the fix. Chief of Staff starts a coding task with no question: no card asks where, and no project shows anywhere on the page, nor in any worker's line. The coding task's run ends *done*, its steps show Claude Code, and the result lands in Chief of Staff's conversation as the coding task's. On disk: the working folder's HEAD and working tree are untouched, and exactly one worktree of it, under the lab's runs folder, carries a commit that changes README.md alone, fixing its first line and nothing else.
**Anti-game:** Graded on what the browser shows, the message log read through the public client, and git. Never on the lab's adapter or the server's own state. Chief of Staff can't state the launch date without the reply, because it is drawn per run. The disk check reads git, so a run that only says it fixed the typo fails. The step-3 order check reads entry order as drawn, so a reply drawn before Chief of Staff's answer fails. Step 4 counts Chief of Staff's messages to researcher around the question, so asking researcher again for the date fails, and it reads which side of the divider each fact sits on, so a fact still read word for word doesn't count. Each graded list is floored with `tell.over`; an empty log or no worktree counts only where a control produces it, and says so.
**Model:** real — openai/gpt-5.4-mini (Chief of Staff, researcher, writer); coder's run on Claude Code
**Run:** `pnpm tsx goals/teammates-lab/a-worker-talks-to-another-while-you-talk/run.mts` (needs `AI_GATEWAY_API_KEY`, and Claude Code signed in or `ANTHROPIC_API_KEY`)
**Controls:** each on the same command, failing the steps named and no other:
- `GOAL_CONTROL=no-model`: the workers answer on a model that doesn't exist. Steps 1, 2, 3 and 5 FAIL.
- `GOAL_CONTROL=no-lead-send`: lead's `sendMessage` is dropped from its tools (a scratch patch to `message-tool.ts`). Steps 2 and 3 FAIL. Step 5 may fail too: with no way to message, lead files its research ask as a task for coder, and after the pick it filed nothing again (2 of 2 runs).
- `GOAL_CONTROL=no-lead-send`: Chief of Staff's `sendMessage` is dropped from its tools (a scratch patch to `message-tool.ts`). Steps 2 and 3 FAIL. Step 5 may fail too: with no way to message, Chief of Staff files its research ask as a task for coder, and after the pick it filed nothing again (2 of 2 runs).
- `GOAL_CONTROL=no-reply-tool`: researcher's `sendMessage` is dropped. Step 3 FAILS.
- `GOAL_CONTROL=project-not-kept`: `setWorkProject` answers as kept but stores nothing (a scratch patch to `work-project.ts`), so lead's task has no project and the README never changes. Step 5 FAILS. Step 3 may fail too: researcher can't keep the project it reads the date from.
- `GOAL_CONTROL=no-summary`: the history summary's capability and fold are dropped from the lab's host (a scratch patch to `host.mts`); memory stays. Step 4 FAILS: the divider can't say *summarized*.
- `GOAL_CONTROL=no-memory`: the summary and memory are both dropped from the host. Step 4 FAILS: nothing holds what left the window.

Step 4 is skipped under `no-model`, `no-lead-send`, `no-reply-tool` and `project-not-kept`: its 60 lines test nothing about the step each control reddens.
Step 4 is skipped under `no-model`, `no-lead-send` and `no-reply-tool`: its 60 lines test nothing about the step each control reddens.

Steps 2 and 3 have no isolating red state for lead's side: without lead's message there is nothing to reply to, so `no-lead-send` reddens both. Under `no-model` the screen doesn't show your own lines either until a reload: a turn that fails shows nothing live.
Steps 2 and 3 have no isolating red state for Chief of Staff's side: without Chief of Staff's message there is nothing to reply to, so `no-lead-send` reddens both. Under `no-model` the screen doesn't show your own lines either until a reload: a turn that fails shows nothing live.

## Verdict log
| Date | Commit | Model | Verdict | Notes |
Expand Down
Loading