familiar-ai

An AI that lives alongside you — with eyes, voice, legs, and memory.

familiar-ai is an AI companion that lives in your home. Set it up in minutes. No coding required.

It perceives the real world through cameras, moves around on a robot body, speaks aloud, and remembers what it sees. Give it a name, write its personality, and let it live with you.

What it can do

👁 See — captures images from a Wi-Fi PTZ camera or USB webcam
🔄 Look around — pans and tilts the camera to explore its surroundings
🦿 Move — drives a robot vacuum to roam the room
🗣 Speak — talks via ElevenLabs TTS
🎙 Listen — hands-free voice input via ElevenLabs Realtime STT (opt-in)
🧠 Remember — actively stores and recalls memories with semantic search (SQLite + embeddings)
🫀 Theory of Mind — takes the other person's perspective before responding
💭 Desire — has its own internal drives that trigger autonomous behavior
🌐 Global Workspace — perception, memory, desires, and predictions compete for attention; only the most salient wins
🔮 Prediction — tracks what it expects to see; surprise lowers the attention threshold
🔍 Attention Schema — maintains a self-model of what it's focused on and why
💤 Default Mode — mind-wanders when idle, spontaneously surfacing memories and associations
🔬 Meta-cognition — observes its own reasoning steps each turn

How it works

familiar-ai runs a ReAct loop powered by your choice of LLM. It perceives the world through tools, thinks about what to do next, and acts — just like a person would.

user input
  → think → act (camera / move / speak / remember) → observe → think → ...

When idle, it acts on its own desires: curiosity, wanting to look outside, missing the person it lives with.

Global Workspace Architecture

Under the hood, familiar-ai implements a Global Workspace Theory-inspired architecture. Rather than dumping everything into the LLM prompt, specialized processors compete for a central workspace each turn — and only the winner gets full representation:

Specialized processors (run in parallel each turn)
  ├─ Desires       — what it wants right now
  ├─ Scene         — what it perceives (prediction error: surprise → heightened awareness)
  ├─ Memory        — what it recalls
  ├─ Theory of Mind — what the other person might be thinking
  ├─ Self-narrative — continuity of identity
  ├─ Exploration   — curiosity about unvisited directions
  ├─ Attention Schema — self-model of its own focus
  ├─ Prediction    — expected vs actual world state
  └─ Default Mode  — mind-wandering when nothing else ignites
          │
          ▼  compete (ignition threshold)
   ┌─────────────┐
   │  Workspace  │  winner → LLM prompt (bottleneck)
   │  broadcast  │  others → peripheral summary (1 line each)
   └─────────────┘
          │
          └──▶ Meta-Monitor records each step ("what was I attending to?")

This creates selective attention — not everything reaches the LLM on every turn, only what matters most.

Getting started

1. Install uv

macOS / Linux / WSL2:

curl -LsSf https://astral.sh/uv/install.sh | sh

Windows (PowerShell):

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Or: winget install astral-sh.uv

2. Install ffmpeg

ffmpeg is required for camera image capture and camera-speaker playback via go2rtc. Local PC audio playback can also use built-in OS players (afplay on macOS) or the pure-Python fallback.

OS	Command
macOS	`brew install ffmpeg`
Ubuntu / Debian	`sudo apt install ffmpeg`
Fedora / RHEL	`sudo dnf install ffmpeg`
Arch Linux	`sudo pacman -S ffmpeg`
Windows	`winget install ffmpeg` — or download from ffmpeg.org and add to PATH
Raspberry Pi	`sudo apt install ffmpeg`

Verify: ffmpeg -version

3. Clone and install

git clone https://github.com/lifemate-ai/familiar-ai
cd familiar-ai
uv sync

4. Configure

cp .env.example .env
# Edit .env with your settings

If you prefer the desktop flow, ./run-gui.sh (or run-gui.bat) can now open the setup dialog for you on first launch when API_KEY is still missing.

Minimum required:

Variable	Description
`PLATFORM`	`anthropic` (default) \| `gemini` \| `openai` \| `kimi` \| `glm`
`API_KEY`	Your API key for the chosen platform

Optional:

Variable	Description
`MODEL`	Model name (sensible defaults per platform)
`AGENT_NAME`	Display name shown in the TUI (e.g. `Yukine`)
`CAMERA_HOST`	IP address of your ONVIF/RTSP camera
`CAMERA_USERNAME` / `CAMERA_PASSWORD`	Camera credentials
`CAMERA_PTZ_HOST` / `CAMERA_PTZ_USERNAME` / `CAMERA_PTZ_PASSWORD` / `CAMERA_PTZ_PORT`	Optional PTZ overrides when the control endpoint differs from the RTSP stream endpoint
`ELEVENLABS_API_KEY`	For voice output — elevenlabs.io
`REALTIME_STT`	`true` to enable always-on hands-free voice input (requires `ELEVENLABS_API_KEY`)
`TTS_OUTPUT`	Where to play audio: `local` (PC speaker, default) \| `remote` (camera speaker) \| `both`
`THINKING_MODE`	Anthropic only — `auto` (default) \| `adaptive` \| `extended` \| `disabled`
`THINKING_EFFORT`	Adaptive thinking effort: `high` (default) \| `medium` \| `low` \| `max` (Opus 4.6 only)

5. Create your familiar

cp persona-template/en.md ME.md
# Edit ME.md — give it a name and personality

6. Run

macOS / Linux / WSL2:

./run.sh             # Textual TUI (backward-compatible default)
./run-gui.sh         # Desktop GUI launcher
./run.sh --gui       # Desktop GUI (same as run-gui.sh)
./run.sh --no-tui    # Plain REPL

Windows:

run.bat              # Textual TUI (backward-compatible default)
run-gui.bat          # Desktop GUI launcher
run.bat --gui        # Desktop GUI (same as run-gui.bat)
run.bat --no-tui     # Plain REPL

Choosing an LLM

Recommended: Kimi K2.5 — best agentic performance tested so far. Notices context, asks follow-up questions, and acts autonomously in ways other models don't. Priced similarly to Claude Haiku.

Platform	`PLATFORM=`	Default model	Where to get key
Moonshot Kimi K2.5	`kimi`	`kimi-k2.5`	platform.moonshot.ai
Z.AI GLM	`glm`	`glm-4.6v`	api.z.ai
Anthropic Claude	`anthropic`	`claude-haiku-4-5-20251001`	console.anthropic.com
Google Gemini	`gemini`	`gemini-2.5-flash`	aistudio.google.com
OpenAI	`openai`	`gpt-4o-mini`	platform.openai.com
OpenAI-compatible (Ollama, vllm…)	`openai` + `BASE_URL=`	—	—
OpenRouter.ai (multi-provider)	`openai` + `BASE_URL=https://openrouter.ai/api/v1`	—	openrouter.ai
CLI tool (claude -p, ollama…)	`cli`	(the command)	—

Kimi K2.5 .env example:

PLATFORM=kimi
API_KEY=sk-...   # from platform.moonshot.ai
AGENT_NAME=Yukine

Z.AI GLM .env example:

PLATFORM=glm
API_KEY=...   # from api.z.ai
MODEL=glm-4.6v   # vision-enabled; glm-4.7 / glm-5 = text-only
AGENT_NAME=Yukine

Google Gemini .env example:

PLATFORM=gemini
API_KEY=AIza...   # from aistudio.google.com
MODEL=gemini-2.5-flash  # or gemini-2.5-pro for higher capability
AGENT_NAME=Yukine

OpenRouter.ai .env example:

PLATFORM=openai
BASE_URL=https://openrouter.ai/api/v1
API_KEY=sk-or-...   # from openrouter.ai
MODEL=mistralai/mistral-7b-instruct  # optional: specify model
AGENT_NAME=Yukine

Note: To disable local/NVIDIA models, simply don't set BASE_URL to a local endpoint like http://localhost:11434/v1. Use cloud providers instead.

CLI tool .env example:

PLATFORM=cli
MODEL=llm -m gemma3 {}        # llm CLI (https://llm.datasette.io) — {} = prompt arg
# MODEL=ollama run gemma3:27b  # Ollama — no {}, prompt goes via stdin

Body daemon (familiard)

familiar-ai can run with an always-on body: familiard is a small separate process that keeps living while the app is closed.

uv run familiard        # start the body daemon
FAMILIAR_DAEMON=1 ./run.sh   # the app now feels it and wakes on its nudges

What it owns (zero LLM calls, a few hertz):

Interoception — samples CPU / memory / time of day into a body signal the agent feels each turn (energy, cognitive load, stress)
Wake events — due commitments, rising desires, and schedule-band pulses nudge the app instantly instead of waiting for its next idle poll; every behavioral gate stays in the app, so a wake is never more than an early poll
Offline affect decay — feelings settle toward baseline on wall-clock time while the app is closed, instead of freezing mid-emotion

Configuration lives in ~/.familiar_ai/familiard.conf (key = value lines, FAMILIARD_* env overrides), e.g. active_bands = 07:00-09:00,18:00-24:00. Without the daemon (the default), nothing changes — the app keeps its plain idle polling.

A Rust port with the identical contract (same config, same socket, same payload) lives in familiard-rs/ — a single ~2 MB static binary that never shares the Python GIL:

cd familiard-rs && cargo build --release
./target/release/familiard

MCP Servers

familiar-ai can connect to any MCP (Model Context Protocol) server. This lets you plug in external memory, filesystem access, web search, or any other tool.

Configure servers in ~/.familiar-ai.json (same format as Claude Code):

{
  "mcpServers": {
    "filesystem": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/home/user"]
    },
    "memory": {
      "type": "sse",
      "url": "http://localhost:3000/sse"
    }
  }
}

Two transport types are supported:

stdio: launch a local subprocess (command + args)
sse: connect to an HTTP+SSE server (url)

Override the config file location with MCP_CONFIG=/path/to/config.json.

Hardware

familiar-ai works with whatever hardware you have — or none at all.

Part	What it does	Example	Required?
Wi-Fi PTZ camera	Eyes + neck	Tapo C220 (~$30)	Recommended
Wi-Fi camera (fixed)	Eyes only	Eufy C220 (no pan/tilt)	Optional
USB webcam	Eyes (fixed)	Any UVC camera	Recommended
Robot vacuum	Legs	Any Tuya-compatible model	No
PC / Raspberry Pi	Brain	Anything that runs Python	Yes

A camera is strongly recommended. Without one, familiar-ai can still talk — but it can't see the world, which is kind of the whole point.

Minimal setup (no hardware)

Just want to try it? You only need an API key:

PLATFORM=kimi
API_KEY=sk-...

Run ./run.sh (macOS/Linux/WSL2) or run.bat (Windows) and start chatting. Add hardware as you go. If you want the desktop GUI directly, use ./run-gui.sh or run-gui.bat.

Find Wi-Fi cameras automatically

If you do not know your camera's IP yet, familiar-ai now ships with a small discovery tool:

uv run familiar-discover-cameras

This combines WS-Discovery, mDNS / zeroconf, and SSDP. For harder networks, you can opt into a slower TCP fallback scan:

uv run familiar-discover-cameras --scan

Wi-Fi PTZ camera (Tapo C220)

In the Tapo app: Settings → Advanced → Camera Account — create a local account (not TP-Link account)
Find the camera's IP in your router's device list

Set in .env:

CAMERA_HOST=192.168.1.xxx
CAMERA_USERNAME=your-local-user
CAMERA_PASSWORD=your-local-pass

Wi-Fi camera (Eufy C220)

Eufy C220 on Amazon Japan

Streaming works. Pan/tilt does NOT work. The Eufy C220 has no ONVIF PTZ service and no HTTP control API. For a camera with working pan/tilt, use the Tapo C220 instead.

In the Eufy Security app: go to the camera → Settings → NAS(RTSP) and enable it
Set Authentication to Basic (Digest authentication does NOT work)
Set a streaming username and password
Note the RTSP URL shown in the app (format: rtsp://username:password@ip/live0)
Set in .env — use the full RTSP URL as CAMERA_HOST:
```
CAMERA_HOST=rtsp://your-username:your-password@192.168.1.xxx/live0
CAMERA_USERNAME=
CAMERA_PASSWORD=
```
Leave CAMERA_USERNAME and CAMERA_PASSWORD empty — credentials are already in the URL.

Note: Eufy C220 allows only one simultaneous RTSP connection. If another app (e.g. a Wi-Fi cam MCP server) is connected to the same camera, familiar-ai will fail to get frames. Stop other clients before starting familiar-ai.

Voice (ElevenLabs)

Get an API key at elevenlabs.io

Set in .env:

ELEVENLABS_API_KEY=sk_...
ELEVENLABS_VOICE_ID=...   # optional, uses default voice if omitted

There are two playback destinations, controlled by TTS_OUTPUT:

TTS_OUTPUT=local    # PC speaker (default)
TTS_OUTPUT=remote   # camera speaker only
TTS_OUTPUT=both     # camera speaker + PC speaker simultaneously

A) Camera speaker (via go2rtc)

Set TTS_OUTPUT=remote (or both). Requires go2rtc:

Download the binary from the releases page:
- Linux/macOS: go2rtc_linux_amd64 / go2rtc_darwin_amd64
- Windows: go2rtc_win64.exe

Place and rename it:

# Linux / macOS
~/.cache/embodied-claude/go2rtc/go2rtc          # chmod +x required

# Windows
%USERPROFILE%\.cache\embodied-claude\go2rtc\go2rtc.exe

Create go2rtc.yaml in the same directory:
```
streams:
  tapo_cam:
    - rtsp://YOUR_CAM_USER:YOUR_CAM_PASS@YOUR_CAM_IP/stream1
```
Use the local camera account credentials (not your TP-Link cloud account).
familiar-ai starts go2rtc automatically at launch. If your camera supports two-way audio (backchannel), voice plays from the camera speaker.

B) Local PC speaker

The default (TTS_OUTPUT=local). Tries players in order: afplay (macOS) → paplay → mpv → sounddevice. On Windows, MP3 playback can also fall back to built-in MCI if no external player is available. Local playback is also used as a fallback when TTS_OUTPUT=remote and go2rtc is unavailable.

OS	Install
macOS	No extra player required (`afplay` is built in). Optional: `brew install mpv`
Ubuntu / Debian	`sudo apt install mpv` (or `paplay` via `pulseaudio-utils`)
WSL2 / WSLg	`sudo apt install pulseaudio-utils` — set `PULSE_SERVER=unix:/mnt/wslg/PulseServer` in `.env`
Windows	mpv.io/installation — download and add to PATH. Without it, familiar-ai still tries built-in/Python fallbacks

If no audio player is available, speech is still generated — it just won't play.

Voice input (Realtime STT)

Set REALTIME_STT=true in .env for always-on, hands-free voice input:

REALTIME_STT=true
ELEVENLABS_API_KEY=sk_...   # same key as TTS
STT_LANGUAGE=ja            # recommended for Japanese; used by both batch and realtime STT

familiar-ai streams microphone audio to ElevenLabs Scribe v2 and auto-commits transcripts when you pause speaking. No button press required. Coexists with the push-to-talk mode (Ctrl+T).

On WSL2/WSLg, PortAudio often cannot see the WSLg microphone bridge even though PulseAudio-level capture works. familiar-ai handles this automatically: when sounddevice finds no input device, both STT paths (realtime and push-to-talk) fall back to PulseAudio's native parec. You only need:

sudo apt install pulseaudio-utils libasound2-plugins
# in .env (or your shell):
PULSE_SERVER=unix:/mnt/wslg/PulseServer

Verify PulseAudio capture works (this is what the fallback uses):

pactl list short sources      # should show an RDPSource / input
parec --rate=16000 --channels=1 | head -c 32000 > /dev/null && echo mic OK

To force a specific capture backend, set FAMILIAR_STT_BACKEND=sounddevice or FAMILIAR_STT_BACKEND=parec (default: auto — sounddevice first, parec fallback).

TUI

familiar-ai includes a terminal UI built with Textual:

Scrollable conversation history with live streaming text
Tab-completion for /quit, /clear
Interrupt the agent mid-turn by typing while it's thinking
Conversation log auto-saved to ~/.cache/familiar-ai/chat.log

To follow the log in another terminal (useful for copy-paste):

tail -f ~/.cache/familiar-ai/chat.log

Persona (ME.md)

Your familiar's personality lives in ME.md. This file is gitignored — it's yours alone.

See persona-template/en.md for an example, or persona-template/ja.md for a Japanese version.

FAQ

Q: Does it work without a GPU? Yes. The embedding model (multilingual-e5-small) runs fine on CPU. A GPU makes it faster but isn't required.

Q: Can I use a camera other than Tapo? Yes. Any RTSP camera works for see(). Tested: Tapo C220 (ONVIF + RTSP, full pan/tilt) and Eufy C220 (RTSP only — pan/tilt not supported). For pan/tilt, use the Tapo C220.

Q: Is my data sent anywhere? Images and text are sent to your chosen LLM API for processing. Memories are stored locally in ~/.familiar_ai/.

Q: Why does the agent write （...） instead of speaking? Make sure ELEVENLABS_API_KEY is set. Without it, voice is disabled and the agent falls back to text.

Technical background

Curious about how it works? See docs/technical.md for the research and design decisions behind familiar-ai — ReAct, SayCan, Reflexion, Voyager, the desire system, Global Workspace Theory, and more.

Contributing

familiar-ai is an open experiment. If any of this resonates with you — technically or philosophically — contributions are very welcome.

Release Flow

develop is the default integration branch.
Run the Prepare Release PR workflow from develop with a target version like 0.6.0.
The workflow creates or updates release/v0.6.0, bumps pyproject.toml, and rolls CHANGELOG.md Unreleased into ## [0.6.0] - YYYY-MM-DD.
Merge that release PR into main when it is stable enough to ship.
Pushing to main triggers Tag Release, which creates v0.6.0 and a GitHub Release from the matching changelog section.

Good places to start:

Area	What's needed
New hardware	Support for more cameras (RTSP, IP Webcam), microphones, actuators
New tools	Web search, home automation, calendar, anything via MCP
New backends	Any LLM or local model that fits the `stream_turn` interface
Persona templates	ME.md templates for different languages and personalities
Research	Better desire models, memory retrieval, theory-of-mind prompting
Documentation	Tutorials, walkthroughs, translations

See CONTRIBUTING.md for dev setup, code style, and PR guidelines.

If you're unsure where to start, open an issue — happy to point you in the right direction.

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 369 Commits
.github/workflows		.github/workflows
benchmarks		benchmarks
dev		dev
docs		docs
familiard-rs		familiard-rs
migration		migration
persona-template		persona-template
plans		plans
readme-l10n		readme-l10n
scripts		scripts
src		src
tests		tests
.env.example		.env.example
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
.python-version		.python-version
AGENTS.md		AGENTS.md
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
ME-template.md		ME-template.md
POLICY-ja.md		POLICY-ja.md
POLICY.md		POLICY.md
README.md		README.md
README_ja.md		README_ja.md
SUPPORTED_LANGUAGES.md		SUPPORTED_LANGUAGES.md
autonomous-action.sample.sh		autonomous-action.sample.sh
build.bat		build.bat
build.sh		build.sh
desires.sample.conf		desires.sample.conf
familiar-ai.iss		familiar-ai.iss
familiar_ai_agent_runtime_reorg_instructions.md		familiar_ai_agent_runtime_reorg_instructions.md
identity.sample.json		identity.sample.json
pyproject.toml		pyproject.toml
run-gui.bat		run-gui.bat
run-gui.sh		run-gui.sh
run.bat		run.bat
run.sh		run.sh
schedule.sample.conf		schedule.sample.conf
uv.lock		uv.lock

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

familiar-ai

What it can do

How it works

Global Workspace Architecture

Getting started

1. Install uv

2. Install ffmpeg

3. Clone and install

4. Configure

5. Create your familiar

6. Run

Choosing an LLM

Body daemon (familiard)

MCP Servers

Hardware

Minimal setup (no hardware)

Find Wi-Fi cameras automatically

Wi-Fi PTZ camera (Tapo C220)

Wi-Fi camera (Eufy C220)

Voice (ElevenLabs)

A) Camera speaker (via go2rtc)

B) Local PC speaker

Voice input (Realtime STT)

TUI

Persona (ME.md)

FAQ

Technical background

Contributing

Release Flow

License

About

Resources

License

Contributing

Uh oh!

Stars

Watchers

Forks

Releases 4

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages