VibePointer

Teaching machines to move the cursor with your vibe.

🚀 Quick Start

0. Install Dependencies

pip install -r requirements.txt

1. Collect Training Data

python collect_data.py

Move your mouse naturally to different targets. Press ESC to stop.

2. Train the Model

python train_model.py
python distill_model.py

3. Optimize for Inference

python optimize_model.py
python optimize_model.py --student

4. Visualize & Test

python visualize_prediction.py
python visualize_prediction.py --student
python visualize_prediction.py --original

🏗️ Architecture

• Input: 12D state (position, velocity, target info, progress)
• Teacher Model: MLP (256→512→256→2) with residual connections
• Student Model: Lightweight MLP (128→128→64→2), ~80% smaller
• Features: Noise injection, progress-aware scheduling, DAgger-style training
• Optimization: BatchNorm fusion, weight folding, dynamic quantization, JIT compilation

📊 Performance

Metric	Value
Hit Rate	>99% (with replanning system)
Avg Error	~4.3 px (+ ~1 px for student model)
Inference Time	<1ms (optimized model on CPU)
Model Size	Teacher: 1.0 MB, Student: 0.1 MB
Training Time	<=10 min (depends on data size and HW performance)

🔧 Key Components

• train_model.py - AdamW optimizer, Huber loss, cosine annealing, residual connections
• distill_model.py - Knowledge distillation for model compression
• optimize_model.py - Post-training optimization (fusion, quantization, JIT)
• visualize_prediction.py - Real-time visualization
• mouse_control_api.py - API for mouse control integration

📈 Advanced Features

• Weighted Loss: Higher penalty for errors near target endpoint
• Noise Scheduling: Reduced randomness as cursor approaches target
• Residual Connections: Improved gradient flow with minimal cost

🧠 Replanning System

Yet another solution for combating error accumulation.

Traditional trajectory prediction models suffer from the Markov assumption: they assume the current state perfectly captures all relevant history. In reality, this assumption breaks down quickly, leading to compounding errors over long trajectories.

The Problem:

• Small prediction errors accumulate over time
• Model "drifts" away from optimal trajectory
• Cursor gets stuck in local minima or diverges completely

Our Solution: The Smart Replanning system continuously monitors trajectory health and triggers automatic recovery:

1. Error Detection: Tracks prediction confidence and deviation metrics
2. Intelligent Triggers: Detects stuck/divergent states before they become critical
3. Seamless Recovery: Resets planning with corrected state while maintaining smoothness
4. Adaptive Thresholds: Adjusts sensitivity based on trajectory progress

Impact:

• Reduces long-trajectory failure rate by ~90%
• Maintains smooth, natural-looking cursor motion
• Works transparently without manual intervention

This makes VibePointer robust in real-world scenarios where perfect Markov assumptions don't hold.

🎯 Use Cases

• Mouse trajectory simulation and prediction
• Human-computer interaction research
• Automated UI testing
• Advanced CAPTCHA bypassing