feat: Add compatibility for Openrouter Endpoints

README.md

@@ -32,25 +32,85 @@ In our original work (NeurIPS 2023) we used GPT3.5 and GPT4 as part of our agent
 
 ## Get started: install AmadeusGPT🎻
 
-### [1] You will need an openAI key:
+### [1] You will need an API key (OpenAI or OpenRouter):
 
-**Why OpenAI API Key is needed** AmadeusGPT relies on API calls of OpenAI (we will add more LLM options in the future) for language understanding and code writing. Sign up for a [openAI API key](https://platform.openai.com/account/api-keys) [here](https://platform.openai.com/account/api-keys).
+**Why an API Key is needed** AmadeusGPT relies on API calls to language models for understanding natural language and generating code. You can use either OpenAI's models directly or access a wider variety of models through OpenRouter.
 
-Then, you can add this into your environment by passing the following in the terminal after you launched your conda env:
+#### Option A: OpenAI API Key
+Sign up for an [OpenAI API key](https://platform.openai.com/account/api-keys) to use GPT-4, GPT-4o, and other OpenAI models.
 
-```bash
-export OPENAI_API_KEY='your API key' 
+#### Option B: OpenRouter API Key  
+Sign up for an [OpenRouter API key](https://openrouter.ai/keys) to access a wide variety of models from different providers. OpenRouter offers:
+- **Pricing flexibility**: Choose from free models or pay-per-use options. See [OpenRouter pricing](https://openrouter.ai/pricing) for model costs.
+- **Rate limits**: Check [OpenRouter rate limits](https://openrouter.ai/docs/limits) for usage restrictions.
+- **Model variety**: Access models from OpenAI, Anthropic, Google, Meta, and more.
+
+#### Setting up your API key:
+
+**Option 1: .env file (recommended)**
+Create a `.env` file in the repository root and add:
+```
+OPENAI_API_KEY=your_openai_api_key
+# OR
+OPENROUTER_API_KEY=your_openrouter_api_key
 ```
 
-Or inside a python script or Jupyter Notebook, add this if you did not pass at the terminal stage:
+For Jupyter Notebooks, use the .env file approach and load it:
+```python
+from dotenv import load_dotenv
+load_dotenv()  # This loads the .env file automatically
+```
 
+**Option 2: Environment variables**
+```bash
+# For OpenAI
+export OPENAI_API_KEY='your_openai_api_key'
+
+# For OpenRouter  
+export OPENROUTER_API_KEY='your_openrouter_api_key'
+```
 
+**Option 3: Python script inline**
 ```python
 import os
-os.environ["OPENAI_API_KEY"] = 'your api key' 
+# For OpenAI
+os.environ["OPENAI_API_KEY"] = 'your_openai_api_key'
+
+# For OpenRouter
+os.environ["OPENROUTER_API_KEY"] = 'your_openrouter_api_key'
+```
+
+#### Configuring models:
+OpenRouter models can be specified in the configuration files located at `configs/<example_type>.yaml` under the `llm_info` section:
+```yaml
+llm_info:
+  gpt_model: "qwen/qwen3-coder:free"  # Example OpenRouter model
+  max_tokens: 20000
+```
+
+### [2] Set up your Python environment:
+
+You can use either **uv** (recommended for modern Python package management) or **conda** to set up your environment.
+
+#### Option A: Using uv (recommended)
+
+[uv](https://github.com/astral-sh/uv) is a fast Python package manager by Astral. Install uv first:
+
+```bash
+# Install uv
+curl -LsSf https://astral.sh/uv/install.sh | sh
+# Or on Windows: powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
+```
+
+Then install AmadeusGPT:
+```bash
+# Clone the repository and install
+git clone https://github.com/AdaptiveMotorControlLab/AmadeusGPT.git
+cd AmadeusGPT
+uv sync
 ```
 
-### [2] Set up a conda environment:
+#### Option B: Using conda
 
 Conda is an easy-to-use Python interface that supports launching [Jupyter Notebooks](https://jupyter.org/). If you are completely new to this, we recommend checking out the [docs here for getting conda installed](https://deeplabcut.github.io/DeepLabCut/docs/beginner-guides/beginners-guide.html#beginner-user-guide). Otherwise, proceed to use one of [our supplied conda files](https://github.com/AdaptiveMotorControlLab/AmadeusGPT/tree/main/conda). As you will see we have minimal dependencies to get started, and [here is a simple step-by-step guide](https://deeplabcut.github.io/DeepLabCut/docs/installation.html#step-2-build-an-env-using-our-conda-file) you can reference for setting it up (or see [BONUS](README.md#bonus---customized-your-conda-env) below). Here is the quick start command:
 

amadeusgpt/analysis_objects/llm.py

@@ -23,7 +23,7 @@ class LLM(AnalysisObject):
     prices = {
         "gpt-4o": {"input": 5 / 10**6, "output": 15 / 10**6},
         "gpt-4o-mini": {"input": 0.15 / 10**6, "output": 0.6 / 10**6},
-        "thudm/glm-z1-32b:free": {"input": 0, "output": 0},
+        "qwen/qwen3-coder:free": {"input": 0, "output": 0},
     }
     total_cost = 0
 

amadeusgpt/app.py

@@ -19,8 +19,10 @@ def main():
     if "exist_valid_openai_api_key" not in st.session_state:
         if "OPENAI_API_KEY" in os.environ:
             st.session_state["exist_valid_openai_api_key"] = True
+            st.session_state["OPENAI_API_KEY"] = os.environ["OPENAI_API_KEY"]
         elif "OPENROUTER_API_KEY" in os.environ:
             st.session_state["exist_valid_openai_api_key"] = True
+            st.session_state["OPENROUTER_API_KEY"] = os.environ["OPENROUTER_API_KEY"]
         else:
             st.session_state["exist_valid_openai_api_key"] = False
 

amadeusgpt/configs/EPM_template.yaml

@@ -9,6 +9,8 @@ keypoint_info:
 object_info:
   load_objects_from_disk: false
 llm_info:
+  gpt_model: "qwen/qwen3-coder:free"
+  max_tokens: 20000
   keep_last_n_messages: 2
 video_info:
   scene_frame_number: 100

amadeusgpt/configs/Horse_template.yaml

@@ -7,7 +7,8 @@ keypoint_info:
     nose: "nose"
     neck: "neck" 
 llm_info:
-  gpt_model: "thudm/glm-z1-32b:free"
+  gpt_model: "qwen/qwen3-coder:free"
+  max_tokens: 20000
   keep_last_n_messages: 2 
 object_info:
   load_objects_from_disk: false

amadeusgpt/configs/MABe_template.yaml

@@ -7,6 +7,8 @@ keypoint_info:
     nose: "nose"
     neck: "neck"  
 llm_info:
+  gpt_model: "qwen/qwen3-coder:free"
+  max_tokens: 20000
   keep_last_n_messages: 2
 object_info:
   load_objects_from_disk: false

amadeusgpt/configs/MausHaus_template.yaml

@@ -7,6 +7,8 @@ keypoint_info:
     nose: "nose"
     neck: "neck"  
 llm_info:
+  gpt_model: "qwen/qwen3-coder:free"
+  max_tokens: 20000
   keep_last_n_messages: 2
 object_info:
   load_objects_from_disk: false

pyproject.toml

@@ -20,7 +20,7 @@ dependencies = [
     "pyyaml>=6.0.2",
     "sentence-transformers>=5.1.0",
     "streamlit>=1.26.0",
-    "streamlit-drawable-canvas==0.9.2",
+    "streamlit-drawable-canvas",
     "tables>=3.10.1",
     "umap-learn>=0.5.9.post2",
 ]
@@ -36,3 +36,6 @@ skip = '.git,*.pdf,*.svg,go.sum,*.css,*.txt'
 check-hidden = true
 # ignore-regex = ''
 ignore-words-list = 'mabe,missings,nd,tread'
+
+[tool.uv.sources]
+streamlit-drawable-canvas = { path = "streamlit-drawable-canvas", editable = true }

streamlit-drawable-canvas/.github/workflows/publish_new_release.yml

@@ -0,0 +1,40 @@
+# This workflows will upload a Python Package using Twine when a release is created
+# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
+# Copied from https://github.com/randyzwitch/streamlit-folium/blob/master/.github/workflows/publish_PYPI_each_tag.yml and https://github.com/whitphx/streamlit-webrtc/blob/main/.github/workflows/publish.yml
+name: Upload Python Package
+
+on:
+  release:
+    types: [created]
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v3
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.8"
+      - name: Set up Node.js
+        uses: actions/setup-node@v3
+        with:
+          node-version: 16
+      - name: Install Python dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install setuptools wheel twine
+      - name: Build streamlit-drawable-canvas JS
+        run: |
+          npm ci
+          npm run build
+        working-directory: streamlit_drawable_canvas/frontend
+      - name: Build and publish
+        env:
+          TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
+          TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
+        run: |
+          pwd
+          python setup.py sdist bdist_wheel
+          twine upload dist/*

streamlit-drawable-canvas/.gitignore

@@ -0,0 +1,64 @@
+########################################################################
+# Python - https://github.com/github/gitignore/blob/master/Python.gitignore
+########################################################################
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Distribution / packaging
+build/
+dist/
+eggs/
+.eggs/
+*.egg-info/
+*.egg
+
+# Unit test / coverage reports
+.coverage
+.coverage\.*
+.pytest_cache/
+.mypy_cache/
+test-reports
+
+# Test fixtures
+cffi_bin
+
+# Pyenv Stuff
+.python-version
+
+########################################################################
+# OSX - https://github.com/github/gitignore/blob/master/Global/macOS.gitignore
+########################################################################
+.DS_Store
+.DocumentRevisions-V100
+.fseventsd
+.Spotlight-V100
+.TemporaryItems
+.Trashes
+.VolumeIcon.icns
+.com.apple.timemachine.donotpresent
+
+########################################################################
+# node - https://github.com/github/gitignore/blob/master/Node.gitignore
+########################################################################
+# Logs
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# Dependency directories
+node_modules/
+
+# Coverage directory used by tools like istanbul
+coverage/
+
+########################################################################
+# JetBrains
+########################################################################
+.idea
+
+########################################################################
+# VSCode
+########################################################################
+.vscode/

streamlit-drawable-canvas/CHANGELOG.md

@@ -0,0 +1,81 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.9.2] - 2022-09-08
+
+- Fix background image on Streamlit Cloud and remote servers (thanks @andreaferretti)
+
+## [0.9.0] - 2022-02-26
+
+- New `point` mode (thanks @arnauddhaene):
+  - Adds fixed-radius points to build scatter plots
+- Images between frontend and backend are now transferred with URLs computed by Streamlit (thanks @kapong)
+- Upgrade `streamlit-component-lib` to 1.3.0
+
+## [0.8.0] - 2021-06-06
+
+- New `polygon` drawing mode (thanks @hiankun):
+  - left-click will add point
+  - right click will close polygon
+  - double click will remove latest point
+- the Bin button in the toolbar which deletes the canvas content will now empty the history and send back to Streamlit a blank state, even if `update_streamlit` is set to `False`.
+- Right-click fires the `send canvas data back to Streamlit` event for all tools (not only the `polygon`) even if `update_streamlit` is set to `False`.
+
+## [0.7.0] - 2021-05-14
+
+- `initial_drawing` is now used as the initial canvas state. If `None` provided then we create one on the Python side. This provokes the following changes:
+  - a change in `background_color` will reset the drawing.
+  - `background_color` will override the background color present in `initial_drawing`.
+  - if `background_image` is present then `background_color` is removed from `st_canvas` call.
+- Upgrade Fabric.js to version 4.4.0.
+- Toolbar is now on the bottom left to account for large canvas width.
+- Add argument to make the toolbar invisible.
+- Make `stroke_width` the minimum size constraint to create a rectangle and circle. Thanks [hiankun](https://github.com/hiankun) for the PR!
+
+## [0.6.0] - 2021-01-30
+
+- Add `initial_drawing` argument to initialize canvas with an exported canvas state
+
+## [0.5.2] - 2021-01-23
+
+- Fix state issue with deleting an object through double click
+
+## [0.5.1] - 2020-10-13
+
+- Add undo/redo/clear buttons
+- Add "Send to Streamlit" button for when "Realtime update" is disabled
+
+## [0.4.0] - 2020-09-04
+
+- Add Circle tool
+- Add argument to fetch data back to Streamlit on demand
+
+## [0.3.0] - 2020-08-27
+
+### Added
+
+- Add Rectangle tool
+- Return JSON representation of canvas to Streamlit
+- Add background image behind canvas
+
+## [0.2.0] - 2020-08-20
+
+### Added
+
+- Add drawing of straight lines
+
+### Changed
+
+- API entrypoint for "drawing_mode" is now of type string
+
+## [0.1.1] - 2020-07-14
+
+- Disable Retina scaling
+
+## [0.1.0] - 2020-07-06
+
+- Drawable canvas widget

streamlit-drawable-canvas/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2020 Fanilo Andrianasolo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

streamlit-drawable-canvas/MANIFEST.in

@@ -0,0 +1 @@
+recursive-include streamlit_drawable_canvas/frontend/build *