Proposal: Integrate mofa-local-llm into mofa core as built-in local inference backend

[SanchitKS12]

Background

The MoFA GSoC project ideas list includes this task as an open contribution item:

Integrate mofa-local-llm into mofa core as the built-in local inference module. :contentReference[oaicite:4]{index=4}

Goals

• Enable first-class local inference support directly in the mofa core framework
• Define a backend interface / trait in the runtime capable of loading and running models
• Integrate mofa-local-llm as one of the default backends
• Ensure Rust-native APIs and workflows (no external HTTP model servers)

Proposed Approach

• Evaluate the current core mofa inference architecture
• Define or extend an InferenceBackend trait in the core
• Adapt mofa-local-llm to implement this interface
• Add loading, lifecycle management, and inference paths from core into the backend
• Write basic tests demonstrating inference pathways

As far as i can think of it ,inline local LLM support improves performance and makes MoFA more self-contained without external services.

---

Happy to expand the approach and get early feedback before coding.

[BH3GEI]

Sorry for the late response, and thanks for proposing this — the direction is valuable.

Quick context: the project has moved fast recently, and AmosLi (with contributors) has already started some
local-inference work in the main mofa repo. I’m not fully sure yet how much overlap/coverage exists versus
mofa-local-llm.

To keep collaboration focused, we’d like to prioritize new integration work in mofa-org/mofa (main repo).
mofa-local-llm will still be maintained for focused fixes/experiments and components that are not yet
covered in main.

Before starting large implementation, could you help with a short gap analysis:

1. what is already covered in main,
2. what is missing in main but available in mofa-local-llm,
3. what should be upstreamed first.

Then we can split concrete tasks and move faster together.

[SanchitKS12]

Sure thing @BH3GEI , i will do the gap analysis and will let you know about all the update regarding the pointers you mentioned latest by tomorrow night ,and then we can move on with the task distribution.
Excited to work with all !!

[BH3GEI]

Thanks！If you have some doubts, feel free to DM me on discord

[SanchitKS12]

@BH3GEI Thanks for the clarification ,here is a short gap analysis based on comparing mofa (main) and mofa-local-llm :

1. What is already covered in mofa (main) :

• Microkernel architecture with plugin-based LLMCapability
• Standardized LLM abstraction layer
• Workflow engine + agent coordination (chain, parallel, debate, etc.)
• Task scheduling, A2A communication bus
• Persistence, observability, dashboard
• OpenAI API integration (as client)

In short: control plane + orchestration + abstraction layer are already mature.

2. What exists in mofa-local-llm but not in main :

• Concrete local inference engines:
• LlmEngine, VlmEngine, AsrEngine, TtsEngine, ImageGenEngine
• Model lifecycle management (load/scan/persist)
• Model catalog definitions
• HuggingFace download integration
• OpenAI-compatible local server implementation
• GUI layer (egui app)

In short: execution engines + model management layer are implemented there.

3. Suggested Upstream Order :

To avoid duplication and keep architecture clean:

• Define/confirm a stable LocalInferenceBackend (or equivalent) trait in mofa
• Upstream minimal text LLM engine (integrated via LLMCapability)
• Then upstream model lifecycle + download layer
• Later extend to multimodal engines (VLM, ASR, TTS, image-gen)
• This keeps main as the control plane while progressively integrating execution engines.

Happy to refine this further based on how the current local inference work in mofa is structured.

[SanchitKS12]

@BH3GEI please approve this or suggest changes so that i can get started with the actual upstream work.
Thank you.

[MrAnayDongre]

@BH3GEI one thing worth flagging before the trait gets locked — the dispatch in mofa-local-llm goes through an mpsc channel to an OS thread, not through the trait itself. Metal/MLX ops are sync, so the boundary has to stay, which means the LocalInferenceBackend in main can't just be a straightforward async trait. Probably worth deciding that before @SanchitKS12 starts the upstream work, otherwise the interface will need reworking mid-PR.

Also wondering if the model lifecycle stuff (ModelEntry, HF download) could be upstreamed independently first — no inference code, no trait decisions, and it unblocks everything that comes after.

[SanchitKS12]

@MrAnayDongre that’s a really great insight — this was actually one of the main difficulties I was running into as well.

The current mpsc + OS thread dispatch in mofa-local-llm makes the async trait abstraction a bit misleading, since the actual inference boundary is inherently synchronous (especially with MLX/Metal). So a straightforward LocalInferenceBackend async trait might not fully capture the real execution model.

That’s partly why I pushed the CandleProvider PR first which you can view here mofa-org/mofa#1167 it gives a cleaner, fully native integration path (no HTTP hop, no extra threading abstraction leakage) and helps establish what a “correct” provider interface should look like inside MoFA.This would helped me understand the mofa llm as a whole on a better scale and i am still waiting for @BH3GEI response for the same.

My thinking was:

• Validate the native backend pattern (via Candle)
• Get clarity / consensus on the provider interface
• Then refactor mofa-local-llm accordingly and integrate it cleanly, instead of forcing it into an interface that might change mid-way

Also +1 on your suggestion , upstreaming model lifecycle pieces (ModelEntry, HF download, etc.) independently first makes a lot of sense. It’s lower risk, doesn’t depend on trait decisions, and would unblock both Candle and future local backends.
Please make more suggestions like this and review the code if possible ,thank you.