Local AI Advisor

Inference runtimes

The runtime is the software that actually loads the model and runs it on your GPU or CPU; your AI tool talks to it over an API.

Compatibility matrix

Which GPU backends each runtime supports (best maturity on any OS) and which APIs it exposes to AI tools.
Runtimes by compute backend and API
RuntimeCompute backendsAPIs exposedFormats
MetalCUDAROCmVulkanSYCLCPUOpenAIAnthropicOllama
Ollamamaturematuregoodexperimental—goodGGUF
llama.cpp (llama-server)maturematuregoodgoodexperimentalmatureGGUF
MLX-LMmature—————MLX
LM Studiomaturematuregoodgood—goodGGUF, MLX
vLLMexperimentalmaturegood——experimentalsafetensors
SGLang—maturegood———safetensors
Jangoodgood—good—goodGGUF
LocalAIgoodgoodexperimentalgoodexperimentalgoodGGUF
Lemonade——goodgood—goodGGUF

Maturity: mature production-ready good works well, fewer usersexperimental expect rough edges. These are editorial assessments.

All runtimes

  • Ollama

    Model serverBeginner-friendly

    The easiest way to run local models. One-line install, model library, background server with Ollama, OpenAI- and Anthropic-compatible (v0.14+) APIs. Since v0.35 (Sep 2026) it also serves decision models through /v1/systemone (returns choices/probabilities/scores instead of text); since v0.40.4 (Oct 2026) decision models also work from the CLI (`ollama run --ask`) and on the MLX runner on Apple Silicon. Since v0.40 (Oct 2026), supported model architectures run on the MLX engine by default on Apple Silicon. Since v0.40.6 (Oct 2026) it also reads q2_0 GGUF files. Since v0.40.7 (Oct 2026) it ships the pplx-decider 27B decision model, `ollama show` reports each model's renderer, parser and chat-template source, and speculative decoding infers the draft type automatically. Since v0.40.8 (Oct 2026) it ships Recursive Flint, a 40B-A2.5B open coding model (`ollama run flint`; macOS only at launch, now also Linux and Windows). Since v0.40.9 (Oct 2026) it generates speech with Phonon TTS on Apple Silicon (`ollama run phonon`, OpenAI-compatible /v1/audio/speech) and ships d3 decision models in 2B, 4B, 9B and 27B sizes. Since v0.40.10 (Oct 2026) it ships the Nemotron 3.5 Bolt E4B vision-language model (`ollama run nemotron-3.5-bolt:e4b`, 128K context, DFlash draft for speculative decoding). Since v0.40.11 (Oct 2026) it adds a Phonon voice catalog, Qwen3 Coder and OpenCode fixes, and a llama.cpp bump. Since v0.40.12 (Oct 2026) it ships Liquid AI's LFM3 family (`ollama run lfm3`, 1.7B, 256K context; -vl, -audio and -omni variants) and a 2B d1 decision model, with faster Apple-silicon prompt processing for Gemma 4, Flint and Laguna, DFlash2 speculative-decoding drafts on MLX, and MLX/llama.cpp bumps (Apple hardware only for LFM3/d1 2B at launch).

    Engine
    llama.cpp
    Model formats
    GGUF
    APIs
    Ollama, OpenAI, Anthropic
    Concurrency
    Moderate
    KV cache quantization
    Supported
    Local API server
    Yes

    Concurrency: Parallel slots work, but each extra request noticeably slows every stream. Prompt cache reuse: Fair — reused within a session, sometimes re-processed.

    Ollama: backend maturity per operating system
    BackendmacOSLinuxWindows
    Metal (Apple GPU)mature——
    CUDA (NVIDIA)—maturemature
    ROCm (AMD)—goodgood
    Vulkan—experimentalexperimental
    CPU onlygoodgoodgood
    Documentation(opens in a new tab)Ollama release notes — v0.40.12(opens in a new tab)verified 2026-10-28high confidence

    v0.40.12 (Oct 27): Liquid AI LFM3 family (lfm3 1.7B text, 256K context; lfm3-vl, lfm3-audio, lfm3-omni) and d1 2B decision models, Apple hardware only at launch; faster Apple-silicon prompt processing for Gemma 4/Flint/Laguna, DFlash2 speculative-decoding drafts on MLX, small-image detail fix for compact Gemma 4, ollama launch works with OpenCode v2, MLX and llama.cpp bumps. v0.40.11 (Oct 23, final): Phonon voice catalog, qwen3coder literal function-closing-tag fix, OpenCode launch uses allocated context, vllm-d3 empty-state fix, llama.cpp bump b11845, MLX Phonon spelling. v0.40.10 (Oct 23): Nemotron 3.5 Bolt E4B library model (Gemma 4-based vision-language model, 128K context, DFlash speculative-decoding draft; `ollama run nemotron-3.5-bolt:e4b`) and gemma renderer updates. v0.40.9 (Oct 22): speech generation via Phonon TTS on Apple Silicon (ollama run phonon; /v1/audio/speech endpoint), d3 decision models (vLLM Semantic Router, 2B/4B/9B/27B), Recursive Flint now on Linux and Windows as well as macOS. v0.40.8 (Oct 21): adds Recursive Flint, an Apache 2.0 40B-A2.5B MoE coding model (262K context, reasoning + multi-turn tool use, 54.9% Terminal-Bench 2.1 / 74.2% SWE-bench Verified publisher-reported). v0.40.7 added the pplx-decider 27B decision model, renderer/parser/template_source fields in ollama show, automatic draft-type inference for speculative decoding, and Responses-API fixes for mixed thinking/history. v0.40.6 (Oct 16) added q2_0 GGUF tensor/file types and Clef cloud decision models. v0.40.4 added decision models to the CLI via --ask and decision-model support in the MLX runner.

  • llama.cpp (llama-server)

    Inference engineAdvanced users

    The reference GGUF engine. Widest hardware coverage, fine-grained control (offload, KV quantization, parallel slots), OpenAI- and Anthropic-compatible server. Since Oct 2026 (build 11361) it also serves decision models through /v1/systemone (choices/probabilities/scores in a single forward pass) and reports each model's input/output modalities in GET /v1/models.

    Engine
    llama.cpp
    Model formats
    GGUF
    APIs
    OpenAI, Anthropic
    Concurrency
    Moderate
    KV cache quantization
    Supported
    Local API server
    Yes

    Concurrency: Parallel slots work, but each extra request noticeably slows every stream. Prompt cache reuse: Good — follow-up turns usually skip re-reading the history.

    llama.cpp (llama-server): backend maturity per operating system
    BackendmacOSLinuxWindows
    Metal (Apple GPU)mature——
    CUDA (NVIDIA)—maturemature
    ROCm (AMD)—goodgood
    Vulkan—goodgood
    SYCL / oneAPI (Intel)—experimentalexperimental
    CPU onlymaturematuremature
    Documentation(opens in a new tab)ggml-org — New in llama.cpp: Decision Models (2026-10-02, /v1/systemone)(opens in a new tab)verified 2026-10-16high confidence

    Decision-models serving confirmed; Anthropic Messages API in llama.cpp (2026-01-19) verified earlier. v0.7.0 released 2026-10-15: LiLiCorr speculative decoding, MTP for GLM5-Next, GPU cache for MoE experts (up to ~2.2x decode speedup).

  • MLX-LM

    Inference engineIntermediate

    Apple's MLX framework with the mlx-lm package. Fast on Apple Silicon, uses M5 GPU Neural Accelerators, OpenAI-compatible mlx_lm.server.

    Engine
    mlx
    Model formats
    MLX
    APIs
    OpenAI
    Concurrency
    Mostly sequential
    KV cache quantization
    Supported
    Local API server
    Yes

    Concurrency: Concurrent requests largely queue up; best for one user at a time. Prompt cache reuse: Fair — reused within a session, sometimes re-processed.

    Hardware restriction: Apple Silicon Macs only.

    MLX-LM: backend maturity per operating system
    BackendmacOSLinuxWindows
    Metal (Apple GPU)· accel.mature——
    Documentation(opens in a new tab)mlx-lm server docs (not recommended for production; one request at a time with --kv-bits)(opens in a new tab)verified 2026-09-30high confidence

  • LM Studio

    Desktop appBeginner-friendly

    Desktop app with model browser, chat UI and local server. Runs GGUF via llama.cpp and MLX models on Macs; OpenAI- and Anthropic-compatible endpoints.

    Engine
    llama.cpp
    Model formats
    GGUF, MLX
    APIs
    OpenAI, Anthropic
    Concurrency
    Moderate
    KV cache quantization
    Supported
    Local API server
    Yes

    Concurrency: Parallel slots work, but each extra request noticeably slows every stream. Prompt cache reuse: Good — follow-up turns usually skip re-reading the history.

    LM Studio: backend maturity per operating system
    BackendmacOSLinuxWindows
    Metal (Apple GPU)· accel.mature——
    CUDA (NVIDIA)—maturemature
    ROCm (AMD)—goodgood
    Vulkan—goodgood
    CPU onlygoodgoodgood
    Documentation(opens in a new tab)LM Studio — Claude Code integration (v0.4.1+)(opens in a new tab)verified 2026-10-13high confidence

    v0.4.26+ ships /v1/systemone and /v1/decisions endpoints, CUDA 13 support on Windows ARM, and llama.cpp 2.50.0 runtime extension packs (official LM Studio changelog, Oct 2026).

  • vLLM

    Model serverAdvanced users

    High-throughput serving engine with PagedAttention and continuous batching. Best for many concurrent requests on NVIDIA/AMD GPUs under Linux.

    Engine
    vllm
    Model formats
    safetensors
    APIs
    OpenAI, Anthropic
    Concurrency
    Excellent batching
    KV cache quantization
    Supported
    Local API server
    Yes

    Concurrency: Continuous batching: parallel requests and agents barely slow each other down. Prompt cache reuse: Excellent — shared prompt prefixes are almost always reused.

    vLLM: backend maturity per operating system
    BackendmacOSLinuxWindows
    Metal (Apple GPU)experimental——
    CUDA (NVIDIA)—mature—
    ROCm (AMD)—good—
    CPU only—experimental—
    Documentation(opens in a new tab)vLLM repository (OpenAI + Anthropic Messages API); vllm-metal community plugin for Apple Silicon(opens in a new tab)verified 2026-09-30medium confidence

  • SGLang

    Model serverAdvanced users

    Serving engine with RadixAttention prefix caching — excellent for agents and many parallel requests sharing long prompts. Linux GPUs.

    Engine
    sglang
    Model formats
    safetensors
    APIs
    OpenAI
    Concurrency
    Excellent batching
    KV cache quantization
    Supported
    Local API server
    Yes

    Concurrency: Continuous batching: parallel requests and agents barely slow each other down. Prompt cache reuse: Excellent — shared prompt prefixes are almost always reused.

    SGLang: backend maturity per operating system
    BackendmacOSLinuxWindows
    CUDA (NVIDIA)—mature—
    ROCm (AMD)—good—
    Documentation(opens in a new tab)SGLang repository (platform list)(opens in a new tab)verified 2026-09-30high confidence

  • Jan

    Desktop appBeginner-friendly

    Open-source desktop chat app built on llama.cpp with a local OpenAI-compatible API server.

    Engine
    llama.cpp
    Model formats
    GGUF
    APIs
    OpenAI
    Concurrency
    Mostly sequential
    KV cache quantization
    Not supported
    Local API server
    Yes

    Concurrency: Concurrent requests largely queue up; best for one user at a time. Prompt cache reuse: Fair — reused within a session, sometimes re-processed.

    Jan: backend maturity per operating system
    BackendmacOSLinuxWindows
    Metal (Apple GPU)good——
    CUDA (NVIDIA)—goodgood
    Vulkan—goodgood
    CPU onlygoodgoodgood
    Documentation(opens in a new tab)Jan repository — OpenAI-compatible server on localhost:1337(opens in a new tab)verified 2026-09-30high confidence

  • LocalAI

    Model serverIntermediate

    Self-hosted OpenAI-compatible API (Docker-first) wrapping llama.cpp and other backends; good for homelab servers.

    Engine
    llama.cpp
    Model formats
    GGUF
    APIs
    OpenAI, Anthropic
    Concurrency
    Moderate
    KV cache quantization
    Supported
    Local API server
    Yes

    Concurrency: Parallel slots work, but each extra request noticeably slows every stream. Prompt cache reuse: Fair — reused within a session, sometimes re-processed.

    LocalAI: backend maturity per operating system
    BackendmacOSLinuxWindows
    Metal (Apple GPU)good——
    CUDA (NVIDIA)—goodgood
    ROCm (AMD)—experimental—
    Vulkan—goodgood
    SYCL / oneAPI (Intel)—experimental—
    CPU onlygoodgoodgood
    Documentation(opens in a new tab)LocalAI repository — OpenAI & Anthropic API compatibility(opens in a new tab)verified 2026-09-30high confidence

  • Lemonade

    Model serverBeginner-friendly

    AMD's local server for Ryzen AI and Radeon: manages llama.cpp (Vulkan/ROCm) plus NPU engines; OpenAI- and Anthropic-compatible APIs.

    Engine
    llama.cpp
    Model formats
    GGUF
    APIs
    OpenAI, Anthropic
    Concurrency
    Moderate
    KV cache quantization
    Supported
    Local API server
    Yes

    Concurrency: Parallel slots work, but each extra request noticeably slows every stream. Prompt cache reuse: Good — follow-up turns usually skip re-reading the history.

    Hardware restriction: AMD hardware only.

    Lemonade: backend maturity per operating system
    BackendmacOSLinuxWindows
    ROCm (AMD)—goodgood
    Vulkan—goodgood
    CPU only—goodgood
    Documentation(opens in a new tab)Lemonade — Anthropic API(opens in a new tab)verified 2026-09-30high confidence