JOURNAL
Magnitude: A YC Startup Betting Generic AI Kernels Leave Speed on the Table
A Y Combinator startup's open-source inference engine claims to beat Ollama by tuning kernels to your exact chip. Its own benchmark says 1.3x-2.2x on average, not the 2x+ headline. Here's what the repo, benchmark and docs actually show.

On this page
60-second summary
Magnitude (github.com/magnitudedev/magnitude) is a Y Combinator Summer 2025 startup’s open-source (Apache-2.0, Rust) local inference engine, built by a two-person San Francisco team — CEO Tom Greenwald, previously a PM at SimpliSafe, and CTO Anders Lie, previously a software engineer at AWS. The pitch: generic runtimes like Ollama, llama.cpp and LM Studio ship one-size-fits-all kernels because hand-tuning each model for each chip doesn’t scale for a small maintenance team; Magnitude instead spends up to a minute per downloaded model trying different kernel configurations on your exact chip, then keeps whichever combination is fastest without changing the model’s output. The company’s own published benchmark against Ollama’s default settings across 11 open models measured 1.32x faster prefill and 2.19x faster decode on average on a Mac M4 Pro, and 1.34x/2.17x on an NVIDIA DGX Spark — real gains, short of the “>2x” headline, and the benchmark openly lists three cases where Magnitude was slower. It ships as a desktop app for macOS, Windows and Linux with a bundled CLI, and connects to Claude Code, Codex, Cline and other agents through an OpenAI-compatible endpoint. If you already run local open-weight models for a coding agent and care about tokens per second, a short side-by-side test against your own workload is worth running before trusting either project’s headline number.
The problem it’s aimed at
Teams running local open-weight models care about latency per token, but general-purpose engines ship identical kernels across wildly different hardware — Apple Silicon’s unified memory, NVIDIA’s discrete GPUs, AMD cards, plain CPU boxes — because hand-tuning per chip is its own engineering project most maintainers don’t have the headcount for. That leaves real speed on the table on every individual machine, speed a one-time, per-machine tuning pass could reasonably claim back. Magnitude’s specific bet is to turn that hand-tuning cost into an automated step that runs once per model per machine instead of once per engine release. The customer hypothesis is narrow and worth stating plainly: developers and small teams who have already chosen to run inference locally — for cost, privacy or offline reasons — and who are latency-sensitive enough to care about a measured, model-dependent 0.9x-3.5x range, not people who haven’t adopted local inference at all.
A concrete scenario
Picture a small team already running Qwen or Gemma locally through Ollama to back an internal coding agent, on a Mac Studio or a shared NVIDIA DGX Spark box, because they don’t want prompts containing proprietary code leaving the building. The agent feels slower than the cloud API they’re trying to avoid paying for, and switching back to a hosted model defeats the point. Instead, they install Magnitude’s desktop app, run its setup once per model, and point the existing agent’s base URL at Magnitude’s OpenAI-compatible endpoint instead of Ollama’s — same prompts, same agent harness, same model weights, just a different engine underneath. The existing alternatives here are Ollama, llama.cpp directly, LM Studio, and server-grade options like vLLM; Magnitude’s own FAQ acknowledges all of them ship generic kernels for the reason above. Its differentiation isn’t a new model or a new agent framework — it’s a drop-in inference layer that claims to beat the others on raw throughput for the same open-weight models, while staying free, open-source and local-only.
A demand test before you commit, not a definitive result
Today Magnitude’s scope is a single-machine or single-server local inference swap — it is not a hosted or managed inference product. There’s no team dashboard, no usage billing, and, notably, no pricing page on the marketing site at all. That’s a narrow, concrete starting point: be the fastest thing a developer points an OpenAI-compatible client at, on hardware they already own. Before adopting it, the cheap test is this (not run for this article, since Magnitude’s cited numbers come from its own published benchmark, not independent verification here): take a model you already run through Ollama, pull the equivalent model into Magnitude, point your actual agent at both endpoints for the same 10-20 real prompts, and log wall-clock tokens per second alongside whether outputs stay close enough to trust. The number that should decide adoption isn’t the average benchmark either project advertises — it’s whether a meaningful speedup holds on your model, your hardware, your prompts and your quantization level, since Magnitude’s own 11-model sample ranged from a 0.91x regression to a 3.51x gain depending on the model.
Getting it running
- Desktop app with a bundled CLI for macOS, Windows and Linux. Documented headless/server install on Linux: download the .deb installer, then
sudo apt install ./magnitude-desktop.deb, thenmagnitude server setup, which registers Magnitude as a systemd service (a LaunchDaemon on macOS) and prints a generated network access key (docs.magnitude.dev/remote-server). - The server listens on port 10100 with an OpenAI-compatible endpoint at
http://SERVER_IP:10100/inference/v1, using the access key as the API key and the exact downloaded model ID as the model name — so Claude Code, Codex, Cline, Oh My Pi, or any custom OpenAI-client-compatible script can point at it with a one-line base URL change (same source). - Model management runs through a small CLI set:
magnitude hardwaredetects the machine,magnitude catalog recommendations --preference balanced --limit 5suggests models,magnitude catalog pull MODEL_IDdownloads one and triggers the autotuning pass, andmagnitude models status MODEL_ID/magnitude models load MODEL_IDcheck or preload it (same source). - The catalog lists 19 models as of this writing, from 4B dense models up to a 180B mixture-of-experts model with 6B active parameters, spanning Qwen, Google Gemma 4, Meta Muse Glimmer, OpenBMB MiniCPM-V and Liquid AI, mostly Apache-2.0 or vendor community-licensed, with context windows up to 256K (magnitude.dev/models).
- Windows has no native server mode yet — the docs suggest wrapping
magnitude servewith NSSM to run it as a Windows service. The server also checks for updates roughly once an hour and installs them during idle periods, worth knowing before trusting it inside an unattended pipeline (docs.magnitude.dev/remote-server).
How it actually fits together

Mechanically, the project’s own FAQ is specific about why this works: generalist engines ship one-size-fits-all kernels because hand-tuning each model for each chip would take too long for a small team to maintain across releases. Magnitude instead spends up to a minute per downloaded model trying different ways to split up the math — GPU thread grouping, chunk size, how far ahead to prefetch data — on the user’s actual chip, and keeps whichever combination is fastest while producing the same output as the untuned version.
The company’s published benchmark (magnitude.dev/benchmarks) compares Magnitude against Ollama v0.35.1 at default settings across 11 popular open models at roughly 4-bit quantization, with speculative decoding on for both. On a Mac mini M4 Pro 48GB versus Ollama’s MLX runner: 1.32x faster prefill and 2.19x faster decode on average. On an NVIDIA DGX Spark (GB10, 128GB) versus Ollama’s llama.cpp runner: 1.34x faster prefill and 2.17x faster decode on average. Per-model decode speedups on the Mac ranged from a 0.91x regression on Gemma 4 E2B (slightly slower, not faster) up to 3.51x on Gemma 4 31B — so the averaged “about 2x” claim is not a guarantee for any one model.
The methodology section is unusually candid about its own caveats: the two engines don’t run identical weights, templates or numerics, so outputs differ; on the Mac, Ollama’s MLX runner drafted very few tokens for three Gemma models even with its drafter loaded, which partly inflates Magnitude’s relative decode numbers there; and the page explicitly names three cases where Magnitude was slower than Ollama — Nemotron 3.5 decode on DGX Spark, Gemma 4 E2B prefill on Mac, and Gemma 4 26B prefill on DGX Spark — stating all three are included in the published averages, not excluded. On the open-source question, the public repository (7,208 stars, 497 forks, 54 open issues as of writing, created 2026-06-12, last pushed the same day as this research) actually contains the engine source: inference, inference-v2, inference-v3, inference-v4, old-inference, the CLI and the desktop app are all directories in the repository root, not an installer wrapping a closed binary. License is Apache 2.0, and the project states prompts, files and models never leave the machine, with no internet connection needed once a model is downloaded.
What this costs and where it’s unproven
The software itself is free and Apache-2.0 licensed. The real cost is the hardware you already own (or would need to buy), plus a one-time per-model tuning pass of up to roughly a minute per model per machine, plus disk space for each model you download. There’s no subscription, no token metering, and no cloud dependency once a model is downloaded.
Where it’s unproven: this is a two-person company whose repository was created in June 2026 — fast-moving, with same-day commits and 54 open issues, but with the fragility that implies for anyone treating it as a production dependency rather than a local tool. There’s no SLA and no enterprise tier documented anywhere. More specifically, the marketing site has no pricing page at all (magnitude.dev/pricing returns a 404), so the company’s business model, if it has settled on one, isn’t publicly stated. Any claim about how Magnitude intends to make money is necessarily speculative — plausible paths for a company shaped like this include a future hosted or managed-server tier, paid enterprise support, or a curated model marketplace, but none of these are confirmed anywhere in the company’s own materials as of this writing. Treat it as an open question, not a roadmap.
The differentiation also has a hard edge: Magnitude only helps once you’ve already decided to run inference locally. It does nothing for teams on hosted APIs, and its entire value proposition evaporates if your actual bottleneck is model quality rather than raw tokens per second.
Sources
- Y Combinator company profile for Magnitude — retrieved 2026-10-20
- magnitudedev/magnitude README — retrieved 2026-10-20
- magnitudedev/magnitude GitHub API repository metadata — retrieved 2026-10-20
- Magnitude vs. Ollama benchmark and methodology — retrieved 2026-10-20
- Magnitude remote server setup documentation — retrieved 2026-10-20
- Magnitude model catalog — retrieved 2026-10-20
Where to go next
If your team already runs local open-weight models for a coding agent or for data that can’t leave the building, the question worth 20 minutes isn’t the headline speedup number from either project — it’s whether a measured gain holds on your own model, hardware and prompts, and whether you’re comfortable depending on a two-person, four-month-old company with no stated pricing for something in your inference path. If you want help scoping a local-inference benchmark like the one above, or weighing local inference against cloud API costs for your team, Thuận Lương’s consulting practice helps run exactly that kind of evaluation; reach out through luonghongthuan.com/en/contact.



Discussion
Comments are reviewed before publication. Your email is kept private.