Edge0: 35B Models in 3 GB of Memory

Sparse Mixture-of-Experts (MoE) models let a large model activate only part of its parameters for each calculation, but they leave a hard edge-deployment problem behind: experts that are not active in the current step still need somewhere to live. Keeping every weight in memory means that sparse compute does not automatically translate into a lower device requirement. Running a large model locally means solving storage, data movement, and scheduling together.

Edge0 is an open-source streaming MoE inference framework built for this problem. It combines SSD expert offloading and Prerouter routing prediction into one inference path, with 35B and 8B model tiers. Today, Edge0 runs through MLX on Apple Silicon: expert weights are read from the SSD on demand, while the trained prediction head ships with the model so storage and computation can work as one system.

Two Model Tiers, One Streaming Inference Framework

The two Edge0 tiers are based on Qwen3.5-MoE 35B-A3B and Ling 3.0. Each combines 4-bit weights with a trained Prerouter head as a complete pipeline. The table includes model scale, active parameters per step, and memory because those numbers determine the capability ceiling, compute cost, and device requirement.

Model tier	Base architecture	Layers / experts	Inference config	Weight footprint	Peak active memory
Edge0-35B-A3B	Qwen3.5-MoE 35B-A3B	40 / 256 routed experts	4-bit / K = 4	~23 GB	~2.9 GiB
Edge0-8B-A1B	Ling 3.0	24 / 128 routed experts	4-bit / K = 8	~4.2 GB	~1.0 GiB

From Sparse Activation to Weights Arriving on Demand

An MoE router selects only a small number of experts for each input. With SSD offloading, selection is only the first step: if a weight is not ready, computation must wait for the read. Repeating “compute, route, load, compute again” at every layer puts storage access directly on the critical path for every generated token.

Edge0 separates that path into two cooperating parts. The SSD stores expert weights, while a streaming expert pool handles caching and movement. Prerouter predicts which experts will be needed next, buying time for reads. Loading, caching, and compute can then flow continuously, reducing the cost of storage access on generation speed.

The framework also separates prompt processing (Prefill) from token-by-token generation (Decode). Prefill can use full-layer loading and batched compute; Decode depends more on expert caching, prefetch, and scheduling. Both model architectures share the same streaming path, with routing width matched to each model.

SSD Expert Offloading: Keep the Active Working Set in Memory

Edge0 does not place every expert weight in memory at once. It reads the pieces required by the current computation from the SSD and keeps recently used data in memory. Experts likely to be needed next are prepared in advance to reduce waiting. The SSD holds the complete model while memory carries the immediate working set, so available memory, cache state, and SSD speed all affect generation speed.

Prerouter: Move the Next Wait Earlier

Prerouter uses a lightweight trained prediction head with hidden states and routing features to predict upcoming expert choices. Its key mechanism is a dual shift across layers and tokens: layer N produces a prediction for the current token, which is used by the next layer for a later token. Predictions are committed at generation-step boundaries, allowing expert loading to overlap with the forward pass.

Under the same model, adapter, and workload, the project alternated Prerouter and native routing. The result was a maximum 59% decoding speedup. Actual gains vary with the expert working set, cache hits, routing width, and storage conditions. Because the predicted route is part of the real execution, the quality results below measure the complete quantized and adapted pipeline.

Together, SSD expert offloading and Prerouter routing prediction change one important relationship: total parameter count no longer equals the amount of memory that must remain resident at every step. The next sections look at speed, active memory, and model quality in actual runs.

Measured Results: Speed, Memory, and Model Quality

Runtime tests used a Mac mini M4 Pro with 24 GB of unified memory. The project used OpenCompass to compare the complete Edge0 int4 pipeline with the original fp16 base models under the same settings. The figure shows five quality evaluations, followed by the corresponding runtime measurements.

Edge0 int4 pipeline compared with the corresponding fp16 base models — Figure 1: Edge0 int4 pipeline versus the corresponding fp16 base models. Scores are out of 100; higher is better.

Performance metric	edge0-35b	edge0-8b
Decode speed	14.9–17.7 tok/s	23.9–25.3 tok/s
Prefill throughput, cold / warm	113 / 140 tok/s	500 / 1428 tok/s
Peak active memory, short context	2.9 GiB	1.0 GiB
Test device	Mac mini M4 Pro, 24 GB	Mac mini M4 Pro, 24 GB

The performance test used a roughly 3.3k-token prompt, 10 warm-up sampling steps, and a timed 200-token decode, with two runs per tier. Cold Prefill is the first request after process startup; warm Prefill benefits from pages already resident in the file cache. Memory is the MLX allocator peak for the short context workload, not total process memory. At 3.3k tokens, the 8B tier uses about 3.3 GiB overall.

Memory Efficiency and Speed, Without Losing the Model's Core

Across AIME 2026, HumanEval, GPQA-Diamond, MMLU-Pro, and IFBench, the 35B pipeline averaged 79.2 versus 83.2 for its fp16 base. The 8B pipeline averaged 69.9 versus 72.7. Based on the published individual scores, the average gaps are about 3.9 and 2.8 points. The 8B MMLU-Pro score is 70.1, above the base model's 65.8, but changes are not uniform across tasks.

These scores measure the complete combination of quantization, training adapters, and predicted routing. The 8B tier is better suited to memory-constrained devices that prioritize interactive speed; the 35B tier preserves a higher capability ceiling for complex tasks. Real-world behavior still depends on the prompt, context length, output length, and whether the page cache is warm.

Make On-Device Inference a Complete Path

Edge0 separates the model, router, streaming expert pool, and serving layer from the concrete backend, with MLX support for Apple Silicon today. Storage, memory, and compute are coordinated by one inference path. On-device inference is therefore not just moving a model onto a device; it is rearranging the whole resource relationship around it.

Edge0 is released under the Apache-2.0 license. See the corresponding release pages for the specific terms of the models and third-party components.