On-device AI runs a trained AI model on the device where a feature is used, such as a phone, laptop, camera, or sensor. The device's own processors perform inference: they apply the model to a new input to produce a result, such as a transcript, an image label, or a text response.
Take a voice-note app. It can transcribe your recording on your phone, then send the transcript to a server for summarization. The transcription is on-device; the summary is a separate inference step. That distinction tells you which parts can work offline and which data the app may need to send elsewhere.
On-Device AI vs. Cloud AI vs. Edge AI
Follow the input to the machine running the model. A result displayed on your phone may have been computed on the phone, on a nearby server, or in the cloud.
| Deployment | Where inference runs | Example |
|---|---|---|
| On-device AI | On the endpoint itself | A phone transcribes a recording locally. |
| Edge AI on nearby infrastructure | On a local gateway or site server | A factory server analyzes video from several cameras. |
| Cloud AI | On remote computing infrastructure | A hosted transcription service processes audio sent by an app. |
On-device AI is usually considered a form of edge AI. The edge is the broader category: the computation can run on the endpoint or on infrastructure near the data source. Nearby servers can serve devices with limited resources, while cloud services provide access to centralized compute over a network.
How Does On-Device AI Work?
Training adjusts the model's weights; inference uses those weights to process new inputs. A model can be trained in a data center and optimized for mobile use before it reaches the phone. Here is what happens when the phone uses it to transcribe a recording.
1. Make the model available on the device
The app needs the model files and the software to run them. The files can ship with the app, download when needed, or be managed by a platform service. Bundling can make the model available at installation, while downloading it separately can keep the initial app package smaller. Google's ML Kit installation guide shows how these choices work for different Android APIs.
2. Prepare the input
The app converts the recording to the form the speech model expects. Whisper's reference implementation, for example, represents audio as a log-Mel spectrogram, which tracks energy across frequency bands over time. A text feature has a different input pipeline: a tokenizer converts text into tokens the language model can process.
3. Run inference with a local runtime
The runtime loads the model and executes its operations on the device. Depending on the model and hardware, it may use a CPU, GPU, or an accelerator such as an NPU. An accelerator can only help with the operations and numerical formats it supports. As the ONNX Runtime mobile guide explains, splitting a model across execution providers can affect performance. Test the complete feature on the target device.
4. Turn the model output into a result
The app decodes the model's predictions into text and displays the transcript. What happens next is a separate design choice. The app can save the transcript locally, sync it to an account, or use it as input to another model.
Recorded audio → input preparation → local inference → transcript
What Makes an AI Model Practical on a Device?
A model needs space for its files and enough working memory for inference. Those are different budgets. In addition to any weights it keeps in memory, a runtime needs room for intermediate results, buffers, and caches. Many language models use a key-value (KV) cache to reuse earlier attention calculations. Longer sequences can make that cache larger.
Quantization uses fewer bits to represent model weights and, in some implementations, intermediate values. It reduces weight storage. Whether it also speeds up inference or reduces runtime memory depends on what the runtime and hardware can do with that format. Quality can change after quantization, so evaluate the model on the inputs and tasks that matter to the application.
Model choice also depends on the job. A model trained for speech recognition and a language model that summarizes long documents have different quality and performance targets. Measure what users will experience: time to the first result, sustained speed, peak memory, battery use, and output quality.
How Edge0 35B Runs Locally on an iPhone
Edge0 35B shows why storage and working memory are separate budgets. It is a 35B-class sparse mixture-of-experts (MoE) language model. Its MoE layers route each token to a small subset of their expert networks. That saves computation per token, but the weights for the other experts still have to be stored somewhere. Edge0 tackles the storage, loading, and quality problems together:
- SSD expert offload reduces how many weights need to stay in memory. The Edge0 runtime reads expert weights from local storage into a fixed set of memory slots as they are needed. Shared weights, the KV cache, and runtime buffers still need memory. The runtime can serve the model without keeping all its expert weights resident at once.
- Prerouter overlaps expert loading with decoding. A trained head predicts expert choices a token ahead. In predicted routing mode, the runtime uses those choices for inference and can begin preparing the weights while the current token is still being processed. In an earlier routing comparison, Edge0 reported up to 59% higher decode throughput versus native routing with the model, adapter, and workload held constant.
- Recover-LoRA helps preserve model quality at 4-bit precision. Edge0 freezes the INT4 base and trains LoRA adapters to follow an FP16 teacher. The adapters recover quality lost to quantization and the routing change. They stay separate from the base weights, so different adapter sets can reuse the same read-only checkpoint. See the Edge0 technical report for the training and deployment details.
In our iPhone 17 Pro Max benchmarks, Edge0 35B reached 70+ tokens/s in prefill on a 627-token prompt and about 14–16 tokens/s in decoding across two workloads. Both used iOS Metal, predicted routing, and the benchmark's default 4,096-token context. Both reported 2.80 GiB peak process memory (phys_footprint). Prefill measures how quickly the model processes the prompt; decode throughput measures how quickly it generates new tokens.
Common On-Device AI Examples
- Speech recognition: Dictation or a voice-note app can produce a transcript without uploading the audio for inference.
- Camera features: A camera can classify an image or flag an object in a video stream locally, then send an alert only when needed.
- Writing assistance: On supported devices, Google's ML Kit GenAI APIs offer local rewriting, proofreading, and summarization.
- Translation: ML Kit's translation API can translate on the device after the required language models have been downloaded.
- Industrial sensors: A sensor can flag an unusual vibration pattern locally and send the event rather than the entire raw stream.
Benefits and Limits of On-Device AI
| What local inference changes | What to check in the full application |
|---|---|
| The model can process sensitive inputs without uploading them for inference. | Check whether backups, diagnostics, sync, or other features send the input or output elsewhere. |
| The inference call avoids a network round trip. | Local compute still takes time. Check responsiveness on the actual device, including under sustained use. |
| Inference can run offline when the model and required data are available locally. | Model downloads, authentication, updates, and live information can still require a connection. |
| Requests processed locally do not use hosted inference capacity. | Budget for device compute, power, model distribution, compatibility, and updates. |
Resource limits still matter. A phone can slow down as it heats up, and the app shares memory with the rest of the device. Results may differ across devices, model versions, and runtimes. For a production feature, also check who can access local data and how model and app updates are delivered.
When Should You Use On-Device, Cloud, or Hybrid AI?
Use on-device AI when a model can meet the feature's quality and performance targets on the device and local operation solves a real problem. Offline dictation, for example, benefits from keeping the entire transcription path available on the phone.
Use cloud AI when the computation is beyond the target devices' capacity or when you need centralized serving across a wide range of hardware. A task that needs live information requires access to that information, but the model itself can still run locally.
Use a hybrid design when different steps have different needs. A meeting app can transcribe locally and use a server model to analyze a long transcript. Decide what information the remote step needs and whether it can work with a smaller input.
Frequently Asked Questions
Do I need an NPU to run AI locally?
No. Models can run on a CPU or GPU if the runtime supports that combination. Some applications require a specific accelerator to deliver the intended speed or battery life.
Can I check whether an AI feature works offline?
After setup, turn off network access and give the feature a new input. A successful result shows that the operation can run offline in that setup. It does not tell you what the app may sync or upload once it reconnects.
Try Edge0 35B
To explore local text generation, start with the runtime and model card. If you’re starting on a desktop, see our guide to running Edge0 35B on Mac and Windows.
- Try Edge0 on GitHub
- View the Edge0-35B-A3B-preview model info on Hugging Face
- Join our Discord to get help with setup or share what you’re building.