Best apps for running on-device quantized LLMs on Android in 2026
Q4_K_M, Q5_K_S, Q8_0. If those tags mean nothing yet, they are the reason a 7-billion-parameter model now fits inside a 6 GB phone. Quantization drops model weights from 16-bit floats to 4-bit integers, which shrinks a chat model from 14 GB to under 5 GB and lets your phone run it without a cloud API bill.
We tested seven apps on a Pixel 8 and a Snapdragon 8 Gen 2 tablet. What we cared about: which apps loaded GGUF and MLC quantized weights without a laptop in the middle, which ran offline in airplane mode, and which held a conversation without thermal-throttling to a crawl.
What to look for in an on-device LLM app
- Support for common quantization formats: GGUF (llama.cpp style) or MLC (compiled kernels).
- A model catalog you can browse without pasting Hugging Face URLs.
- Persistent chat history saved locally, not streamed to a server.
- Tokens-per-second readout, so you know when a model is too heavy for the phone.
- CPU vs GPU vs NPU inference toggles, since a Snapdragon 8-series NPU can double throughput.
- Airplane-mode operation. If it needs Wi-Fi to answer “hello”, it is not on-device.
Quick comparison
| App | Best for | Free plan | Starting price/mo | Rating |
|---|---|---|---|---|
| MLC Chat | Compiled MLC kernels, best throughput | Yes (open source) | Free | 4.4 |
| PocketPal AI | GGUF models with a friendly catalog | Yes (open source) | Free | 4.6 |
| Layla | Roleplay and long-context chats | Free tier | $9.99 one-time | 4.5 |
| Sherpa | Minimal llama.cpp shell | Yes (open source) | Free | 4.2 |
| LLMFarm Companion | Cross-device model sync | Yes | $4.99 one-time | 4.3 |
| ChatterUI | Multi-model chat frontend | Yes (open source) | Free | 4.5 |
| Enchanted | Frontend for a home Ollama server | Yes (open source) | Free | 4.4 |
The apps
1. MLC Chat — Best for compiled MLC kernels and top throughput
MLC Chat ships pre-compiled models tuned for the exact Adreno or Mali GPU in your phone. That compilation step is what lets a 3-billion parameter Phi model run at 15+ tokens/second on a Pixel 8, roughly twice what a generic GGUF runtime manages. The catalog inside the app is short but curated: Llama, Phi, Gemma, Mistral, RedPajama.
Where it falls short: No support for arbitrary GGUF files. If you already have a downloaded model, you cannot load it here without recompiling for MLC.
Pricing:
- Free: Apache 2.0 open source.
- Paid: N/A.
Platforms: Android 8.0 and up.
Bottom line: Get MLC Chat when maximum tokens-per-second matters and you can live with the fixed catalog.
2. PocketPal AI — Best for GGUF models with a friendly catalog
PocketPal AI loads any GGUF you point at it. In-app search browses Hugging Face without switching to a browser, and the download screen shows quant tier, size, and a rough tokens-per-second estimate for your device before you commit 4 GB to the disk. Chat history stays on device.
Where it falls short: GPU acceleration lags MLC on the same phone. Long-context conversations chew RAM fast on 6 GB devices.
Pricing:
- Free: MIT-licensed open source.
- Paid: N/A.
Platforms: Android 9.0 and up.
Bottom line: The friendliest way in for anyone new to GGUF quantization on Android.
3. Layla — Best for roleplay and long-context chats
Layla targets the roleplay and long-story-writing crowd. It ships with prompt templates, character-card imports, and larger default context windows tuned to keep story continuity across many turns. Runs GGUF locally; a paid tier adds voice input and premium models.
Where it falls short: UI is busy. Some prebuilt characters lean adult-themed; the safety filter needs manual tuning for family use.
Pricing:
- Free: Base chat with a couple of preset models.
- Paid: $9.99 one-time (Pro) unlocks voice, larger models, and cloud sync.
Platforms: Android 9.0 and up.
Bottom line: Pick Layla if the reason you want a local model is fiction writing that a cloud model would refuse.
4. Sherpa — Best for a minimal llama.cpp shell
Sherpa is the app for people who already know what they want. Point it at a GGUF file on the SD card, set your context length and thread count, and go. No catalog, no sync, no cruft.
Where it falls short: No hand-holding. First-timers may load a model that will not fit and crash the app.
Pricing:
- Free: MIT license.
- Paid: N/A.
Platforms: Android 8.0 and up.
Bottom line: For power users who want a llama.cpp runtime without a chat-app wrapper.
5. LLMFarm Companion — Best for cross-device model sync
LLMFarm started on iOS and has an Android companion that mirrors the same catalog and prompt library. Handy if you split time across an iPad and an Android tablet and want the same characters and system prompts on both.
Where it falls short: Fewer format toggles than PocketPal. GPU support on Adreno is still catching up.
Pricing:
- Free: Base chat.
- Paid: $4.99 one-time removes model-size restrictions.
Platforms: Android 10 and up.
Bottom line: Pick this if you already use LLMFarm on iOS and want prompt parity.
6. ChatterUI — Best multi-model chat frontend
ChatterUI treats every backend the same, whether you are running llama.cpp on the phone, KoboldCpp on a laptop, or Ollama on a home server. Switch mid-conversation from a small local model to a beefier remote one without losing the thread.
Where it falls short: Configuration heavy on first launch. No opinion on which model to pick, so newcomers stall on the selector.
Pricing:
- Free: GPLv3 open source.
- Paid: N/A.
Platforms: Android 8.0 and up.
Download: GitHub Releases
Bottom line: The right pick when the phone is only one of several places you run inference.
7. Enchanted — Best frontend for a home Ollama server
Enchanted is not itself an on-device engine. It is a polished chat app that talks to an Ollama instance you run on a laptop or home server. On the go, use a local quantized model in one of the apps above; at home, jump into Enchanted for a bigger model your phone cannot host.
Where it falls short: Needs a machine somewhere running Ollama. Requires a tunnel or LAN reachability for remote use.
Pricing:
- Free: MIT open source.
- Paid: N/A.
Platforms: Android 9.0 and up.
Download: GitHub Releases
Bottom line: Pair it with any on-device app for the days when your phone cannot fit the model you want.
How to pick the right one
- If you want the fastest tokens/second and can live with a fixed catalog: pick MLC Chat.
- If you want to browse and download GGUF models from Hugging Face inside the app: pick PocketPal AI.
- If your goal is fiction writing and long-context chats: pick Layla.
- If you already know llama.cpp and want a minimal shell: pick Sherpa.
- If you split time across iOS and Android: pick LLMFarm Companion.
- If you want to switch between local and remote backends mid-chat: pick ChatterUI.
- If a home Ollama server is your main model host: pair Enchanted with one of the above for offline days.
The acronyms are less scary once you use them. A Q4_K_M quant of Llama 3.1 8B fits comfortably on any 8 GB phone; a Q8_0 of Phi-3 Mini runs surprisingly well on a 6 GB one. Start there, watch the tokens/second, and step up or down.
FAQ
What does Q4_K_M actually mean?
Q4 means 4-bit quantization of the model weights. K refers to k-quants, a smarter grouping introduced by llama.cpp. M is the medium variant, which mixes some 5-bit tensors for quality. In practice: Q4_K_M is the “default good enough” tier.
How much RAM does an on-device model need?
Rule of thumb: model file size plus 30% for the runtime and context cache. A 4 GB model needs about 5.5 GB free. Phones with 8 GB or more of RAM handle 7B-parameter models at Q4; 6 GB phones stick to 3B.
Will my phone overheat?
Sustained inference on a mid-tier phone will thermal-throttle within a few minutes. Prefer newer Snapdragon 8 Gen 2/3 or Tensor G4 devices, and keep the phone off a soft surface while it runs.
Do these apps send my prompts anywhere?
On-device inference stays on device. Enchanted and ChatterUI are exceptions when configured to talk to a remote backend. Read each app’s network prompts on first launch.
Which model should I start with?
Phi-3 Mini Q4_K_M, Llama 3.2 3B Q4_K_M, or Gemma 2 2B are the safe starting points for a modern Android phone. Step up to Llama 3.1 8B if the device has 12 GB of RAM or more.
