A modern Android phone with 8 GB of RAM can hold a 3B model in memory and answer questions in a few seconds, offline. The point is not to replace GPT or Claude for hard problems. The point is a private, no-network chat that survives a flight, a border crossing, or a bad hotel Wi-Fi. The best apps for running LLMs on-device on Android differ in three ways: which quantized model formats they load, how much RAM they need to stay stable, and how usable the chat interface is on a small screen. We put seven through a month of daily use.
What to look for in an on-device LLM app
Six things separate a chat app that stays installed from one that gets uninstalled in a week:
- Model format support. GGUF is the most common; MLC and ONNX are next. Whichever the app supports decides which models are available.
- Quantization options. Q4_K_M is the sweet spot for a 3B model on 8 GB; Q3 is smaller but noticeably worse.
- Context window handling. Whether the app keeps a running history or resets at each turn.
- Tokens per second on modest hardware. A Snapdragon 8 Gen 2 phone runs a 3B Q4 at roughly 15 to 25 tok/s.
- Battery behaviour. Long generations warm the phone. A background flag or a timer that limits sessions helps.
- Cost and closed-source status. Some options are freemium with cloud fallbacks that need to be turned off.
Quick comparison
| App | Best for | Model formats | Free tier | Standout |
|---|---|---|---|---|
| PocketPal AI | General on-device chat | GGUF | Free, open-source | Best download UX for GGUF models |
| MLC Chat | Speed on Snapdragon 8 | MLC | Free, open-source | Fastest tok/s of anything here |
| Layla | Character chat and roleplay | GGUF | Free tier, subscription | Polished UI, character library |
| MyDeviceAI | Assistant with tools | GGUF, ONNX | Free, open-source | Web-search grounding, still on-device |
| SmolChat | Small phones (4 to 6 GB) | GGUF (small) | Free, open-source | Runs 1B and 1.5B models cleanly |
| Private AI Chat | No account, no telemetry | GGUF | Free | Ships with a curated model list |
| Enchanted | Ollama on the home server | Remote to Ollama | Free, open-source | Best when the phone talks to a home LLM |
The apps
1. PocketPal AI, best general on-device chat
PocketPal AI downloads GGUF models from Hugging Face inside the app and manages them in a library. Presets for Phi-3.5, Gemma 3, Llama 3.2, and Qwen 2.5 give sensible starting quantizations. The chat UI supports system prompts, temperature, top-p, and repeat penalty per model, which is more control than most apps here.
Where it falls short: First-time users need a rough sense of which model to pick. There is no in-app recommendation.
Pricing: Free, open-source (MIT).
Platforms: Android, iOS.
Download: GitHub · Google Play
Bottom line: Start here. It is the most complete free option and does not push a paid tier.
2. MLC Chat, best raw speed on Snapdragon 8
MLC Chat compiles models to the MLC format ahead of time and runs them via TVM. On a Snapdragon 8 Gen 2 phone, MLC’s Gemma 2B runs at roughly twice the tok/s of the same model in GGUF on llama.cpp. Model options are narrower because each model is prebuilt.
Where it falls short: Adding a new model requires the MLC toolchain on a laptop. Not a “download and go” workflow.
Pricing: Free, Apache-licensed.
Platforms: Android, iOS.
Download: MLC Chat · Google Play
Bottom line: The pick if the phone is a recent flagship and the goal is the fastest possible on-device response.
3. Layla, best for character chat and roleplay
Layla wraps llama.cpp in a polished chat UI with characters, voice input, and a library of community personas. The free tier includes local model download; the subscription adds cloud-hosted models and image generation, both optional. Because the app is on-device by default, the character library works offline.
Where it falls short: The RP focus shows in the default prompts. Turning it into a generic assistant means overwriting the system prompt each session.
Pricing:
- Free tier: local models only
- Pro: from a modest monthly fee adds hosted models and images
Platforms: Android, iOS.
Download: Layla
Bottom line: For anyone who wants a chat that leans creative rather than utility.
4. MyDeviceAI, best assistant with tools
MyDeviceAI runs a local GGUF model, then adds a web-search step that reads snippets and hands them back to the model as context. It stays private because the search query is the only thing that leaves the phone. It is the closest thing to a Perplexity-style assistant with local reasoning.
Where it falls short: Web-search integration depends on a search backend that changes. Occasional broken tool calls are on the operator to notice.
Pricing: Free, open-source.
Platforms: Android.
Download: GitHub
Bottom line: The pick when the assistant needs current information, not just what fits in the model weights.
5. SmolChat, best for small phones
SmolChat targets phones with 4 to 6 GB of RAM. It ships with tuned settings for 1B and 1.5B models (Phi-3-mini, Gemma 2 2B, Llama 3.2 1B) and refuses to open larger ones. The result is a chat app that stays responsive on a two-year-old midrange phone.
Where it falls short: Deliberately limited. If the phone can run a 3B model, use PocketPal AI instead.
Pricing: Free, open-source.
Platforms: Android.
Download: GitHub · Google Play
Bottom line: For midrange phones that would freeze on a 7B model.
6. Private AI Chat, best “no account, no telemetry”
Private AI Chat ships a small list of vetted GGUF models, downloads them on first run, and never asks for an account. Chat history stays in an on-device database. The privacy policy fits on one screen because the app does not collect data.
Where it falls short: No cloud fallback if the model is not enough for a hard question. No user-added models beyond the curated list.
Pricing: Free.
Platforms: Android.
Download: Google Play
Bottom line: The pick when the whole point is “no data leaves the phone” and choosing a model is not the fun part.
7. Enchanted, best if you already run Ollama at home
Enchanted is a phone client for Ollama running on the home server. It is not fully on-device (the model runs on the home machine, which handles the compute), but the chat stays on the LAN or over a Tailscale tunnel. For anyone with a beefy home box already hosting Ollama, this is the shortest path to a 70B model from the phone.
Where it falls short: Requires a home Ollama instance. Cellular use needs a tunnel like Tailscale to reach the server.
Pricing: Free, open-source.
Platforms: Android, iOS.
Download: GitHub
Bottom line: The pick when the phone is a thin client to a much bigger local model.
How to pick the right one
If the phone has 8 GB of RAM or more, start with PocketPal AI and a 3B Q4 model. The UI is the friendliest and the model library is the widest.
If the phone is a recent flagship and speed matters more than model choice, switch to MLC Chat.
If the phone has 4 to 6 GB of RAM, use SmolChat with a 1.5B or 2B model.
If the goal is a private assistant that can look things up, use MyDeviceAI.
If the day involves creative chat or roleplay, use Layla.
If a beefy home server already runs Ollama, use Enchanted and treat the phone as a thin client.
FAQ
Which Android phones can run a 7B model? Phones with at least 12 GB of RAM (Pixel 9 Pro XL, Galaxy S25 Ultra, OnePlus 13). A 7B Q4 model plus the OS holds fine on 12 GB and moves. On 8 GB the phone swaps and the app crashes within minutes.
Do these apps drain the battery? Yes, during generation. A 3B model at 20 tok/s on a modern SoC pulls 4 to 6 W. A ten-minute conversation is roughly one percent battery. Long generations warm the phone.
Can I use these offline? Yes, that is the point. PocketPal AI, MLC Chat, SmolChat, Layla, and Private AI Chat run entirely on-device once the model is downloaded. Airplane mode does not affect them.
Is my data private? On the six on-device apps, yes. Enchanted sends chat to your own Ollama server. None of them send prompts to a third-party cloud in default mode.
Which model should I start with? Phi-3.5-mini at Q4_K_M is the safest default on 8 GB. It answers common questions well and fits in about 2.5 GB of RAM. Move to Gemma 2 2B or Llama 3.2 3B if the phone can handle it.
Can these replace ChatGPT? For quick questions, drafting short text, and code snippets under 200 lines, yes. For long reasoning, current events, and complex code, no. Treat on-device as the private, offline layer, not the whole toolbox.
