The hardest part of AI toys in 2026 is not whether they can chat, but where the conversation goes. A cloud model drags in privacy, latency, and subscription headaches; an on-device model bumps into just a few watts and a sliver of memory in the palm. How to fit it in is the real engineering fight.
The NPU is the engine of on-device inference
Put simply, the NPU inside the phone is the engine for local AI. Qualcomm's Hexagon NPU in the Snapdragon 8 Elite reaches 75 TOPS, and MediaTek's APU 790 in the Dimensity 9400 lands around 46 to 50 TOPS, built to accelerate low-precision matrix math. For a toy, a 2B to 3B small model is enough to sustain a natural chat, which is exactly the sweet spot for on-device models: capable, fast, and frugal.
Quantization squeezes the model inside the memory and power wall
The chip alone is not enough; the model itself has to shrink. An 8B model at FP16 needs 16GB of memory, which a phone cannot hold; dropped to INT4 four-bit precision it shrinks to about 4.5GB and fits mainstream devices. A 4-bit quantized model like Gemma 4 E2B takes only about 1.5GB and runs 30 to 45 tokens per second on the Snapdragon Hexagon, with first-token response under 50 milliseconds. The whole chain relies on the NPU's low-precision math to keep power under 5W, or the battery would die and the shell would burn.
One-line advice: the on-device payoff is real, the conversation stays on the device, it works offline, and there is no hidden monthly fee. The cost is a capability ceiling pinned to 3B-class models, with heavy tasks still falling back to the cloud. When shopping, do not just ask if it can chat; ask whether that brain lives locally or in the cloud, the difference matters more than the spec sheet.
© SWPO. All rights reserved.