RAM, Not Compute, Is Constraining Edge AI Deployment

Cormac Brick frames tiny models as necessary if intelligence is to reach more than expensive robots and high-end devices. Running models locally offers familiar advantages: consistent latency, privacy because sensitive data remains on-device, offline availability, and avoidance of cloud-token costs that compound at application scale. But those benefits are constrained first by available RAM.

Brick describes DRAM cost as a newly acute obstacle. Some phone manufacturers are shipping devices with less memory than in prior years, he says, while the cost of a 16 GB Raspberry Pi has risen roughly 2.5 times since launch. That makes the usual edge-AI problem less about whether a processor can execute a model than whether a product can economically reserve the memory needed to hold its weights, runtime state, and cache.

The consequence is a design discipline: quantize aggressively and select the smallest model that can do the job. It is also complicated by the range of target hardware—CPUs, GPUs, and NPUs across phones, laptops, browsers, IoT devices, and robots—and by the fact that much current language-model research is aimed at far larger models and mixture-of-experts techniques rather than the low end of the size spectrum.

Google’s AI Edge team works on that deployment layer through open-source projects including LiteRT-LM, LiteRT, and MediaPipe. Brick says the group develops core edge technology for Google products, works with the Gemma team to make models run well across devices, and releases tools when its internal work can be packaged for wider use.

The division is practical. Use a promptable model in the roughly one-to-four-billion-parameter range when a device can support a four-GB-plus memory budget and stronger hardware. Use a 50-million-to-500-million-parameter model when device reach, responsiveness, or headroom for the rest of the application matters more—and expect to fine-tune it for a defined task.

Brick uses “small models” to mean roughly the one-to-four-billion-parameter range. These models can be practical in high-end phones, laptops, applications, and better-provisioned robotics hardware. In IoT and robotics, however, deploying one usually implies around four to eight GB or more of DRAM once the model, runtime, KV cache, operating system, and other active software are considered. That hardware requirement narrows the market considerably.

At this scale, the implementation playbook remains relatively simple. A developer can often use zero-shot prompting, perhaps with a LoRA adaptor, and obtain useful behavior for function calling and agent-like skills. Brick points to Gemma 4’s two-billion-parameter model as an example of how much reasoning capability has moved into smaller weights. On the benchmarks shown in the presentation, Gemma 4 2B scored 20.4% on MMLU Pro and 15.6% on GPQA Diamond, below its 9B and 27B counterparts but still demonstrating usable general reasoning at the low end of that family.

The constraint is not merely the nominal parameter count. Google’s optimized Gemma 4 E2B configuration combines 2-bit, 4-bit, and 8-bit quantization, bringing its effective weight storage to about 2.9 bits per weight. Brick says the text-only model weights occupy 841 MB in memory; with runtime and KV-cache requirements, active RAM may reach about two GB. A device needs more headroom beyond that, which is why he treats four GB or more as the practical starting point.

The presentation’s memory-footprint chart makes the distinction visible: its 841 MB text-only in-memory figure covers model weights, while the shown multimodal in-memory figure rises to 2,489 MB after embeddings, audio and vision encoders, metadata, and other components are included. The point is not that every deployment requires the full multimodal stack, but that a model file alone is not a device memory budget.

The performance numbers explain why this class is attractive where that budget exists. On a Raspberry Pi 5 with 16 GB, the E2B model reaches 7.6 decoded tokens per second without multi-token prediction. Brick says enabling multi-token prediction can roughly double that depending on the task. On a Jetson Orin Nano, using Google’s toolchain, it reaches 24.2 tokens per second. On a Qualcomm IQ-8275 evaluation kit’s NPU, the displayed figures are 3,747 prefill tokens per second and 31.7 decode tokens per second.

For multimodal work, those prefill speeds can translate into near-real-time behavior. Brick estimates that a medium-resolution image corresponds to roughly 500 tokens and a high-resolution image to 1,120. On the Qualcomm NPU, he says, that leaves room to process about three high-resolution frames per second while retaining useful generation speed. The tradeoff is explicit: these applications become possible on more capable hardware, but the device bill of materials must absorb a higher DRAM line item.

For a direct test of local performance, Brick points to the open-source Google AI Edge Gallery application for iOS and Android. It lets users download and run models on a phone, and its source code demonstrates use of the open-source runtime.

An Open Duck Mini v2 hobby robot demonstration illustrates the distinction. Brick describes one version running on a Jetson Nano and another on a Raspberry Pi. Both can take voice and image input, read signs, react, and move their heads. But the Jetson-based version has what he characterizes as good real-time interaction, while the Raspberry Pi version works much more slowly.

A two-billion-parameter model can fit and execute on a Raspberry Pi, yet it may not meet the responsiveness a user expects from an interactive robot. For many potential deployments, it does not fit at all: older laptops, watches, lower-tier consumer devices, and systems where the AI feature must share constrained memory and compute with the application’s primary function.

In those settings, “system health” is itself a reason to shrink the model. The model may need to leave enough memory and responsiveness for the rest of the application. Brick’s answer is a separate category: tiny models, typically between 50 million and 500 million parameters, designed to run on consumer hardware with under two GB of DRAM and to ship directly with an application.

General small models can frequently be prompted into a feature. Tiny models work especially well when the task is fixed or when the model has been fine-tuned specifically for the desired output.

There are two viable approaches for tiny models, according to Cormac Brick. The first is to use an off-the-shelf model for a narrowly defined capability. Speech recognition, visual understanding, and embeddings are established categories where this can work immediately. The models shown include Parakeet, Moonshine, Qwen2-ASR, and Whisper variants for automatic speech recognition; EmbeddingGemma for text embeddings; and FastViLM, a 0.5-billion-parameter vision-language model.

Brick shows FastViLM running with hardware acceleration on an Android device and describes it as a useful route to a first layer of visual awareness for an edge or IoT product. The relevant test is not whether a tiny model can replicate a general-purpose assistant. It is whether it can recognize speech, describe an image, match text, or perform another bounded function quickly and locally.

The second approach is fine-tuning a small base model into a dedicated component. Brick’s example starts with 270-million-parameter Gemma 3 or FunctionGemma variants. FunctionGemma has additional pretraining for function-calling patterns; Gemma 3 is a general-purpose base. On a Raspberry Pi 3, the presentation reports roughly 250 tokens per second for prefill and 45 decoded tokens per second—far above the mid-single-digit decoding rate Brick cited earlier for the two-billion-parameter model on a Raspberry Pi.

The mobile-actions demonstration turns arbitrary text into one of ten predefined functions, such as scheduling a calendar event or changing Wi-Fi settings. Brick says the fine-tuned model reaches more than 86% reliability in selecting the appropriate function from free-text input. An ASR model placed ahead of it creates a voice-to-function-calling pipeline.

Voice matters particularly for small devices, Brick argues, because conventional settings menus are often difficult to use. A user should be able to express an outcome rather than navigate a constrained interface. A model trained for a defined action space can turn voice into a robust interface for many edge and IoT situations.

The difficult part of this workflow is not choosing the base model. It is creating the training data. Brick’s recommended process is to establish that a base model’s speed and memory footprint are acceptable, then synthetically generate a task-specific dataset that includes positive and negative examples. He says the team has found anywhere from roughly 10,000 to 10 million synthetic samples sufficient to fine-tune small models to a high degree of reliability, depending on the task and model.

That investment can change the economics of familiar language features. Brick says that summarization and proofreading—tasks a two- or four-billion-parameter model can perform reasonably reliably through prompting—can achieve similar or greater quality with a much smaller model when the developer is willing to create a synthetic dataset and fine-tune for the single task. The result is broader hardware compatibility and greater responsiveness, rather than generic flexibility.

AI Edge Eloquent, an iOS voice-dictation application, is Brick’s production example of the specialized-model approach. The app performs speech dictation locally, without a subscription, and does more than simply transcribe audio. It removes filler words such as “um” and “ah,” while also biasing the output toward words and names relevant to an individual user.

Its architecture is deliberately modular. A hardware channel and personal dictionary feed an ASR engine and a text-polishing engine, which in turn produce the text shown in the Eloquent interface. Both engines are fine-tuned, tiny Gemma-based models in the low hundreds of millions of parameters. Brick presents this as a way to put what would otherwise be a server-only feature on the device while maintaining what he calls very good quality.

The same logic is beginning to shape browser features. Brick notes that Chrome has summarization and proofreading APIs in developer preview. Delivering them through tiny models, he says, lets the Chrome team make them available to a wider set of users than would be possible with larger local models.

The remaining ambition is to make the fine-tuning path itself more accessible. In response to a question about where tiny models could go next, Brick identifies the generalization of voice to function calling as a key objective. He suggests that agents could eventually generate the necessary synthetic data for developers, reducing the current burden of producing task-specific training sets. Faster visual models capable of broader functions, including segmentation, are another direction he identifies.