Gemma 4 Makes Perception Part of the Main Transformer

Károly Zsolnai-Fehér argues that Gemma 4’s consequential design choice is not merely support for images and audio, but the way the 12-billion-parameter model incorporates them. Rather than sending non-text inputs through dedicated specialist encoders before they reach the language model, DeepMind’s 12B design feeds raw image patches and audio segments into the main transformer.

A conventional multimodal system is assembled from separate neural networks. An image first passes through a vision model; audio passes through an audio encoder. Those modules translate their inputs into representations that the central model can use. The system can answer questions about what it has been shown or heard, but perception and language reasoning remain divided across specialized components.

Gemma 4 does not eliminate that arrangement throughout its entire model family. Zsolnai-Fehér says the smaller models continue to use specialist processing, and the DeepMind technical report describes improved vision and audio encoders across all model sizes. The important exception is the 12B model, which the report calls a unified, encoder-free architecture that ingests raw image patches and audio.

The architecture shown in DeepMind’s diagram is straightforward. An image is divided into a grid of patches, and those patches are mapped directly into positions in the model’s internal representation. The model retains information about where each patch originated rather than receiving only an external vision encoder’s translation of the picture. Audio follows the same basic logic: it is divided into 40-millisecond chunks and supplied to the main transformer as tokens.

The distinction matters because the main model is no longer receiving a pre-interpreted account of an image or sound from a separate specialist network. Zsolnai-Fehér says this removes hundreds of millions of specialist parameters and blurs the line between “thinking” and “perception.” The architectural wager is that one transformer can learn both tasks together, reducing the parameter overhead of separate perception modules while retaining multimodal inputs.

The DeepMind report describes Gemma 4 as a family of open-weight, natively multimodal models built for compute efficiency and reasoning. It includes dense and mixture-of-experts variants from 2.3 billion to 31 billion parameters, along with a thinking mode intended to generate reasoning traces before responses.

Károly Zsolnai-Fehér frames the architecture against the scale of large open models, citing DeepSeek models with more than 1.6 trillion parameters. His point is not that parameter count is irrelevant. It is that size alone does not supply visual capability: a model that has not been built to accept and interpret images cannot answer a basic question about one.

A displayed contrast makes that claim concrete. One mock chat interface responds “dunno lol” when asked what an uploaded image of water droplets depicts. In the Gemma 4 version of the same interface, the response is: “Ray-traced volumetric caustics. Light focused through a glass sphere.” The comparison is meant to show what the architecture enables: a much smaller model can take an image as input and return a specific visual interpretation.

Zsolnai-Fehér characterizes Gemma 4 as roughly 99% smaller than the trillion-parameter system used for contrast, while still being able to see. He says it runs on a laptop, is free and open, and has been downloaded more than 300 million times. Those are his claims about why the design matters beyond a capability demonstration: multimodal processing is presented as something users can run and own locally rather than something reserved for the largest hosted systems.

A second displayed demonstration extends the same idea to video. A system monitors a room, draws labels around objects including books, a couch, and a banana, and narrates the scene in medieval language. The example combines object recognition with an imposed writing style, illustrating the perception-and-language behavior Zsolnai-Fehér attributes to the model.

DeepMind published the technical report explaining the encoder-free 12B architecture. For Zsolnai-Fehér, that disclosure is as important as the model itself: it gives other open-model developers a design pattern for adding multimodal capability more efficiently.

He calls the disclosed architecture “secret sauce,” but the point is that it is no longer held only by DeepMind. The report offers builders a route toward systems that see, hear, and reason without simply pushing parameter counts toward the largest available models.

Zsolnai-Fehér also points to continuing improvements in the Gemma 4 ecosystem. A displayed chart compares later and earlier 31B and E4B results on agentic-reasoning and tool-calling benchmarks. The chart reports gains across every displayed category.

His closing argument is that open models should not be treated as an inevitable free resource. As capabilities rise, he says, there is no law requiring capable open systems to keep arriving. He calls them valuable to scientists, students, and other users, and urges support for the people building them.