Pi Adds On-Device GGUF Models Through llama.cpp
Alejandro-ao shows how Pi can use GGUF models served locally through llama.cpp, keeping prompts, code and data on the user’s machine rather than sending them to third-party APIs. The setup depends on running a local llama.cpp server and choosing a quantized model that fits the available hardware; in his Apple M4 Max example, Hugging Face recommends Qwen 3.8-27B’s Q4_K_M variant. Pi’s `/llama` command then downloads and loads that model, making it available through the standard `/model` selector.
Pi can use a model served from the same machine
Alejandro AO shows Pi connected to a Qwen 3.8-27B GGUF model served locally through llama.cpp. After the model is loaded, Pi exposes it through its normal model-selection flow: /model lists the llama.cpp option alongside the user’s other available models, and selecting it sends prompts to the local machine.
The operational result is straightforward, but it depends on two conditions. llama.cpp must be running as a local server for Pi to connect to, and the selected model variant must fit the user’s hardware. In Alejandro’s demonstration, a prompt asking “what is this project about?” is sent to ggml-org/Qwen3.8-27B-GGUF:Q4_K_M and answered locally.
Alejandro says that, in this configuration, prompts, code, and data stay on the computer rather than being sent to third-party APIs. He presents the setup as a way to use a local model inside Pi while avoiding per-token API costs for work handled by that model.
“No data, no code, no prompts are going away to third-party APIs.”
The connection begins with a local llama.cpp server
llama.cpp is the local layer that runs the model and makes it available to Pi. Alejandro describes it as a performant way to run models on a machine. The installation path shown on llama.app is:
curl -LsSf https://llama.app/install.sh | sh
That command installs the llama command-line tool. llama.cpp can download and serve models independently, but the workflow Alejandro demonstrates uses it primarily as Pi’s local endpoint. Running:
llama serve
starts the local server. Pi’s /llama menu is shown using http://127.0.0.1:8080, connecting Pi to that local llama.cpp instance.
The server needs to be running before Pi can select or load a local model. Once it is, Pi becomes the interface for downloading supported GGUF variants, loading them, and switching to them for a task.
The useful model choice is the one that fits the machine
Alejandro begins model selection with llama.app’s recommendations for models intended to run on local hardware, including Qwen 3.8, Gemma 4, and GPT-OSS. He chooses Qwen 3.8 and opens the Hugging Face page for ggml-org/Qwen3.8-27B-GGUF, where the displayed monthly download count is 1,285,913.
Popularity is not the deciding factor in the workflow. The relevant question is which quantized version of the model the machine can run. Hugging Face’s Hardware settings allow a user to enter a hardware profile; Alejandro configures an Apple M4 Max with 36 GB of unified memory, then refreshes the model page to display a compatibility panel for that configuration.
The compatibility panel recommends the four-bit Q4_K_M variant for Alejandro’s configured machine. It also displays larger alternatives, making the hardware tradeoff visible: a model’s precision and file size are not interchangeable with the memory available on the device.
| Quantization | Compatibility-panel size | Status shown |
|---|---|---|
| Q4_0 | 1.68 GB | Listed |
| Q4_K_M | 19 GB | Green checkmark / recommended |
| Q8_0 | 28.6 GB | Listed |
| BF16 | 53.8 GB | Listed |
Alejandro’s recommendation is specific to the machine he entered. Four-bit quantization is not presented as a universal default; it is the option Hugging Face recommends for his M4 Max configuration.
`/llama` turns the selected GGUF file into a Pi model
With llama.cpp serving locally, Pi’s /llama command opens a menu for managing llama.cpp models. Alejandro notes that a first-time user will see only “Download model…”; his own menu includes models he has previously downloaded.
The download workflow is model-ID based. The user chooses “Download model,” pastes the Hugging Face model ID, selects the model Pi finds, and then chooses a quantization. For ggml-org/Qwen3.8-27B-GGUF, Pi presents Q4_K_M as the recommended option, with a displayed download size of 17.7 GiB. The menu also offers Q4_0, Q8_0, and BF16 variants.
After download, the chosen model remains in Pi’s llama.cpp model list. Selecting it starts the loading process; Alejandro’s screen shows the text model reaching 100%. At that point, pressing Escape and opening /model makes the loaded llama.cpp model available for use. His final test sends “Hello there” to the selected local model, which begins generating a response.
The model selection, then, is split across two interfaces. Hugging Face’s configured hardware view helps identify an appropriate quantization; Pi’s /llama flow downloads, loads, and exposes that version to the user.
Local models are one part of a coding workflow
Alejandro AO does not frame local inference as a requirement to use local models for every task. He suggests splitting work between a state-of-the-art model and a local model according to the task. A stronger model might sketch an architecture, create a plan, or begin an implementation; a local model can help finish implementation work.
The rationale is both economic and operational. Alejandro says using a local model for suitable work can reduce token spending while keeping the code, prompts, and data being manipulated on the machine. The practical constraint remains the same: the user has to operate the local server and choose a model variant appropriate for the available hardware.