Task-Specific Detectors Can Beat Vision-Language Models at Runtime

Merve Noyan works on computer vision and multimodal systems at Hugging Face, and wrote a book on vision-language models. Her advice to developers is not to put those models in front of every vision task. For many applications, she argues, a task-specific model is cheaper, faster, and more robust.

The practical meaning of “real time” matters here. Noyan uses the example of a small device—a “toaster”—that needs to process roughly 30 to 40 frames per second. A vision-language model is too large and memory-bound for that kind of on-device throughput, she says. A detector trained for a specific use case can meet the speed requirement and outperform the general-purpose model. She says she will demonstrate that with RF-DETR, an object detector from Roboflow.

Licensing is another reason to choose carefully. Noyan says developers often ask her about YOLO when she posts about object detection. She calls it a good model, but says she thinks it has an AGPL-3.0 license. She suspects some developers deploy it without realizing that, as she puts it, “you have to pay for it.” Her point is to check the license rather than assume that an open-source model is automatically suitable for a given product. She recommends considering commercially permissive alternatives, including Apache 2.0 models.

That advice sets up her larger aim: make it practical for developers to build vision applications end to end, using coding agents to help with the work rather than calling a large VLM directly at runtime.

Noyan built VibeVision as a toolkit for agentic computer vision. The idea was prompted in part by a post she showed in which Gemma 4 decides what to ask of SAM 3.1, then calls the segmentation model to identify vehicles. Noyan wanted to make a broader set of vision models available to coding agents as tools.

She describes the agent as “a bit of a clueless computer vision engineer.” The toolkit is her attempt to compensate: it packages model choices for different tasks and a reusable workflow for training detectors. When selecting models, she prioritizes licensing, then performance in relation to model size and architecture, and, as she puts it, “vibes.”

The workflow addresses a common gap. If developers already have labeled images, they can train a model directly, or ask an agent to help. If they only have images, they first need annotations and a way to assess them. VibeVision uses a vision-language model to label images, other VLMs to judge those labels, and then trains a detector on the resulting dataset.

The labeler, Qwen 3.5 9B, produces bounding boxes. Rather than passing box coordinates as tokens to the judges, the system draws the boxes over the source image and sends that image with a label and description. The judges assess the proposed label in the context of the image and the marked location, and approve or reject it.

The descriptions tell judges what they are being asked to verify. A coding agent generates them, but the system pauses for a person to approve them. Noyan’s example is a “soda bottle” description that specifies the expected shape and features. A generated description can be too long for a simple class or fall short for a specialized one, so the user checks it before it is used to judge the labels.

The workflow uses two judges: Gemma 4 E4B and LFM 2.5 VL. Noyan says she favored an ensemble of smaller judges, drawing on research she had reviewed, rather than asking a model to assign a confidence score. In her view, scores do not work well for comparing judgments from models of different sizes. The pipeline merges the judges’ verdicts and passes the resulting dataset to RF-DETR-medium or large for training.

For the merge, Noyan uses a minimum-agreement rule rather than consensus: an example is retained if at least one judge approves it. A coding agent drives the sequence, which can run locally or on Hugging Face infrastructure. Noyan describes using Jobs for batch processing and training, Inference Providers to route inference across providers, and Buckets to store intermediate data. If a dataset already has labels, the user can go directly to training; otherwise, the agent can start the annotation workflow.

The models in her demonstration were Apache 2.0 licensed except LFM 2.5 VL, whose license she describes as allowing use subject to a revenue threshold. The slide lists Qwen 3.5 9B at about 19 GB for one-off labeling, Gemma 4 E4B at 15.1 GB in BF16, LFM 2.5 VL at about 2.34 GB in BF16, and RF-DETR-medium at 33.7 million parameters.

Noyan says an end-to-end run on about 1,400 images costs roughly three or four dollars. In her setup, Qwen labeling through DeepInfra cost about 16 cents; she used Hugging Face Jobs for the judges and an L4 for training. She notes that she has Hugging Face compute credits and prefers larger batches because she is impatient. She says labeling can also be done through other providers, and the small detector can be trained locally.

Noyan tested the workflow on road-sign detection and document parsing. For road signs, she had human ground-truth labels and compared the results against both the pseudo-labels and the human annotations. Her slide reports mAP50 of 0.9693 for pseudo-labels and 0.8028 for human ground truth, along with a ROC-AUC of 0.987 for human ground truth. She says the gap between the mAP50 results was expected because the detector learned from Qwen’s labels.

For document parsing, the task was to identify non-text elements—such as images, tables, and signatures—in a DocVQA dataset. Noyan showed a case where the trained detector identified a signature that Qwen’s annotation had missed. She said the result suggested the model could generalize, and credited RF-DETR’s strength as part of the outcome.

The judges did not reject examples at the same rate. Noyan found that LFM rejected many more than Gemma. On road-sign detection, requiring both judges to agree would have left her with very few examples, she said, so she kept an example if either judge approved it. The gap between the judges was less pronounced on document parsing.

Noyan’s guidance on when to use consensus reflects different concerns. She says that on road signs she needed minimum agreement to avoid losing too much of the dataset. Her slide suggests consensus when a dataset is large and avoiding false positives matters; in her spoken explanation, she also suggests considering consensus when recall is important. She advises observing how the judges behave on the particular task.

The workflow also leaves two decisions for human review. First, the generated descriptions may not fit the dataset: a description can be too long for a simple class or too vague for a specific one. Noyan keeps an approval gate for those descriptions and says some familiarity with the dataset is unavoidable.

Second, a coding agent can make training choices that are wrong for the task. In the road-sign example, the agent applied horizontal or vertical flips that could change the meaning of left- and right-turn signs, and color jitter that could alter traffic-light colors. Noyan later patched the pipeline so users can choose whether to augment the data. Even a capable coding agent, she says, may lack the computer-vision judgment to recognize when a standard augmentation corrupts the dataset.

The training workflow is one half of VibeVision; the other is a collection of vision models that an agent can call as tools. Noyan describes the list as non-exhaustive and says it draws partly on Hugging Face benchmark leaderboards and partly on her own reading of computer-vision research. It includes models for depth estimation, OCR, zero-shot detection and segmentation, pose, and other human-centered tasks, with some choices available in different sizes.

One example is Falcon-Perception, a 600-million-parameter model that Noyan uses for zero-shot segmentation from natural-language descriptions. She contrasts it with SAM: SAM can segment a “red box,” but, in her example, cannot handle a relational request such as “the red box on the left next to the orange box.” Falcon-Perception can handle that kind of open-ended reference, she says, and has an Apache 2.0 license.

For other tasks, she lists Depth-Anything-V3 for depth estimation; several OCR models drawn from the olmOCR benchmark; Moondream 3 and MM-Grounding-DINO for zero-shot detection; SAM 3 for visual prompting; and models from the Sapiens family for pose and other human-centric tasks. The options vary in size and license. For zero-shot detection, for instance, she lists Moondream 3 at 9 billion parameters and MM-Grounding-DINO in large and tiny versions. Users can choose a smaller model when hardware or speed is a constraint.

Noyan distinguishes visual prompting from natural-language prompting in segmentation: SAM 3 is her choice for the former, while Falcon-Perception serves the latter. The toolkit’s premise is that an agent should have task-specific options to call, rather than defaulting to one model for every vision problem.

Noyan presents the pipeline as a baseline, not a finished solution for every industry use case. It does not include a hyperparameter sweep, and natural language may be a poor way to describe specialized parts that are unfamiliar or difficult to name.

One direction she sees is image-guided detection. Rather than describe a target in words, a user could provide an example image and ask the model to find matching instances across other images. Noyan suggests this could help when an object is hard to describe. She also wants to experiment with merging a judge’s bounding box with the labeler’s box using intersection over union, instead of asking judges only to accept or reject a label. Segmentation support with RF-DETR is in progress.

In the question period, Noyan was asked whether she planned to train vision-language models themselves, including through a self-improvement process. She said that would be exciting, but her immediate priority is helping developers train task-specific models and deploy them on edge devices. Asked about automatic prompt optimization, she said she had not explored it; she used the coding agent to generate judge prompts because it had the project context.