Finding objects with text: how OV-DEIM works

Abstract: an open-vocabulary detector

Real-time open-vocabulary detection is important in dynamic environments where a large, evolving set of categories must be recognized under strict latency constraints. Existing real-time approaches predominantly use YOLO-style models, while DETR-based alternatives lag in latency, compactness and overall performance.

OV-DEIM is an end-to-end DETR-style open-vocabulary detector built on DEIMv2 with integrated vision–language modeling. A lightweight query supplement strategy improves Fixed AP without compromising inference speed.

The authors also introduce GridSynthetic augmentation, composing multiple training samples into structured image grids. A single forward pass exposes the model to richer object co-occurrences and spatial layouts, mitigating the effect of noisy localization on classification loss and improving semantic discrimination, especially for rare categories.

Six architectural components

The first component is an image backbone. The second is a hybrid encoder aggregating multi-scale features. The third is a text encoder representing categories, phrases or descriptions.

The fourth component selects high-quality object queries conditioned on text. The fifth is a transformer decoder refining those queries. The sixth is a prediction head outputting box coordinates and vision–text alignment scores.

One-to-one matching removes the separate NMS post-processing step. The authors associate this with inference that scales favorably with vocabulary size. Query supplementation adds candidates for Fixed AP evaluation without extra inference cost under the reported protocol.

Original OV-DEIM architecture. Source photographs, mathematical symbols and connections are preserved.

Open research figure ↗

GridSynthetic: several scenes in one training sample

GridSynthetic assembles several source images into an image grid. It augments training data; it is not a camera operating mode or a way to merge video streams during deployment.

The published chart shows EMA-smoothed GIoU loss. The authors report that GridSynthetic consistently produces the lowest loss throughout training, explaining that easier localization supports more effective classification supervision.

Training improvement and inference speed are different properties. Evaluate augmentation under comparable training protocols and deployment latency on the intended computer.

Original GIoU loss curves, preserving axes, values and trajectories.

Open research figure ↗

LVIS: rare classes and the speed protocol

On LVIS minival, OV-DEIM-S, M and L exceed YOLOEv8-S, M and L by 2.0, 0.7 and 0.4 standard AP points. The authors separately report 8.9×, 6.4× and 5.4× speedups on NVIDIA T4 against the comparison that includes NMS.

YOLOE speed is presented as a pair such as 216/18*, with the starred number including NMS. Comparing 161 FPS with 18 FPS gives the reported OV-DEIM-S speedup; comparing it with 216 FPS as the same protocol would be incorrect. Original values are reproduced unchanged.

AP/AP Fixed pairs denote two evaluation settings. The r, c and f subscripts identify rare, common and frequent LVIS categories. OG is Objects365v1, Cap4M is GLIP’s four million image–text pairs, and G-20M is Grounding-20M. FPS was measured on NVIDIA T4 using TensorRT.

COCO: zero-shot transfer and linear probing

The COCO val2017 comparison separates zero-shot transfer from linear probing. OV-DEIM-S, M and L achieve 40.8, 43.3 and 45.9 zero-shot AP respectively.

Every original table row and its evaluation setting are preserved below. AP50 and AP75 use different bounding-box overlap thresholds. Linear probing is not equivalent to zero-shot transfer, so its protocol is shown explicitly.

Deepvizor note: validate text descriptions on real video

Open vocabulary describes how categories are specified; it is not a promise to find any object from any phrase. Wording, object size, occlusion and visual similarity affect results. A pilot should include both target objects and similar negative examples.

Deployment also needs a path from detections to a verifiable event: zones, direction, duration and notification rules. Measure operational event quality on the site’s video: benchmark AP and the proportion of correct alarms evaluate different tasks.

Source and citation

Translated from the original OV-DEIM project page. The full paper and source code are linked above, with the original citation below.

@misc{wang2026ovdeim,
      title={OV-DEIM: Real-time DETR-Style Open-Vocabulary Object Detection with GridSynthetic Augmentation}, 
      author={Leilei Wang and Longfei Liu and Xi Shen and Xuanlong Yu and Ying Tiffany He and Fei Richard Yu and Yingyi Chen},
      year={2026},
      eprint={2603.07022},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2603.07022}, 
}