Image segmentation in video analytics: the GTR method

What image segmentation means

Object detection returns a class and rectangular bounding box. Segmentation assigns pixel labels or groups pixels into an object mask. GTR’s original examples show people, vehicles, roads and surrounding scene regions.

Semantic segmentation assigns a class such as “person” or “road” to each pixel. Instance segmentation also separates individual objects of the same class: two people can share a semantic label while having two distinct masks.

The appropriate representation depends on the task. Counting or line crossing often needs boxes and tracking. A mask provides more detail when shape or occupied area matters. The operational rule still has to be defined separately.

Why detailed images increase computation

Transformer models divide images into patches, or tokens. Global softmax attention compares tokens with one another. Increasing resolution creates more tokens and quadratically increases the cost of these interactions.

GTR replaces global softmax attention with gated recurrence and gated linear attention (GLA). At a fixed model width, its dependence on token count is linear, offering a different trade-off between detail, memory and computation.

The work studies a feature-extraction architecture rather than a complete alarm-management system. It was published as an arXiv preprint in September 2026; full protocols and limitations are linked.

How GTR gathers scene context

The model keeps a stride-16 patch grid through 12 blocks. Each block uses one directional scan; successive blocks cycle left-to-right, right-to-left, top-to-bottom and bottom-to-top to gather complementary spatial context.

This recurrence traverses patches in a single image; it is not memory between video frames. Recognizing temporal actions requires an additional temporal method. The name GTR alone does not imply fall or action-sequence recognition.

Spatial SwiGLU adds local spatial mixing. Features from blocks 4, 8 and 12 feed a lightweight multi-scale pyramid. Detection then uses a DETR-style decoder with 300 queries.

The original diagram preserves scan arrows, block numbers and mathematical symbols. GLA, RMSNorm, LayerNorm, SiLU and SwiGLU remain as method names for comparison with the source code.

Original GTR architecture diagram. Geometry, arrows, numbers and symbols are preserved.

Open research figure ↗

Distillation: transferring a strong model’s features

The teacher is a DINOv3 backbone specialized for detection using the EdgeCrafter recipe. Its weights are frozen. Both teacher and GTR process the same full image, aligning patch locations.

A learned linear projection aligns feature dimensions. The distillation objective is a squared error between final student and teacher patch features, without input masking or intermediate-layer losses.

Distillation supplies an initial representation before task-specific training. The shared design supports detection, instance segmentation, pose, oriented detection, semantic segmentation and depth in separate task-specific models, rather than one model executing every task simultaneously.

• On-site video analytics and EdgeCrafter ↗

Accuracy, resolution and speed: what was measured

With Objects365 detector pretraining, GTR-L reaches 58.9 box AP on COCO val2017. Median forward-pass latency is 1.908 ms on RTX 4090 using FP16, batch one, torch.compile and CUDA Graphs. This measures the network under a specific protocol, not camera-to-notification latency.

Increasing GTR-S input from 640 × 640 to 1280 × 1280 raises detection accuracy from 53.6 to 57.0 AP, and small-object APS from 36.4 to 42.4. Median latency increases from 1.225 to 2.529 ms and peak memory from 0.096 to 0.249 GB. These are detection-branch measurements, not segmentation metrics.

A separate TensorRT deployment on DRIVE AGX Thor evaluates six tasks and four model scales. The table preserves batch-one medians in milliseconds. Oriented detection and semantic segmentation use 1024 × 1024 inputs; other tasks use 640 × 640.

Evaluating segmentation on enterprise cameras

First decide whether the task needs boxes, individual object masks or a whole-scene class map. For occupied-area checks, define which pixels count as the object and how overlap with a zone is measured.

Test boundaries in difficult cases: people beside equipment, occlusions, reflections and small distant objects. Use reference annotations and assess boundary errors and missed objects, rather than judging only the colored visualization.

Measure all required streams through reception, decoding, processing and event delivery. RTX 4090 or Thor latency cannot automatically be converted into a microserver camera capacity. Event rules, notifications and peripheral responses follow recognition validation.

• How cameras recognize objects ↗
• DV-AIBOX-C-8 microserver ↗

Questions about segmentation and video analytics

Is a mask always better than a box? No. Masks describe shape but add annotation and evaluation requirements. Some scenarios only need boxes and tracking. Choose based on the operational rule and camera conditions.

Does segmentation recognize an action by itself? It describes regions and shapes in an image. Actions depend on changes over time, requiring a video sequence and an appropriate temporal method.

Can processing run locally? A system can use an on-site networked compute device. Model compatibility, accelerator choice and stream capacity require testing; GTR benchmarks describe specific research hardware configurations.

@article{gtr2026feng,
  title={GTR: Gated Token Recurrence for Efficient Dense Prediction},
  author={Feng, Zhe and Liu, Longfei and Liu, Wei and Chen, Kai and Kong, Jiangang and Zhou, Wei and Qian, Yifeng and Chen, Dexiong and Yu, Xuanlong and Shen, Xi},
  journal={arXiv preprint arXiv:2609.26590},
  year={2026}
}