SURE+: joint reliability assessment for classification

Motivation: the limitations of a single score

Traditional approaches do not capture the complete reliability picture. Misclassification detection and out-of-distribution (OOD) detection are often considered separately. Optimizing in-distribution (ID) accuracy or OOD detection alone makes the overall trade-off difficult to assess.

Cross-entropy (CE) and CutMix illustrate the problem: CutMix improves OOD detection yet has lower joint DS-F1. The DS-F1 surface maximum is 0.565 for CE and 0.539 for CutMix. A single metric could lead to a different conclusion.

Double scoring: DS-F1 and DS-AURC

The authors propose Double Scoring (DS): two scoring functions and two thresholds jointly determine acceptance. One score assesses whether the input is in-distribution; the other assesses classification confidence. Both checks must pass.

DS-F1 accounts for misclassifications and OOD examples. P is the fraction of correctly classified ID examples among all accepted examples; R is their fraction among all ID examples. The metric selects the largest harmonic mean across threshold pairs.

DS-AURC extends the risk–coverage framework to OOD inputs. It measures the area under the risk surface as acceptance coverage varies in two directions. Lower DS-AURC indicates better joint reliability.

Interactive DS-F1 surface

The surfaces use the original 50 × 50 grid from the project page. Axes represent the MSP threshold τID, the KLM threshold τOOD and DS-F1. Drag to rotate and switch training strategies. Research values are preserved.

CECutMix←→Reset view

CE max: 0.565 · CutMix max: 0.539 · τID / MSP · τOOD / KLM

Comparison of OOD detection methods

The original comparison uses CIFAR-100 with ResNet-18 and ImageNet with DeiT-B. F1 and AURC use single scoring; DS-F1 and DS-AURC use double scoring. Number pairs report Near-OOD / Far-OOD. Arrows indicate the preferred direction.

ID accuracy: ResNet-18 — 77.32; DeiT-B — 81.79.

SURE+: four training components

SURE+ combines four components to improve joint reliability while maintaining or improving classification accuracy.

RegMixup uses regularized mixup augmentation for better calibration and robustness.

RegPixMix uses regularized pixel-level mixing that preserves semantic information while improving robustness.

F-SAM is Friendly Sharpness-Aware Minimization, seeking flatter loss minima for improved generalization.

EMA(ReBN) combines exponential moving averages with re-normalized Batch Normalization for stable training.

Results: ResNet-18 on CIFAR-100

Complete DS-F1 and DS-AURC results compare training strategies and post-processors. Cells show Near-OOD / Far-OOD, with classification accuracy in a separate column. For example, SURE+ with MSP reaches 70.67 / 61.35 DS-F1 and 81.66 accuracy. Numeric scales are retained from the source.

Results: DINOv3 on ImageNet-1K

Large-scale experiments use DINOv3 ViT-L/16. SURE+ with MSP reaches 77.10 / 86.15 DS-F1 and 88.49 classification accuracy. Tables show joint reliability across different confidence post-processors.

These are classification results on research datasets, rather than measured Deepvizor alarm accuracy at a specific site. An operational scenario also includes detection, temporal rules and event delivery.

Related work and citation

The earlier SURE project, SUrvey REcipes for building reliable and robust deep networks, was presented at CVPR 2024. It studies recipes for reliable and robust deep networks. The SURE+ bibliographic record follows.

Related research project ↗

@article{li2026from,
  title={From Misclassifications to Outliers: Joint Reliability Assessment in Classification},
  author={Li, Yang and Sha, Youyang and Wang, Yinzhi and Hospedales, Timothy and Hu, Shell Xu and Shen, Xi and Yu, Xuanlong},
  journal={arXiv preprint arXiv:2603.03903},
  year={2026}
}