---
title: "Too Sure to Be Safe: Model Calibration for Reliable Log Anomaly Detection"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.17965v1"
permalink: "https://feed.astralor.com/items/itm_37fd465ba2bba1a50d771097"
published: "2026-08-18T16:19:45.000Z"
score: 77
channel: "ai"
tags: ["LoRD","Log Anomaly Detection","Confidence Calibration","Overconfidence"]
---

# Too Sure to Be Safe: Model Calibration for Reliable Log Anomaly Detection

## AI commentary

> As the 'sentry' for operations, log anomaly detection models must err on the side of caution, not overconfidence. The value of LoRD is that it doesn't touch the original model, only rechecks high-risk predictions—a 'post-hoc safety net' that is especially friendly for production systems. Engineers focused on observability or AIOps should follow up and reproduce this right away.

## At a glance

Online log anomaly detection is critical for maintaining the reliability of large-scale computing systems. While recent language model-based detectors achieve strong performance, their confidence estimates remain poorly calibrated, frequently assigning excessive confidence to incorrect predictions, especially for anomalous logs under severe class imbalance. This paper proposes Log Reconstruction and Distance (LoRD), a lightweight post-hoc calibration framework that learns prediction-route-specific reliability models from latent representations of correctly classified validation samples and estimates reliability via route-wise reconstruction distances. LoRD selectively recalibrates high-risk predictions to suppress overconfident errors while preserving reliable ones. Extensive experiments on four large-scale log benchmark datasets and multiple language model-based detectors show that LoRD consistently improves confidence reliability and substantially reduces overconfident anomaly-related errors without sacrificing detection performance.

## Key points

- The study reveals that language model-based log anomaly detectors generally suffer from poor confidence calibration, being overconfident on anomalous samples.
- Proposes LoRD, a lightweight post-hoc calibration framework that estimates prediction reliability via reconstruction distances and selectively recalibrates high-risk predictions.
- Validated on four large-scale log benchmarks and multiple detectors, reducing overconfident errors without sacrificing detection performance.

## Full text

Online log anomaly detection is critical for maintaining the reliability of large-scale computing systems. Although recent language model-based log anomaly detectors achieve strong detection performance, their confidence estimates remain poorly calibrated. We show that these detectors frequently assign excessive confidence to incorrect predictions, particularly for anomalous logs under severe class imbalance. Moreover, confidence on erroneous predictions remains persistently high even when conventional calibration metrics indicate good calibration, creating a critical reliability gap for operational monitoring systems. To address this issue, we propose Log Reconstruction and Distance (LoRD), a lightweight post-hoc calibration framework for reliable log anomaly detection. LoRD learns prediction-route-specific reliability models from latent representations of correctly classified validation samples and estimates prediction reliability through route-wise reconstruction distances. Based on the estimated reliability, LoRD selectively recalibrates high-risk predictions to suppress overconfident errors while preserving reliable predictions. Extensive experiments on four large-scale log benchmark datasets and multiple language model-based detectors demonstrate that LoRD consistently improves confidence reliability and substantially reduces overconfident anomaly-related errors without sacrificing anomaly detection performance.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_37fd465ba2bba1a50d771097