---
title: "An Information Budget for Label-Noise Memorization, and What Statistical Complexity Tracks: A Study with Multilayer Perceptrons"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.04199v1"
permalink: "https://feed.astralor.com/items/itm_e5636e6d66db7d56c051ed91"
published: "2026-11-03T02:35:14.000Z"
score: 60
channel: "ai"
tags: ["Information Budget","Label-Noise Memorization","What Statistical Complexity","Tracks","Study","Multilayer Perceptrons How","Standard"]
---

# An Information Budget for Label-Noise Memorization, and What Statistical Complexity Tracks: A Study with Multilayer Perceptrons

## 速览

How much label noise has a network memorized, and when? We study a multilayer perceptron and what statistical complexity adds to a description of its predictions. Standard identities decompose predictive information int…

## 关键点

- How much label noise has a network memorized, and when?
- We study a multilayer perceptron and what statistical complexity adds to a description of its predictions.
- Standard identities decompose predictive information into signal, memorization, and heterogeneity.

## 全文

How much label noise has a network memorized, and when? We study a multilayer perceptron and what statistical complexity adds to a description of its predictions. Standard identities decompose predictive information into signal, memorization, and heterogeneity. The memorization term vanishes in the population under the stated noise-independence condition. A Fano-type bound relates it to training accuracies, but the memorized fraction M provides no upper bound. Across 89 MNIST runs with ten pre-specified hypotheses (five supported, four unsupported, one not testable), normalized memorization information rose with M on training samples, remained near zero on held-out samples with fresh noisy labels, and was close to its Fano bound. Unlike raw M, it has a zero population null under the independence condition and chance correction largely removed this difference. When every label was replaced by an incorrect class, test accuracy fell to 0.5%, yet predictions retained information about the true class. Mean statistical complexity peaked before memorization at corruption rates up to 40%. At higher rates, its peak fell within the memorization phase, near the epoch when mean training confidence reached 0.6 and the two could not be separated. As a stopping rule, it matched established rules only at low noise and detected corrupted samples less well than simple label-free scores. An exploratory extension of 280 runs covering Fashion-MNIST, pair-flip noise, and four training variants yielded similar results under uniform noise, but not pair-flip noise, where complexity peaked within two epochs. Conclusions are limited to this architecture, two small datasets, and synthetic noise.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_e5636e6d66db7d56c051ed91