---
title: "Less Loss, Fixed Weights: Logit Corrections for Forgetting"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2610.42000v1"
permalink: "https://feed.astralor.com/items/itm_1926bdc62781f57e05ae0edc"
published: "2026-10-30T16:44:30.000Z"
score: 53
channel: "ai"
tags: ["Less Loss","Fixed Weights","Logit Corrections","Forgetting When","When","Across"]
---

# Less Loss, Fixed Weights: Logit Corrections for Forgetting

## 速览

When a language model is fine-tuned to follow instructions, or further pretrained on a new subject, it often gets worse at predicting standard web text. This is known as "forgetting." We ask whether this loss can be und…

## 关键点

- When a language model is fine-tuned to follow instructions, or further pretrained on a new subject, it often gets worse at predicting standard web text.
- This is known as "forgetting." We ask whether this loss can be undone by simple, fixed adjustments to the model's final token predictions.
- Across 14 pairs of base and derived models, from 1B to 7B parameters, we fitted a fixed bonus or penalty for each token (a vocabulary bias), alone or together with a temperature.

## 全文

When a language model is fine-tuned to follow instructions, or further pretrained on a new subject, it often gets worse at predicting standard web text. This is known as "forgetting." We ask whether this loss can be undone by simple, fixed adjustments to the model's final token predictions. Across 14 pairs of base and derived models, from 1B to 7B parameters, we fitted a fixed bonus or penalty for each token (a vocabulary bias), alone or together with a temperature. On text held out from correction fitting, the largest top-4K bias recovery estimate is under 10%, with loss conditional on the retained vocabulary. Combining the bias with a temperature recovers up to 36%. Models further pretrained on math lose the most and recover almost nothing. Finally, in a controlled simulation where a model is damaged at random, with no planted bias or temperature change, the bias-plus-temperature fix still recovers 29%. Most of the real-model loss increase stays out of reach of our restricted fits.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_1926bdc62781f57e05ae0edc