---
title: "On Convergence of Gradient Temporal-Difference Learning with Regularized Corrections and Eligibility Traces"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.05869v1"
permalink: "https://feed.astralor.com/items/itm_e1516f32991994ae47f45f7d"
published: "2026-11-04T03:02:33.000Z"
score: 58
channel: "ai"
tags: ["LLM","Convergence","Gradient Temporal-Difference Learning","Regularized Corrections","Eligibility Traces Temporal-difference","TDR","Temporal-difference"]
---

# On Convergence of Gradient Temporal-Difference Learning with Regularized Corrections and Eligibility Traces

## At a glance

Temporal-difference (TD) learning underlies many prediction and control algorithms, but can diverge when combined with off-policy learning, bootstrapping, and function approximation. TD with Regularized Corrections (TDR…

## Key points

- Temporal-difference (TD) learning underlies many prediction and control algorithms, but can diverge when combined with off-policy learning, bootstrapping, and function approximation.
- TD with Regularized Corrections (TDRC) is theoretically sound and practically simple, but existing convergence theory covers only the one-step case under i.i.d.
- sampling, while the Huber Bellman-error formulation has not been extended to TDRC or eligibility traces.

## Full text

Temporal-difference (TD) learning underlies many prediction and control algorithms, but can diverge when combined with off-policy learning, bootstrapping, and function approximation. TD with Regularized Corrections (TDRC) is theoretically sound and practically simple, but existing convergence theory covers only the one-step case under i.i.d. sampling, while the Huber Bellman-error formulation has not been extended to TDRC or eligibility traces. In this work, we establish convergence of constrained TDRC($λ$) with diminishing stepsizes and non-i.i.d. off-policy data, for state-dependent and history-dependent $λ$-parameters. We also propose TDRC($λ$)-Huber, which preserves the TDRC($λ$) fixed point while reducing sensitivity to large Bellman errors. For both algorithms, we apply an ODE-based weak-convergence method from stochastic approximation theory. For the nonlinear mean dynamics of TDRC($λ$)-Huber, we give an integral small-gain condition that guarantees uniqueness and global asymptotic stability of the target equilibrium. We then establish convergence of the constrained algorithm, derive computable sufficient conditions for convergence, and identify a regularization-based convergence mechanism shared by linear TDRC($λ$) and TDRC($λ$)-Huber. When the mean TD matrix is positive stable, both constrained algorithms converge under sufficiently strong regularization without requiring positive definiteness.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_e1516f32991994ae47f45f7d