---
title: "ENGINE: Endogenous Variational MultiScale Optimization for Zeroth-Order LLM Fine-Tuning"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.03293v1"
permalink: "https://feed.astralor.com/items/itm_e9bd6daefe80c97b891400e3"
published: "2026-11-02T15:31:38.000Z"
score: 60
channel: "tech"
tags: ["LLM","ENGINE","Endogenous Variational MultiScale","Optimization","Zeroth-Order LLM Fine-Tuning","Zeroth-order","However","LLMs"]
---

# ENGINE: Endogenous Variational MultiScale Optimization for Zeroth-Order LLM Fine-Tuning

## 速览

Zeroth-order (ZO) optimization enables memory-efficient large language model (LLM) fine-tuning, and recent methods incorporate curvature-aware strategies. However, in the high-dimensional parameter space of LLMs, these …

## 关键点

- Zeroth-order (ZO) optimization enables memory-efficient large language model (LLM) fine-tuning, and recent methods incorporate curvature-aware strategies.
- However, in the high-dimensional parameter space of LLMs, these methods inherently suffer from exploding gradient variance and prohibitive costs.
- Conversely, confining updates to a low-dimensional subspace reduces variance but discards critical high-dimensional information, degrading performance.

## 全文

Zeroth-order (ZO) optimization enables memory-efficient large language model (LLM) fine-tuning, and recent methods incorporate curvature-aware strategies. However, in the high-dimensional parameter space of LLMs, these methods inherently suffer from exploding gradient variance and prohibitive costs. Conversely, confining updates to a low-dimensional subspace reduces variance but discards critical high-dimensional information, degrading performance. This creates a fundamental dilemma: full-space optimization is informative but has high variance, while subspace optimization has low variance but loses information. To break this trade-off, we propose ENGINE (Endogenous Variational MultiScale). Inspired by computational mechanics, we adapt static Variational MultiScale (VMS) theory to dynamic optimization, treating low- and high-dimensional spaces as resolved coarse and unresolved fine scales. Through a multigrid-style V-cycle, ENGINE alternates between spaces, dynamically probing fine-grained information in the high-dimensional space and injecting it to correct the efficient low-dimensional sampling process. Under appropriate assumptions, statistical analysis characterizes variance reduction and correction of curvature-induced bias, and convergence analysis establishes a stationarity bound for the corrected coarse updates. Evaluations on standard LLM benchmarks show that ENGINE achieves higher accuracy and faster convergence than existing ZO methods, achieving up to a $2.61\times$ speed-up in forward passes with zero additional peak memory overhead. Code is available at https://github.com/AI4Science-WestlakeU/ENGINE.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_e9bd6daefe80c97b891400e3