---
title: "Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.11110v1"
permalink: "https://feed.astralor.com/items/itm_eb5c8af2ff93f784c590790b"
published: "2026-08-11T16:18:34.000Z"
score: 77
channel: "ai"
tags: ["arXiv","Action Policy","Tool-Using Agents","Cross-lingual Evaluation"]
---

# Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents

## AI commentary

> This is one of the rare papers that moves the crux of multilingual evaluation from answers to actions. For teams building agent evaluation or multilingual products, it offers a mirror: if you judge model ability only by text outputs, you may not even notice a 26x error. I especially recommend the finding that non-English tasks are routed through English—it implies current multilingual ability is essentially 'English policy plus a translation channel,' with real consequences for cost, latency, and failure boundaries.

## At a glance

This arXiv study measures the action policy of tool-using agents instead of final answers. Across 8 models, 6 parallel benchmarks, 41 languages, and 2.38M rollouts, the authors remove five confounds that can flip conclusions. After normalization by each model's own reproducibility, four diverse frontier models converge under greedy decoding to 71–73% cross-lingual policy retention, with model identity explaining only 5.7% of variance. Below roughly 10B parameters, retention breaks down, and smaller-model rankings are largely a chance-floor artifact measured by permutation. The paper also shows agents route non-English tasks through English in a causally load-bearing pivot that cannot be abandoned via instructions. Finally, a single trace-extraction regex manufactured a 26-fold accuracy gain while readable outputs barely changed, warning that multilingual evaluation must inspect actions, not just answers.

## Key points

- For the first time, action trajectories are the measured object across 8 models, 41 languages, and 2.38M rollouts, with five confounds systematically removed.
- After reproducibility normalization, frontier models retain only 71–73% of their action policy across languages, with model identity explaining just 5.7% of variance.
- Below roughly 10B parameters, cross-lingual policy retention collapses, and smaller-model rankings are largely driven by a chance floor.
- Non-English tasks are causally routed through English, and models do not abandon this pivot when instructed; one trace-extraction regex can fake a 26-fold accuracy gain.

## Full text

When a tool-using agent is given the same task in a different language, does it still take the same steps? Multilingual evaluation rarely asks: it compares final answers and discards the actions. Yet those actions are the product: they fix cost and latency, decide how the system fails, and are the only auditable part of its behaviour. We make the action policy the measured object across 8 models, 6 parallel benchmarks and 41 languages (2.38M rollouts). The naive measurement fails: five confounds sit between raw trace similarity and any defensible claim, each able to flip a conclusion. Short traces score higher, empty traces score perfectly, unrelated traces agree by chance over half the time, the gap is capped by each model's reproducibility, and a model asked the same question twice in one language answers differently, leaving no baseline. We remove all five, and every correction makes the effect larger. Divergence proves structural, not sampling noise: it survives greedy decoding in every cell and stays flat as temperature rises, even as models grow less self-consistent. Normalised by their own reproducibility, four very different frontier models converge under greedy decoding, each keeping 71-73% of its action policy across languages, with model identity explaining only 5.7% of the variance. Below roughly 10B parameters it breaks down, and the ordering among smaller models is largely an artifact of a chance floor we measure by permutation rather than assume. Agents route non-English tasks through English; this pivot is causally load-bearing, confirmed by a pre-registered prediction across four models, and models will not abandon it when told to. Finally, a single trace-extraction regex, not the model, manufactured a multilingual failure: two worked examples raise one model's measured accuracy twenty-sixfold while its accuracy on readable outputs barely moves.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_eb5c8af2ff93f784c590790b