---
title: "M-GATE: A Multilingual Grammar, Translation Accuracy, and Efficiency Evaluation Benchmark"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.03803v1"
permalink: "https://feed.astralor.com/items/itm_f9da740b5fa2bd29ae16f280"
published: "2026-08-04T15:16:05.000Z"
score: 78
channel: "ai"
tags: ["Common Crawl","M-GATE","Multilingual Language Model","Grammatical Error Detection"]
---

# M-GATE: A Multilingual Grammar, Translation Accuracy, and Efficiency Evaluation Benchmark

## AI commentary

> As an editor, what stands out most is how this paper separates “can translate” from “actually knows the language.” For teams working on multilingual infrastructure, M-GATE’s adversarial grammar tests and low-resource round-trip scores are a useful reference baseline. I’d recommend low-resource model practitioners to run it.

## At a glance

M-GATE is a multilingual proficiency benchmark spanning 30 typologically diverse languages, from high- to low-resource, with three tasks: grammatical error detection on adversarially selected linguist-crafted sentences, round-trip translation across 29 target languages judged by a three-provider LLM panel validated against human annotators, and tokenizer-efficiency measurement. Evaluating over 50 models across 80+ configurations, the authors find that fluency and proficiency diverge sharply: top models achieve a Matthews correlation coefficient of only 0.36 on grammar items, and errors lean toward under-flagging ungrammatical text. Translation quality tracks pretraining data share (r=0.86 against log Common Crawl share), producing a steep low-resource penalty that nonetheless narrows with newer models. Private test items behind a public leaderboard help resist contamination.

## Key points

- M-GATE introduces a three-task benchmark: adversarial grammatical error detection, 29-language round-trip translation, and tokenizer efficiency, covering 30 languages.
- Evaluation of 50+ models shows a sharp gap between translation ability and grammatical proficiency, with the best grammar MCC at only 0.36 and errors leaning toward under-flagging.
- Translation quality strongly correlates with pretraining data share (r=0.86), and the low-resource penalty narrows with successive model releases.
- Private test items behind a public leaderboard help resist contamination, with sample data released.

## Full text

Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency. We introduce M-GATE (Multilingual Grammar, Accuracy in Translation, and Efficiency), a benchmark of linguistic proficiency spanning 30 typologically diverse languages from high- to low-resource. M-GATE comprises three tasks: grammatical error detection on linguist-crafted, adversarially selected sentences that turn on hard, language-specific phenomena; round-trip translation of shared English sources across 29 target languages, scored by a three-provider LLM judge panel validated against professional annotators; and a supplementary tokenizer-efficiency measure. We evaluate over 50 models in more than 80 configurations. Fluency and proficiency come apart sharply: models that translate competently sit near chance on the adversarial grammar items, the best reaching a Matthews correlation coefficient (MCC) of only 0.36, and their errors lean systematically toward under-flagging, accepting ungrammatical text rather than raising false alarms. Translation quality closely tracks a language's share of pretraining data (r = 0.86 against log Common Crawl share), producing a steep low-resource penalty that is nonetheless narrowing with successive model releases. Enabling reasoning reliably improves translation, while its effect on error detection is smaller and for some models negative, so the best configuration is task-dependent. To resist contamination, test items are kept private behind a continuously updated public leaderboard, with illustrative examples released (https://m-gate.ai).

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_f9da740b5fa2bd29ae16f280