---
title: "Validity of LLMs as Data Annotators: AMALIA's Challenge on Authority Moral Coding"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.08731v1"
permalink: "https://feed.astralor.com/items/itm_d56ad7936c5a50dfcd15adf5"
published: "2026-07-09T17:34:25.000Z"
score: 79
channel: "ai"
tags: ["AMALIA","Large Language Models","Data Annotation","Validity Assessment","Moral Foundations Theory"]
---

# Validity of LLMs as Data Annotators: AMALIA's Challenge on Authority Moral Coding

## AI commentary

> This paper hits a common blind spot in current LLM evaluation: we too easily equate agreement with validity. Its 'recovery gap' design cleanly exposes the model's shortcut thinking. For teams aiming to use LLMs for automated social science coding, this is almost mandatory reading—a warning not to be fooled by surface accuracy. I especially appreciate its call for sovereign-LLM benchmarks to examine the evidential path, not just the final answer.

## At a glance

Portugal's national language model AMALIA (9B parameters) achieves agreement with human coders on the moral foundation of authority close to models 8-13 times its size, but agreement is not validity. By decomposing holistic prompts into atomic codebook clauses and recombining them under explicit theory rules, the study reveals the model relies on surface correlates (e.g., moral outrage near authority figures) rather than theoretical constructs, recovering only about half of holistic performance. Cross-lingual transfer fails, pointing away from corpus as the main explanation. The paper argues sovereign-LLM benchmarks should test the evidential route to agreement, not just agreement itself.

## Key points

- AMALIA's agreement with human coders on authority moral coding is competitive with larger open models, but decomposition tests reveal reliance on surface correlates.
- The model lacks theoretical validity, suitable only for large-scale screening and pre-coding, not standalone measurement.
- Cross-lingual transfer failure points to model-construct misalignment rather than corpus differences.
- The study introduces 'recovery gap' as an actionable test for evaluating LLM annotation validity.

## Full text

A national language model offers a linguistic community its own instrument for measuring what its citizens say and value. Portugal's AMALIA, a publicly funded 9B-parameter model for European Portuguese, appears competitive on agreement alone: asked to code the moral foundation of authority, it agrees with trained human coders to within six F1 points of open models eight to thirteen times its size. Yet agreement is reliability, not validity. For theoretical constructs that must be inferred rather than read from surface features, the question is whether the model follows the construct's theory or reaches the right code by correlated shortcuts. We test this with the recovery gap: the loss in performance when a holistic prompt is decomposed into the codebook's atomic clauses and recombined by the theory's explicit rule. If calibration closes that gap, some portability should survive across models and languages; where it does not, the construct-model instrument is the likely locus of failure. We ask whether a calibrated English instrument transfers to AMALIA-9B and to European Portuguese. For one construct and one corpus, it does not. Decomposition recovers only about half of AMALIA's holistic performance, and error analysis suggests reliance on surface correlates, especially moral outrage near authority figures. An open multilingual LLM closes the gap on the same Portuguese corpus under the same instructions, pointing away from the corpus as the main explanation. AMALIA can still screen and pre-code at scale, but it cannot yet measure this construct well enough to stand alone. The study is a single counterexample, not a verdict on national models; it argues that sovereign-LLM benchmark batteries should test not only agreement with human coders, but the evidential route by which that agreement is warranted.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_d56ad7936c5a50dfcd15adf5