---
title: "Replicability, Reproducibility, and Robustness in Political and Social Science Research with Large Language Models"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.06639v1"
permalink: "https://feed.astralor.com/items/itm_9fb092a565bb1f34d31a0488"
published: "2026-11-04T15:20:03.000Z"
score: 57
channel: "tech"
tags: ["LLM","Replicability","Reproducibility","Robustness","Political","Social Science Research","Large Language Models"]
---

# Replicability, Reproducibility, and Robustness in Political and Social Science Research with Large Language Models

## At a glance

This chapter addresses the central reproducibility problem of research with large language models. LLM output is probabilistic and varies across models and versions, which can have significant effects on downstream stat…

## Key points

- This chapter addresses the central reproducibility problem of research with large language models.
- LLM output is probabilistic and varies across models and versions, which can have significant effects on downstream statistical results.
- The chapter distinguishes between replicability, reproducibility, and robustness, and proposes a probabilistic standard based on the distribution of results, not identical outputs.

## Full text

This chapter addresses the central reproducibility problem of research with large language models. LLM output is probabilistic and varies across models and versions, which can have significant effects on downstream statistical results. The chapter distinguishes between replicability, reproducibility, and robustness, and proposes a probabilistic standard based on the distribution of results, not identical outputs. It introduces a standardized prompt script, tests within-model variation for encoders and decoders, and develops error-structure analysis to diagnose where models err and whether their errors align. Using three datasets, the chapter shows that measurement choice can reverse the sign of downstream regression coefficients. Existing design-based corrections with gold-labeled subsamples restore validity, but come at the cost of estimate precision. The main contribution of the chapter is a stacked ensemble of open-weight models that improves precision in the correction step while retaining estimate validity.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_9fb092a565bb1f34d31a0488