---
title: "机构藏书——富文本：可定制多语言开源流水线，实现OCR文本大规模去噪、去重与标注"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.19026v1"
permalink: "https://feed.astralor.com/items/itm_d144444d98958cc56c7770c5"
published: "2026-08-19T15:20:54.000Z"
score: 75
channel: "tech"
tags: ["Enriched Text","Harvard Library","Institutional Books: Harvard Library","OCR","多语言处理","数据流水线"]
---

# 机构藏书——富文本：可定制多语言开源流水线，实现OCR文本大规模去噪、去重与标注

## AI 点评

> 这篇论文最值得关注的地方在于，它反思了AI数据预处理中‘一刀切’的粗暴做法，并给出了一个可落地的工程方案。对于需要在大规模历史文本上训练模型或做数字人文研究的团队，IB-HL-ET和那个开源pipeline可以帮你省下大量重复清洗工作，同时保留对数据的自主控制。

## 速览

哈佛图书馆联合谷歌图书项目数字化了约98.3万卷藏书，形成IB-HL数据集（242B tokens）。针对现有预处理流水线过度过滤、丢失元数据的问题，研究者提出Enriched Text方法：保留原文并通过HTML式注释提供逐段落语言检测、重复簇识别、每字节比特数等信息。用户可按需解析注释，定制输出。IB-HL-ET版本含217B tokens、1.39B个标注子主题段落，覆盖约250种语言，并开源整个流水线。

## 关键点

- IB-HL数据集包含98.3万卷、242B tokens，源自哈佛图书馆与谷歌图书合作
- Enriched Text方法避免过度过滤，通过HTML式注释保留语言、重复簇和压缩率等元数据
- 发布IB-HL-ET：217B tokens、983,003卷、1.39B标注子主题段落，覆盖约250种语言
- 流水线开源，用户可解析注释按需定制输出，而非接受全局编辑决策

## 全文

Released in 2025, Institutional Books: Harvard Library (IB-HL) is a collection of 983,004 volumes (242B o200k_base tokens), originally digitized through Harvard Library's participation in the Google Books Library project. As researchers and developers have begun to use IB-HL, a tension has emerged between standard large-scale preprocessing practices and the goals of careful information stewardship. Many existing pipelines optimize for web text: as a result, they tend to aggressively filter, deduplicate, restrict by language, and sometimes discard meaningful metadata. Meanwhile, researchers seeking to use IB-HL duplicate effort while performing similar processing and analysis. We describe an approach that we call Enriched Text. Instead of producing a single 'complete' stream of tokens, we normalize the text while preserving metadata through annotations. We separate endmatter, detect per-paragraph language, identify clusters of duplicate paragraphs, and compute per-paragraph bits-per-byte scores. We provide this information through HTML-like annotations layered on top of the text. By parsing these annotations, users can tailor the output to their own needs instead of accepting a global editorial decision on content. The pipeline applies to all $\approx$250 languages in the collection. This report describes this project's goals, implementation, and design rationale. The release includes IB-HL-ET (an enriched-text version of IB-HL containing 217B o200k_base tokens across 983,003 volumes, organized into 1.39B annotated subtopic paragraphs) and the pipeline that produced it. These serve to make the collection easier for machines to parse and for humans to study.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_d144444d98958cc56c7770c5