---
title: "Institutional Books - Enriched Text: A Customizable Multilingual Open-Source Pipeline for Denoising, Deduplicating, and Annotating OCR Text at Scale"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.19026v1"
permalink: "https://feed.astralor.com/items/itm_d144444d98958cc56c7770c5"
published: "2026-08-19T15:20:54.000Z"
score: 75
channel: "tech"
tags: ["Enriched Text","Harvard Library","Institutional Books: Harvard Library","OCR","Multilingual processing","Data pipeline"]
---

# Institutional Books - Enriched Text: A Customizable Multilingual Open-Source Pipeline for Denoising, Deduplicating, and Annotating OCR Text at Scale

## AI commentary

> What stands out here is the reflection on heavy-handed data preprocessing in AI and a concrete, engineering-ready solution. For teams training models on large-scale historical texts or conducting digital humanities research, IB-HL-ET and the open-source pipeline can save enormous duplicated cleanup effort while preserving user control over the data.

## At a glance

Harvard Library, in partnership with the Google Books Library project, digitized approximately 983,004 volumes, forming the IB-HL dataset (242B o200k_base tokens). To address the issues of aggressive filtering and metadata loss in existing preprocessing pipelines, the researchers propose the Enriched Text approach: instead of a single processed stream, the text is normalized while preserving metadata through HTML-like annotations. Per-paragraph language detection, duplicate paragraph cluster identification, and bits-per-byte scores are provided. Users can parse these annotations and tailor outputs to their needs. The release includes IB-HL-ET, an enriched-text version containing 217B tokens across 983,003 volumes, organized into 1.39B annotated subtopic paragraphs, covering about 250 languages, along with the open-source pipeline that produced it.

## Key points

- IB-HL contains 983,004 volumes and 242B o200k_base tokens, from Harvard Library's Google Books collaboration
- Enriched Text avoids over-filtering by layering HTML-like annotations for per-paragraph language, duplicate clusters, and compression scores
- IB-HL-ET release: 217B tokens across 983,003 volumes, organized into 1.39B annotated subtopic paragraphs, covering ~250 languages
- The open-source pipeline lets users parse annotations and customize outputs instead of accepting a global editorial decision

## Full text

Released in 2025, Institutional Books: Harvard Library (IB-HL) is a collection of 983,004 volumes (242B o200k_base tokens), originally digitized through Harvard Library's participation in the Google Books Library project. As researchers and developers have begun to use IB-HL, a tension has emerged between standard large-scale preprocessing practices and the goals of careful information stewardship. Many existing pipelines optimize for web text: as a result, they tend to aggressively filter, deduplicate, restrict by language, and sometimes discard meaningful metadata. Meanwhile, researchers seeking to use IB-HL duplicate effort while performing similar processing and analysis. We describe an approach that we call Enriched Text. Instead of producing a single 'complete' stream of tokens, we normalize the text while preserving metadata through annotations. We separate endmatter, detect per-paragraph language, identify clusters of duplicate paragraphs, and compute per-paragraph bits-per-byte scores. We provide this information through HTML-like annotations layered on top of the text. By parsing these annotations, users can tailor the output to their own needs instead of accepting a global editorial decision on content. The pipeline applies to all $\approx$250 languages in the collection. This report describes this project's goals, implementation, and design rationale. The release includes IB-HL-ET (an enriched-text version of IB-HL containing 217B o200k_base tokens across 983,003 volumes, organized into 1.39B annotated subtopic paragraphs) and the pipeline that produced it. These serve to make the collection easier for machines to parse and for humans to study.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_d144444d98958cc56c7770c5