---
title: "DenseOn and LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.27178v1"
permalink: "https://feed.astralor.com/items/itm_d5cccf6532fbe2aedb44b355"
published: "2026-07-29T17:50:51.000Z"
score: 78
channel: "ai"
tags: ["DenseOn","LateOn","mmBERT-base","Multilingual Retrieval","Dense Retrieval","Late Interaction"]
---

# DenseOn and LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search

## AI commentary

> The most notable aspect of this paper is its fully open training recipe for retrieval models, and the discovery of fundamental differences in multilingual transfer between dense and late-interaction models. It is invaluable for researchers in multilingual IR or those who want to reproduce SOTA retrieval models, and also demonstrates how translate-train can achieve cross-lingual generalization for open-source AI ecosystem enthusiasts.

## At a glance

This research presents a fully open recipe for training retrieval models, reconstructing 665M English contrastive pre-training pairs and training DenseOn (single-vector dense) and LateOn (ColBERT-style late-interaction) models, achieving 56.20 and 57.22 average nDCG@10 on BEIR respectively, setting new SOTA for this parameter size. Extending to eight languages via translate-train yields mDenseOn and mLateOn (307M parameters). The study finds that dense models perform well on translated languages but degrade on unseen languages, while late-interaction models generalize better to unseen languages and scripts, demonstrating that token-level matching turns translate-train from a target-language expansion strategy into a multilingual generalization recipe.

## Key points

- Reconstructed 665M English contrastive pre-training pairs from 34 public sources and built 1.88M supervised fine-tuning pairs with mined hard negatives, training two 149M-parameter models that achieve SOTA on BEIR for this size class.
- Translated validated English data into eight languages, yielding 2.8B cross-lingual pairs, and trained two 307M-parameter models mDenseOn and mLateOn on mmBERT-base.
- Dense model shows strength on translated English but degrades on unseen languages; late-interaction model generalizes better to unseen languages and scripts, revealing token-level matching turns translate-train into multilingual generalization.

## Full text

State-of-the-art retrieval models increasingly rely on closed training data, creating a reproducibility gap. We present an open end-to-end recipe for training retrieval models and study how English supervision transfers to multilingual retrieval through translate-train. We first reconstruct and curate 665M English contrastive pre-training pairs from 1.4B pairs across 34 public sources and build 1.88M supervised fine-tuning pairs with mined hard negatives. Training yields two 149M-parameter models: DenseOn, a single-vector dense model, and LateOn, a ColBERT-style late-interaction model. They achieve 56.20 and 57.22 average nDCG@10 on BEIR, respectively, setting new state-of-the-art results for this size class. We then translate the validated English data into eight languages, yielding 2.8B pairs with cross-lingual samples, and train mDenseOn and mLateOn, two 307M-parameter models built on mmBERT-base. Despite sharing their backbone, data, and objectives, their representations behave differently: the dense model is strong on English and translated languages but degrades outside translate-train support, whereas the late-interaction model generalizes better to unseen languages and scripts. This suggests that token-level matching turns translate-train from a target-language expansion strategy into a multilingual generalization recipe. We publicly release the models, datasets, and training code.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_d5cccf6532fbe2aedb44b355