---
title: "Benchmark Prediction with Sparse and Unpaired Cached Responses"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.03614v1"
permalink: "https://feed.astralor.com/items/itm_8577a33d11a709b7e96a6bd8"
published: "2026-11-02T18:49:33.000Z"
score: 57
channel: "ai"
tags: ["Benchmark Prediction","Sparse","Unpaired Cached Responses","Evaluating","Product Kernel Perspective","Space"]
---

# Benchmark Prediction with Sparse and Unpaired Cached Responses

## 速览

Evaluating a generative model on all queries from all available benchmark tasks is prohibitively expensive. As such, models are often evaluated on different subsets of tasks and/or different subsets of queries from a gi…

## 关键点

- Evaluating a generative model on all queries from all available benchmark tasks is prohibitively expensive.
- As such, models are often evaluated on different subsets of tasks and/or different subsets of queries from a given task.
- The resulting collection of responses and the corresponding (model, task) score matrix are sparse and unpaired.

## 全文

Evaluating a generative model on all queries from all available benchmark tasks is prohibitively expensive. As such, models are often evaluated on different subsets of tasks and/or different subsets of queries from a given task. The resulting collection of responses and the corresponding (model, task) score matrix are sparse and unpaired. In this paper, we introduce the Product Kernel Perspective Space (PKPS), a collection of low-dimensional representations of models based on their embedded responses. PKPS generalizes existing methods for representing black-box generative models by comparing responses to similar -- rather than just identical -- queries and is therefore able to utilize sparse and unpaired information. Theoretically, we show that benchmark prediction using PKPS is query-efficient relative to methods that only use information from dense and paired portions of the evaluation data. Empirically, we show that PKPS-based methods outperform baselines on benchmark prediction across two large benchmark suites.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_8577a33d11a709b7e96a6bd8