---
title: "CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.06352v1"
permalink: "https://feed.astralor.com/items/itm_a6532616d3e4c3cc6e27d2cb"
published: "2026-08-06T17:53:18.000Z"
score: 75
channel: "ai"
tags: ["Agent Training Data Synthesis","arXiv","CalibForge","Terminal-Bench 2.0","Solver Calibration"]
---

# CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

## AI commentary

> This paper is worth attention because it shifts task difficulty from human-defined to solver-behavior-defined, which is highly relevant for automated data synthesis. Teams working on agent training data, especially terminal-operation agents, should carefully study its calibration strategy and transfer results.

## At a glance

CalibForge presents an autonomous terminal-task synthesis system that revises candidate tasks through adversarial solver calibration using verified solver behavior. Multi-solver calibration targets disagreement within a heterogeneous solver pool, while contrastive calibration targets a strong-pass/weak-fail relation, both operationalizing a solver-relative learnable zone. Using CalibForge, the authors construct 5,431 calibrated terminal tasks. Ablations show that both strategies yield more effective supervision than authoring and validation alone or single-solver feedback. Models trained on the full collection achieve 32.58% and 47.57% on Terminal-Bench 2.0, with the largest improvements reaching 24.71 points on Terminal-Bench 2.0, 27.68 points on SWE-bench Pro, and 30.04 points on Doc2Repo. These results support solver-relative learnability as a practical target for constructing effective and transferable agent training data.

## Key points

- Proposes adversarial solver calibration using multi-solver disagreement and strong-pass/weak-fail contrast to define task difficulty.
- Constructs 5,431 calibrated terminal tasks, significantly outperforming authoring/validation alone and single-solver feedback.
- Achieves up to 24.71, 27.68, and 30.04 percentage-point improvements on Terminal-Bench 2.0, SWE-bench Pro, and Doc2Repo respectively.

## Full text

Training terminal agents requires executable and verifiable tasks that are not merely solvable, but appropriately challenging for learning. Executable validation establishes feasibility, yet does not reveal how a task behaves relative to a given solver setting. In this paper, we present CalibForge, an autonomous terminal-task synthesis system that uses verified solver behavior to revise candidate tasks through adversarial solver calibration. Multi-solver calibration targets disagreement within a heterogeneous solver pool, whereas contrastive solver calibration targets a designated strong-pass/weak-fail relation; both operationalize a solver-relative learnable zone anchored in demonstrated solvability. Using CalibForge, we construct 5,431 calibrated terminal tasks. Our ablations show that both strategies yield more effective supervision than authoring and validation alone or ordinary single-solver feedback. Models trained on the full collection achieve 32.58% and 47.57% on Terminal-Bench 2.0. The largest improvements over the corresponding base model reach 24.71 percentage points on Terminal-Bench 2.0, 27.68 points on SWE-bench Pro, and 30.04 points on Doc2Repo. Together, these results support solver-relative learnability as a practical target for constructing effective and transferable agent training data.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_a6532616d3e4c3cc6e27d2cb