---
title: "Canary Tools: Diagnosing Tool-Selection Reasoning in LLM Agents"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.04719v1"
permalink: "https://feed.astralor.com/items/itm_cfc6b4654fbfc424fae47be9"
published: "2026-08-05T11:38:33.000Z"
score: 78
channel: "ai"
tags: ["Claude Opus 4.8","LLaMA-3.1-8B","MCP","Prompt Engineering","Tool Selection"]
---

# Canary Tools: Diagnosing Tool-Selection Reasoning in LLM Agents

## AI commentary

> This paper's value lies in breaking the black-box 'wrong tool' problem into six actionable defect types, giving agent developers a free diagnostic toolkit. The finding that cheaper models can be safer is a useful reminder not to blindly trust frontier models. If you debug MCP toolchains or assess agent safety, this is a must-read.

## At a glance

This paper introduces Canary Tools, a method that plants diagnostic probe tools into an agent's MCP toolset to systematically identify six types of tool-selection reasoning failures: semantic decoys, parameter traps, capability mirages, prerequisite blindness, temporal decoys, and granularity traps. Evaluating 8 models (frontier and 8B open-weight) across 8,640 runs, the study finds a 36x variation in susceptibility; capability tier alone does not predict safety, with a mid-tier hosted model being most susceptible and a cheaper model sometimes safer. Capability mirages most reliably trap frontier models, while other failure types mostly affect small open models. Softening giveaway phrases leaves frontier performance unchanged, confirming the probes measure reasoning, not phrase-spotting.

## Key points

- A six-type taxonomy converts single wrong-tool outcomes into multi-dimensional reasoning profiles
- Across 8 models and 8,640 runs, susceptibility varies 36x; mid-tier hosted models are most vulnerable
- Capability mirages trap frontier models best, while other weaknesses mainly fire on small open models
- Robust models show no significant degradation under canary pressure; susceptibility correlates with task failure

## Full text

Agent evaluations tell us that a model picked the wrong tool, but rarely why. We introduce canary tools: diagnostic probe tools planted in an agent's Model Context Protocol (MCP) tool set, each engineered to probe one specific tool-selection weakness. A six-type taxonomy (semantic decoys, parameter traps, capability mirages, prerequisite blindness, temporal decoys, and granularity traps) turns a single "wrong tool" outcome into a multi-dimensional profile of how a model reasons about tools. We evaluate eight models -- six hosted and two 8B open-weight -- spanning three capability tiers, on 120 tasks across three canary-density conditions and three seeds (8,640 runs), plus a 2,880-run subtlety ablation. Task success is graded by a provider-independent judge, corroborated by a second independent judge (Cohen's kappa = 0.75). We report three findings. First, susceptibility drops sharply as models get more capable: the per-task canary susceptibility rate (CSR) ranges about 36x across models, lowest for Claude Opus 4.8 and highest for Llama 3.1 8B. Second, capability tier alone does not predict safety: the most susceptible hosted model is mid-tier, and within a provider the cheaper model can be the safer one. Third, the taxonomy is capability-stratified: capability mirages most reliably trap frontier models, while the other types are largely inert on strong models but fire on small open models, so they discriminate by capability rather than being weak. Softening each canary's give-away phrase leaves frontier CSR essentially unchanged, evidence that the probes measure reasoning, not phrase-spotting. Susceptibility also predicts task failure (Spearman rho = -0.34), while the most robust models are not significantly degraded by canary pressure. We release the framework, canary schemas, tasks, and logs.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_cfc6b4654fbfc424fae47be9