---
title: "Are AI Labs Pelicanmaxxing? A Rigorous Evaluation Returns a Negative"
source: "Simon Willison 博客（Atom）"
source_tier: "t2"
url: "https://simonwillison.net/2026/Jul/22/are-ai-labs-pelicanmaxxing/#atom-everything"
permalink: "https://feed.astralor.com/items/itm_c7e43a69005cbdc715f61615"
published: "2026-07-22T23:01:00.000Z"
score: 75
channel: "ai"
tags: ["AI Model Evaluation","Claude Sonnet 5","Gemini 3.5 Flash","GLM-5.2","GPT-5.6 Terra","Image Generation"]
---

# Are AI Labs Pelicanmaxxing? A Rigorous Evaluation Returns a Negative

## AI commentary

> This study is a perfect example of how to scientifically address a fun internet meme. For AI evaluators, model training engineers, and anyone concerned about data contamination, it offers a rare template: rigorous data shows that despite the community's playful speculation about a 'pelican-on-bicycle' training bias, the actual evaluation does not support it. Notably, the experimental design (multi-factor crossover + independent evaluators) itself can be reused to verify other similar suspicions.

## At a glance

Dylan Castillo systematically evaluated the viral claim that AI labs deliberately train models to draw pelicans riding bicycles. Using 8 animals × 6 vehicles = 48 prompts, each run three times across 7 major models (GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2, DeepSeek V4 Pro), with GPT-5.6 Luna and Gemini 3.1 Flash-Lite assisting evaluation. The results found no evidence of pelicanmaxxing: no model performed significantly better on the specific pelican-bicycle combination beyond what its individual components predicted, debunking the rumor.

## Key points

- Castillo created 48 prompt combinations (8 animals × 6 vehicles) and ran each 3 times across 7 models, yielding over 1000 images for evaluation.
- Used auxiliary models (GPT-5.6 Luna, Gemini 3.1 Flash-Lite) to objectively assess outputs, minimizing subjective bias.
- Main finding: no lab significantly outperformed on the 'pelican-on-bicycle' cell beyond their baseline ability to draw pelicans and bicycles separately, debunking pelicanmaxxing.
- GLM-5.2 showed a slight boost in some samples, but the effect was small and statistically insignificant, not proving deliberate training.

## Full text

Are AI labs pelicanmaxxing? Excellent piece of work by Dylan Castillo, who took a deep-dive into the frequently pondered question of whether the AI labs have been deliberately training models to draw pelicans riding bicycles in response to my deeply unscientific benchmark. I've been randomly spot-checking this in the past by testing models against other animals riding other types of vehicle, but never with anything close to the diligence of Dylan's methodology here. Dylan took 8 animals × 6 vehicles = 48 prompts and ran them three times each through 7 different models ( GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2, and DeepSeek V4 Pro). He then used GPT-5.6 Luna and Gemini 3.1 Flash-Lite to help evaluate the results. There's a neat filter view for exploring the results: For the models he tested he could find no evidence of pelimaxxing: The pelicans on bicycles don’t look any better Labs are not better at drawing pelicans Labs are not better at drawing bicycles Labs are not better at drawing pelicans on bicycles, even adjusting for difficulty The pelican-bicycle scenes don’t look memorized [...] Pelicans aren’t drawn any better than other animals. Bicycles aren’t drawn any better than other vehicles. And no lab draws the combination better than its pelicans and bicycles already predict. GLM-5.2 comes closest: it has the largest boost on the exact pelican-bicycle cell, and and its first pelican-on-bicycle sample caught my eye. But the effect is small and not significant, so I wouldn’t put too much weight on it. Via Hacker News Tags: ai, generative-ai, llms, evals, pelican-riding-a-bicycle

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_c7e43a69005cbdc715f61615