Are AI Labs Pelicanmaxxing? A Rigorous Evaluation Returns a Negative
Score
75针对广泛流传的都市传说提供了首次系统化、可复现的证据，颠覆了社区对模型训练偏向的普遍猜测。
方法设计严谨（多因素、重复、辅助评估），数据公开，结论统计清晰，文本表述专业且易读。
对模型评测社区提供了可复用的流程思路，但对普通用户直接使用价值有限。
2026年7月22日刚发布，紧跟当前关于模型训练数据污染的讨论热点。
来源为Simon Willison博客（知名AI观察者），实验由独立研究者Dylan Castillo完成，方法透明。
Base 83 × T2 Source weight ×0.9 = 75
Curation line 70Selected
Why selected
Well-designed experiment with transparent data and statistically grounded conclusions, offering high reference value for understanding training biases in models.
AI commentary
「This study is a perfect example of how to scientifically address a fun internet meme. For AI evaluators, model training engineers, and anyone concerned about data contamination, it offers a rare template: rigorous data shows that despite the community's playful speculation about a 'pelican-on-bicycle' training bias, the actual evaluation does not support it. Notably, the experimental design (multi-factor crossover + independent evaluators) itself can be reused to verify other similar suspicions.」
Quick overview
Dylan Castillo systematically evaluated the viral claim that AI labs deliberately train models to draw pelicans riding bicycles. Using 8 animals × 6 vehicles = 48 prompts, each run three times across 7 major models (GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2, DeepSeek V4 Pro), with GPT-5.6 Luna and Gemini 3.1 Flash-Lite assisting evaluation. The results found no evidence of pelicanmaxxing: no model performed significantly better on the specific pelican-bicycle combination beyond what its individual components predicted, debunking the rumor.
- Castillo created 48 prompt combinations (8 animals × 6 vehicles) and ran each 3 times across 7 models, yielding over 1000 images for evaluation.
- Used auxiliary models (GPT-5.6 Luna, Gemini 3.1 Flash-Lite) to objectively assess outputs, minimizing subjective bias.
- Main finding: no lab significantly outperformed on the 'pelican-on-bicycle' cell beyond their baseline ability to draw pelicans and bicycles separately, debunking pelicanmaxxing.
- GLM-5.2 showed a slight boost in some samples, but the effect was small and statistically insignificant, not proving deliberate training.
Full text
Showing the source text; the full translation is being generated.
I've been randomly spot-checking this in the past by testing models against other animals riding other types of vehicle, but never with anything close to the diligence of Dylan's methodology here.
Dylan took 8 animals × 6 vehicles = 48 prompts and ran them three times each through 7 different models ( GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2, and DeepSeek V4 Pro). He then used GPT-5.6 Luna and Gemini 3.1 Flash-Lite to help evaluate the results.
There's a neat filter view for exploring the results:
For the models he tested he could find no evidence of pelimaxxing:
- The pelicans on bicycles don’t look any better
- Labs are not better at drawing pelicans
- Labs are not better at drawing bicycles
- Labs are not better at drawing pelicans on bicycles, even adjusting for difficulty
- The pelican-bicycle scenes don’t look memorized [...]
Pelicans aren’t drawn any better than other animals. Bicycles aren’t drawn any better than other vehicles. And no lab draws the combination better than its pelicans and bicycles already predict. GLM-5.2 comes closest: it has the largest boost on the exact pelican-bicycle cell, and and its first pelican-on-bicycle sample caught my eye. But the effect is small and not significant, so I wouldn’t put too much weight on it.
Via Hacker News
Tags: ai, generative-ai, llms, evals, pelican-riding-a-bicycle
Related items
FeaturedDeepSeek V4 Doubles Token Share on OpenRouter, Driven by Agentic Workloads
DeepSeek V4 在 OpenRouter 上的代币份额翻倍，智能体工作负载成主要驱动力
The rapid adoption of DeepSeek V4 signals a strong market preference for high-performance, low-cost inference models, especially as agentic use cases reshape AI deployment.
91
FeaturedEnabling Two Settings Triples ARC-AGI-3 Benchmark Scores
启用两项设置使ARC-AGI-3基准测试分数提升三倍
This finding is crucial! It shows that model performance depends not only on training but also on inference-time configuration. For AI engineers and researchers, this is a must-know optimization direction. Teams deploying models on resource-constrained devices in particular could find a low-cost way to boost capabilities.
91
FeaturedGPT-5.6: Fusing Frontier Intelligence with Frontier Efficiency
GPT-5.6：前沿智能与前沿效率的融合
This news is worth watching as it marks a shift from pure parameter scaling to efficiency competition. For AI developers and business decision-makers, GPT-5.6 means higher performance at lower cost, especially for long-context or agentic tasks previously hindered by inference expenses. OpenAI's move may redefine what 'frontier' means.
91
FeaturedLFM2.5-Encoders: New Models for Fast Long-Context Inference on CPU
LFM2.5-Encoder：面向CPU快速长上下文推理的新模型
This blog post catches my eye because it directly solves the pain point of running long-context LLMs on CPU. For teams or individual developers without high-end GPUs, the LFM2.5-Encoder series offers a ready-to-use open-source solution. Highly recommended for those working on RAG, document search, or code analysis—now a standard server can handle million-token documents.
91
FeaturedGPT-Red: OpenAI Automates Red Teaming via Self-Play to Boost Model Robustness
GPT-Red：OpenAI通过自我对弈实现AI红队自动化，提升模型鲁棒性
GPT-Red marks a milestone from human to automated red-teaming—essential for safety engineers and alignment researchers. While reducing cost, it also raises ethical questions about automated attacks, worth pondering for everyone in AI governance.
91
FeaturedOpenAI Launches GPT-5.5 Bio Bug Bounty Program
OpenAI推出GPT-5.5生物漏洞赏金计划
This program deserves attention as it pioneers the extension of AI bug bounty to biosecurity, offering a new research direction for the security community and a practical training ground for AI risk governance. It is particularly important for model safety researchers, biosecurity experts, and practitioners concerned about dual-use AI risks.
91
FeaturedvLLM Introduces Native-Speed Transformers Modeling Backend for Enhanced Inference Performance
vLLM推出原生速度Transformers建模后端，大幅提升推理性能
This update is critical for AI engineering teams: eliminating format conversion speeds up model iteration, and 2-5x speedup directly reduces GPU costs. If you're building LLM online services or batch inference, this is practically a must-read technical update.
90
FeaturedGoogle DeepMind Introduces SL2T, a Sign Language-to-Text Model for Deaf Users
Google DeepMind 发布手语转文本模型 SL2T，助力聋人用户
This news matters not only as a technical breakthrough but also as a sign that AI is beginning to address the needs of marginalized groups. For developers, the architecture and technical approach of SL2T merit deep study; for accessibility advocates, it's a key case for advancing industry standards.