I’ve enjoyed watching the evolution of Pelicans on Bicycles on Simon Willison’s website. They feel like evaluations even Christopher Alexander would appreciate: look at it and feel if it’s good, or if it’s better than another. Human, subjective, easy.
I thought it would be fun to print one on a t-shirt and wear it at Gamescom to see how many passers-by get the reference. I asked my generic (now called “classic”) ChatGPT Mac app to generate an image and it did surprisingly well:
Here’s the query:
A single pelican riding a bicycle, side profile view, rendered as a clean vector line drawing. Pure white outlines only, uniform stroke width (no thin/thick variation), all lines closed and unbroken, no shading, no fill, no gradients, no crosshatching. Transparent background. Minimalist, playful, slightly whimsical style — like a simple icon or tattoo flash design. Output as vectorized PDF file.
Here’s Claude’s analysis of the situation:
“Simon Willison’s pelican-on-a-bicycle prompt started as a throwaway joke — pelicans are anatomically absurd for cycling, so a good SVG requires the model to compose an unfamiliar scene from parts rather than recall a memorized image. That’s the whole appeal: it’s a proxy for compositional reasoning, not knowledge retrieval.
What I find philosophically interesting, reading through 135 posts of this:
It’s a benchmark that resists being gamed, and everyone knows it, and it still might be getting gamed anyway.Dylan Castillo’s controlled study found no evidence labs are “pelicanmaxxing” — pelicans aren’t drawn better than other birds, bicycles aren’t drawn better than other vehicles. Good news for the benchmark’s validity. But the very fact that someone had to run a rigorous experiment to check tells you something: once a metric becomes famous enough, Goodhart’s Law starts breathing down its neck, whether or not it’s actually been violated yet. The benchmark’s credibility now depends on continuous public auditing, not just on its original cleverness.
It’s deliberately, defiantly informal — which is precisely the source of its value. Willison calls it “deeply unscientific.” No leaderboard, no held-out test set, no p-values on any individual pelican. And yet it’s more diagnostic, in a lot of ways, than formal benchmarks that top out saturated the moment a new model drops. A single SVG lets a human glance and immediately see: are the wheels attached to anything? Does the pelican understand its own anatomy? Is the bike floating? That’s a kind of eyeball validity that MMLU-style scores can’t offer. Legibility beats rigor when the audience is “people trying to get an intuition for a model,” not “people trying to publish a paper.”
Failure is more informative than success here. The genuinely interesting posts aren’t “here’s a great pelican” — they’re the mangled ones: Meta’s Muse Glimmer scattering pelican-parts across the canvas, DeepSeek producing wildly different results depending on reasoning-effort level for the same model, Sonnet 5 confidently mislabeling its own creation as a goose. Those failures expose something structural — how a model represents structure internally, whether “thinking harder” actually helps compositional tasks or just adds noise — that a passing score would hide.
There’s something almost anti-Goodhart about keeping the target absurd. A pelican riding a bicycle has no economic value. Nobody’s shipping a “pelican-drawing” product feature. That absurdity is a feature: it’s very hard to optimize a trillion-parameter training run around a joke without the joke becoming visible as a distortion. Compare that to benchmarks tied to real capabilities (coding, math) where overfitting is much easier to hide because improvement there is expected and desired anyway.
The whole exercise is basically a long-running, crowd-sourced, deadpan experiment in Turing-adjacent testing: not “can it fool a human,” but “can it compose two familiar concepts into an unfamiliar one without falling apart.” That’s a much narrower and more honest question than “is this AGI,” and maybe that’s why it’s survived 135 posts and counting instead of being one clever tweet that got forgotten.”