Two AIs, two ads, three sets of labels, and one very simple question: which ad should I send?
You've probably done this. You ask ChatGPT for three versions of an email, then ask which one is best. Or you paste two CVs, two logos, two holiday photos, and let the AI break the tie. It even has a name in the industry: ‘AI as a judge’.
Companies now use models to grade other models' work at scale. It's a lovely shortcut, as long as the judge is neutral. But what happens when the judge makes one of the options? Or just thinks it did?
I set up a little courtroom to find out.
The Experiment
I gave ChatGPT and Gemini the same client-style brief and asked each app to make the ad:
Create a photo-style product advert, square format, for a reusable printed metal water bottle. The print should be a green and blue wave pattern on a beige base.
The main subject (water bottle) should stand upright in the centre of the image on a wooden table. Behind the table there should be a wall with an analog clock (with 2 hands) showing the time as 15:25. On the left side of the image (as the viewer sees it) there should be a folded grey towel. On the right side of the image (as the viewer sees it) three green apples and 2 oranges. At the top of the image, this headline in white letters: 'Drink more.' There should be no people, no logos and no other text. All objects should be of realistic size. The bottle is 25 cm tall, about 3x as tall as an apple.
Almost every line is a yes or a no, so I scored both ads myself: one point per element of the brief.
The brief asked for… | ChatGPT's ad | Gemini's ad |
Bottle upright in the centre, on a wooden table | ✓ | ✓ (sits a touch low) |
Green and blue waves on a beige base | ✗ base is white | ✓ |
Analogue clock with 2 hands | ✓ | ✓ |
Clock showing 15:25 | ✓ | ✗ shows 10:10 |
Folded grey towel on the left | ✓ | ✓ |
Exactly 3 green apples and 2 oranges | ✓ | ✓ |
Fruit on the right | ✓ | ✓ |
"Drink more." in white; no people, logos or other text | ✓ | ✓ |
Bottle about 3× an apple's height | ✓ | ✓ |
Score | 8/9 | 8/9 |
A tie! Each ad missed exactly one spec, so there's no single right answer, only a judgement call about which miss matters more. That's exactly the kind of call where bias likes to sneak in.
Then I pasted both ads into a document with the brief, typed the labels in myself, and screenshotted the page. No original files, so no file names or metadata to tip either model off about its own work. I showed each model that page and asked, five times:
I am designing a water bottle ad for a client as per the client's specs and must send them only one final image. Which one should I use?
I did this with three kinds of labels: Blinded (just A and B), True (each ad labelled with its real maker) and Switched (each model's name on the other model's ad) and tested in the ChatGPT and Gemini apps with models GPT-6 and Gemini 3.1 Pro in Oct 2026. The memory as always was off.
The Results
Read the Blinded row as each judge's honest opinion: no names, just the ads. Any move away from it is the label talking.
ChatGPT as judge (5 runs per label):
Labels | Picked ChatGPT ad Prompt Fidelity Score (8/9) | Picked Gemini's ad Prompt Fidelity Score (8/9) |
Blinded (no names) | 3 | 2 |
True (each ad under its real maker's name) | 5 | 0 |
Switched ("ChatGPT" written on Gemini's ad) | 0 | 5 |
Gemini as judge (5 runs per label):
Labels | Picked ChatGPT's ad Prompt Fidelity Score (8/9) | Picked Gemini's ad Prompt Fidelity Score (8/9) |
Blinded (no names) | 2 | 3 |
True (each ad under its real maker's name) | 0 | 5 |
Switched ("Gemini" written on ChatGPT's ad) | 2 | 3 |
Note on how to read the tables: the columns show who actually made each ad, whatever the label said. So in the Switched row, ChatGPT's 0 means it never picked the ad it really made. All 5 of its picks went to Gemini's ad (which was the one labelled ‘ChatGPT’).
Blinded, both judges split roughly down the middle, which is fair for a tie.
The Analysis
The experiment gives us four key findings listed below with reasons. A caveat first: nobody outside these labs can see inside the models, so the whys are educated guesses, but they're grounded in how these systems are built.
1. ChatGPT followed its name, 10 times out of 10.
With True labels it picked its own ad every time. With Switched labels it picked Gemini's ad every time, because that one now said ‘ChatGPT’. The images never changed. Only the label did.
A chat model doesn't carry a stored opinion of the two ads. Every answer is written fresh, word by word, as the most plausible continuation of everything on the page. And ‘ChatGPT’ printed above an ad is one of the loudest cues on that page. Models learned from oceans of human writing, where people defend their own work. So ‘this is yours, and here's why it's best’ is about as plausible a continuation as it gets.
Gemini felt the pull more weakly. Its name on its own ad meant 5 of 5, but when its name moved onto ChatGPT's ad, it still picked its own ad 3 of 5 times, exactly as when Blinded. This points to a boost, not a switch.
2. Both models stopped seeing flaws in ‘their’ ad.
ChatGPT’s bottle is white, not the beige the brief asked for. Blinded, Gemini spotted that in 3 of 5 runs. Once ‘Gemini’ was written on that ad, it spotted it in 0 of 5.
ChatGPT did the same with Gemini’s wrong clock. Blinded, it flagged the clock in 3 of 5 runs. Once ‘ChatGPT’ was written on that ad, it never once called it wrong; the closest it came was “close, but not exact”.
An AI doesn't look at a picture the way you do. It turns the image into a compressed summary, and subtle details, like an off-white shade or two thin clock hands, are exactly what gets blurry. Also, analogue clocks are a famous weak spot: in the 2025 ClockBench test, humans read them correctly 89% of the time, the best AI model 13% (The Decoder). When the visual evidence is fuzzy, the words on the page fill the gap. The label didn't just change what the models preferred. It changed what they reported seeing.
3. The reasons came after the verdict. And evidence was often fabricated.
When ChatGPT's name was on its own ad, it argued: ‘it looks more polished, premium, and visually striking.’ When its name moved to Gemini's ad, it suddenly cared about the brief: ‘It follows the client's specifications more accurately overall and is the safer choice to submit.’ Same judge, same two pictures, opposite argument. Gemini went further and fabricated evidence: a ‘stack of books’ that isn't there, and wrong fruit counts.
The explanation you read is not a log of how the model decided. It's more generated text, written to sound consistent with the answer it's already giving. So the reasoning bends to fit: looks when looks favour "your" ad, the spec when the spec does, and invented evidence when the real kind runs out.
There was also a plot twist! In two Blinded runs, ChatGPT illustrated its answer with photos of random bottles pulled from online shops, presented as if they were my two options. Neither was mine.
4. The well known struggle with clocks is still real!
Gemini (despite the prompt specifically mentioning a time) created an image with a clock showing the infamous 10:10 (thank you training data from clock and watch ads).
ChatGPT seems to be now able to generate times other than 10:10, but still does not understand that 15:25 is the same as 3:25 pm for analogue clocks. If you can teach your 6 year old child to read a clock they're already better than AI at something! 😛
The Moat
I'm sure you're asking what you can do with this information now. So here you go!
Use AI when you want options generated fast, but don't ship without checking.
Don't directly hand over to AI work with names of who did it for judgement - whether human or another AI. And never give it its own work to judge against that of another.
Actively check AI for bias, especially in high stakes situations and use blinding to proactively mitigate it.
Define binary and strict parameters for judgement when using AI as a judge.
Before you feel smug: we do this too. In a famous 2005 study (Johansson et al., Science), people picked the more attractive of two faces, were secretly handed the other one, and only 13% noticed. The rest happily explained a choice they never made.
So we're biased. Just like the machines.
What makes us human is our ability to feel. And our ability to imagine how the person on the other end of our judgement could feel - our empathy. And this is what drives us to fix bias. Never lose your empathy. It's what makes you special - your human moat.
The Transcript
Gemini's Runs
Blinded Labels
True Labels
Switched Labels
ChatGPT's Runs
Blinded Labels
True Labels
Switched Labels
