We did a recent study with over 4,000 American consumers ranging from age 18 to 65. The study tasked our participants with selecting which one of two photos was generated by AI.

The results showed that when asking them to determine which photo was real and which one was AI, over 70% of consumers on average could correctly select the AI generated image.

In all but 2 tests the percentage of respondents correctly selecting the AI generated photo were extremely high. However, when it came to selecting a photo of the Eiffel Tower in France or a painting of George Washington, the respondents failed much more frequently.

On average our respondents were able to detect the GenAI created image 71.63% of the time

Some of the AI generated images feel obvious, others more tricky as you can see when you look at them below. Perhaps the most confusing is the painting of George Washington, AI created an odd looking version of the man and our first President during the Revolutionary War with what appears to be the flag of China flying in the background. Consumers selected this really odd painting as the real version 49.11% of the time.

Our study suggest a few things with the current state of AI imagery:
1. Consumers are highly likely to be able to detect if your images are generated by AI.

2. Consumers on social media confusing AI and real imagery are possibly in the lowest quartile of detection showing there might be some other issues there such as vision impairment, technical prowess, or intelligence levels.

3. If consumers determine that AI images are poor quality or a bad fit they may hold that against your brand / product / services. High-quality custom imagery, videography, and photography then is still the best solution when viable.

4. GenAI making images for things consumers deem as acceptable is likely ok. For example we suspect people are fine with memes/jokes, video game sprites, cartoons, and diagrams being done with generative AI tools as long as the finished product is fairly clean and artistic looking. The more important the decision for the consumer, the less they might though accept AI created image content and seek out competitors who either look more realistic or are in fact authentic. This means using a tool like Pebblely to place your product on interesting backgrounds might also be fine, but using generative AI to make a video of an AI person using your product and passing it off as a real person is probably not ok with consumers.

The Images:

1. Grinch Billboard

73.01% selected the generative AI image correctly

Real:

AI Generated:

A popular ad campaign when it launched in 2018, the billboard became fodder for memelords who for years made various renditions of it. The AI version we made is a long standing Texas joke about the state, which famously has a lot of space and low cost real estate, being full.

2. Scarlett Johansen as Black Widow

88.78% selected the generative AI image correctly

Real:

AI Generated:

This one is fairly obvious, every AI system created something similar and virtually none of them would pose her the same was as the photo.

3. Photo of The Italian Countryside

88.46% selected the generative AI image correctly

Real:

AI Generated:

For some reason photos of the Italian countryside always came back as if they were realist paintings instead of photos, no matter how we structured the prompt or which system we used. We selected the best one here.

4. Photo of Jupiter

83.58% selected the generative AI image correctly

Real:

AI Generated:

Jupiter is one of the most photographed celestial objects in our solar system beyond the Moon and we thought we would get a much more realistic feeling of it from generative AI, while it looks close humans appear able to tell which of these is real quite easily.

5. Photo of a Baby Peafowl (Peacock)

87.97% selected the generative AI image correctly

Real:

AI Generated:

This is sort of a famous thing to look at and can help determine if an AI image generator knows the difference between baby versions and adult versions of animals. All generative AI systems appear to create something similar looking. And, even when asked to make it look like a photograph the vast majority tried to make it more like a cartoon, this was the only photograph version that was produced in all attempts.

6. Painting of George Washington

50.89% selected the generative AI image correctly

Real:

AI Generated:

Ah yes George Washington, notorious for his ponch and famously fighting for China in the 1700s. The results here are quite scary for anyone who cares about history.

7. Photograph of Ocean Life

70.57% selected the generative AI image correctly

Real:

AI Generated:

Ocean life we thought would be an easy target for generative AI tools to replicate photos of, but something about this image was a dead giveaway to most humans comparing the two.

8. Photo of the Eiffel Tower

18.05% selected the generative AI image correctly

Real:

AI Generated:

AI seriously tricked humans here. Perhaps it is because this is one of the most famous landmarks on the planet, or the training data for various generative AI systems include lots of photos of this structure, but most humans were unable to correctly identify the generative AI image.

9. Photo of Arrowhead Stadium in Kansas City, MO

75.12% selected the generative AI image correctly

Real:

AI Generated:

The best stadium in the NFL and frankly probably in all of sports sits in Kansas City, MO. This was the best attempt at generative AI tools making an image of the interior of the stadium with snow in it.

10. Photo of a SpaceX Starship Launch

79.84% selected the generative AI image correctly

Real:

AI Generated:

All generative AI systems seemed to get this wrong, even though there are a plethora of photos of Starship and now Starship launching on the web. Hilariously enough the best attempt at this was Grok 3 which created the extremely incorrect photo you see above.

Methodology

We surveyed 4,016 Americans through random sampling using the Eureka survey system and a Typeform based survey. Respondents were shown two images in random order and then asked to determine which one was generated by AI. Our team generated 5 different versions of an image across GenAI systems including Midjourney, Grok 3, and DALL-E / ChatGPT (versions used include -o1, -o3, -4o, note the study was completed prior to the image update to -4o which appears to have improved output on our test prompts) then selected the most accurate image to use in our testing in order to make this as challenging as possible.

We validated that each real image was indeed real cataloging the original photograph and other details if known to ensure the test was correctly pitting GenAI content versus real photography or artwork.

Some of the generative AI images were way off the mark for example the request to make a photo of Arrowhead Stadium in Kansas City, MO covered in snow was universally incorrect by all GenAI systems used as was SpaceX’s Starship launching (this one was created by Grok 3 hilariously enough). Nearly all GenAI systems tested made inaccurate baby peafowl as well and nearly all GenAI systems created extremely accurate photos of the Eiffel Tower in Paris, France.

Note on ChatGPT-4o Updated Image Generation

We held off on publishing this right as OpenAI’s newest image generation version of ChatGPT-4o was being released in order to at least see how well that model does when generating the same images with the same prompts. The updated model does a much better job with newer subjects like SpaceX’s Starship and with lesser known landmarks like Arrowhead Stadium, but it frequently crashes and the results are sometimes spotty. It might generate a prompt near perfectly one day and the next completely wrong.

Our plans do include a follow up of this study in the near future when more image generation sees similar improvements.