A comprehensive study involving 11,000 participants revealed surprising insights into the perception of AI-generated art. Most people struggled to differentiate between human-made and AI-created images, scoring only slightly above chance. Interestingly, participants showed a slight preference for AI-generated works, even among those who claimed to dislike AI art. The study uncovered significant biases in art appreciation based on perceived style rather than actual origin. Professional artists demonstrated better discernment, but the results challenge conventional notions of art appreciation and creativity. This report examines the methodology, key findings, and implications of this thought-provoking study, shedding light on the evolving relationship between human perception and AI-generated art.

Test Setup and Methodology

The study, conducted by Astral Codex Ten, involved a large-scale survey of 11,000 participants ¹. The test comprised 50 pictures, which participants were asked to classify as either human art or AI-generated images. The selection included famous paintings and AI-generated art across various styles, including Renaissance, 19th Century, Abstract/Modern, and Digital.

To ensure fairness, the test creators deliberately excluded obvious AI “tells” such as garbled text or misshapen hands. They also avoided certain styles that are easier to identify as AI-generated. As noted in the source, “I tried to make the test as fair as possible by including only the best works from each category; on the human side, that meant taking prestigious works that had survived the test of time; on the AI side, it meant tossing the many submissions that had garbled text, misshapen hands, or some similar deformity.” ¹

Key Finding: Difficulty in Identifying AI Art

The study revealed that most participants struggled to accurately identify AI-generated art. The median score on the test was 60 out of 100, only slightly above chance (50/100). The mean score was 60.6%. Participants rated the task as harder than expected, with a median difficulty of 4 on a 1-5 scale.

Importantly, 40% of the time, participants incorrectly identified AI vs human art. This result suggests that AI artists may have passed a modified Turing Test for art, as Alan Turing recommended that if 30% of humans couldn’t tell an AI from a human, the AI could be considered to have “passed” the Turing Test ¹.

Style Bias in Judgments

A significant finding of the study was the strong influence of style bias on participants’ judgments. Despite being warned not to judge art based solely on its style, respondents often made incorrect assumptions based on the artistic genre.

For instance, participants incorrectly assumed that Impressionist paintings were human-made, while digital art was often mislabeled as AI-generated. As stated in the source, “Your instincts were worst for Impressionism; you identified every single Impressionist painting as human except the sole actually-human Impressionist work in the dataset.” ¹

This bias led to some interesting results, such as Mitchell Stuart’s “Victorian Megaship” being incorrectly identified as AI-generated by 84% of participants, despite being a human-created digital image.

Surprising Preference for AI-Generated Images

One of the most unexpected findings of the study was the slight preference participants showed for AI-generated images over human art. Six of the top ten most-liked images were AI-generated, with the top two favorite images being AI-created.

This preference persisted even among those who claimed to dislike AI art. As noted in the source, “The 1278 people who said they utterly loathed AI art (score of 1 on a 1-5 Likert scale) still preferred AI paintings to humans when they didn’t know which were which (the #1 and #2 paintings most often selected as their favorite were still AI, as were 50% of their top ten).” ¹

Expert Perception and Performance

While the average participant struggled to distinguish between human and AI art, professional artists showed better discernment. They scored higher on the test, with averages ranging from 66-68 out of 100, compared to the overall average of 60.

Some individuals achieved near-perfect scores, with five out of 11,000 participants scoring 98 out of 100. These results suggest that trained artists may have a keener eye for the subtle differences between human and AI-generated art.

One experienced artist provided insight into how they distinguished between human and AI art: “When real pictures have details, the details have logic to them. I think of [the painting] Ancient Gate being in the genre ‘superficially detailed, but all the details are bad and incoherent’.” ¹

Philosophical Implications and Analogies

The study raises important questions about the nature of art and human perception. It challenges notions of what constitutes art and suggests that human biases in art appreciation may be based more on perceived origin rather than actual content.

One participant drew an analogy between AI art and nutrition, comparing it to replacing food with nutritious protein shakes: “Imagine if everyone got the ability to create mostly nutritional adequate meals for like five cents, but they all were mediocre rehydrated powder with way too much sucralose or artificial grape flavor or such… You’d probably be like ‘wow, I’m happy that you have easy access to food you enjoy, and it is convenient for me to use sometimes, but this is kind of driving me crazy’.” ¹

Conclusion and Interpretation

The study’s results suggest that art judgments may be more influenced by beliefs about origin than actual content. As the source states, “Humans keep insisting that AI art is hideous slop. But also, when you peel off the labels, many of them can’t tell AI art from some of the greatest artists in history.” ¹

These findings raise questions about human perception of art and the nature of creativity. They challenge us to reconsider our preconceptions about art and its origins, and suggest that AI may be capable of producing works that are indistinguishable from, and sometimes preferred to, human-created art.

The study concludes with a provocative statement: “Marcel Duchamp famously tried to put a urinal in an art museum to challenge people’s view of what art was. […] By this standard, I submit that Sam Altman is the greatest artist of the 21st century.” ¹ (Sam Altman, as the CEO of OpenAI and a key figure in the development of advanced AI systems like GPT-4 and image generator Dall-E, plays a pivotal role in this artistic revolution, inadvertently becoming a modern-day Duchamp in the world of AI-generated art.) This underscores the potential of AI to not only create art but to challenge our very understanding of what art is.

Source:

1. https://www.astralcodexten.com/p/how-did-you-do-on-the-ai-art-turing