The human gaze meets the machine gaze as NYU’s Institute of Fine Arts explores AI in art conservation

New uses for artificial intelligence have soared n recent years as innovators seek advancement in medicine, law enforcement, and many other industries. These technologies are increasingly turning up in cultural disciplines such as filmmaking, music, and visual art—including art history and conservation, fields that traditionally rely on deep but narrow expertise and the subjective notion of connoisseurship.

Still in its infancy, AI-aided research and conservation techniques offer great potential for art conservation, a field that is as subjective as it is scientific. In recent years, AI has helped to restore damaged paintings, recover details from artifacts that the human eye cannot detect, and trace the history of valuable artworks.

“This is a negotiation between the machine gaze and the human gaze,” says Joan Kee, the Judy and Michael Steinhardt Director of the Institute of Fine Arts. “The potential is great, because there are many who are trying to think more systematically about their analysis.”

The Institute of Fine Arts hosted a panel discussion in February that brought together experts in art history, conservation, and computer science to explore the practical, ethical, and philosophical aspects of these tools. And this semester, the Institute is offering its first course in AI conservation in partnership with the Tandon School of Engineering.

This is just the beginning. Kee envisions a slow and steady approach that will grow as the data sets that the AI is trained on improve and expand.

“AI can provide valuable insights in the analysis of art, but it should be viewed as a tool rather than a replacement for human expertise,” Kee says.

NYU News met with Kee to learn about how the Institute of Fine Arts is using AI, what she thinks are its strengths and weaknesses, and how it might shape the future of art conservation.

Let’s start with the basics. How is AI used in the conservation of art?
I rely on my wonderful colleagues in the conservation center—Christine Frohnert, Lisa Conte, and Matt Hayes—to help me understand the complexity of this topic. AI is being leveraged to analyze patterns in materials and artistic techniques to assess authenticity, digitally restore damaged or missing sections of artworks, and support provenance research by tracing ownership histories, identifying stylistic connections between artworks, and mapping relationships among artists and their workshops.

As Professor Frohnert, a specialist in the conservation of  “time-based media” (works that have a durational element) has described, AI can also extend human vision by using machine learning and computer vision to perceive details beyond human capability. A striking example is its role in decoding the carbonized papyrus scrolls from Herculaneum, the Roman city that was destroyed by eruptions from Mount Vesuvius in the first century. AI has helped reveal hidden layers of text that were previously unreadable. Similarly, in the conservation of Rembrandt’s The Night Watch, deep learning—a form of machine learning that uses neural networks modeled after the human brain—was employed to digitally reconstruct missing sections of the painting based on an historical copy. These examples demonstrate how, when thoughtfully applied, AI can support and enhance traditional conservation methods.

How reliable is the current AI technology, and what are the primary concerns around it?
AI’s effectiveness depends largely on the quality and quantity of the data it is trained on. But conservation is a small field and it produces relatively limited datasets.  For example, well-documented genres like European painting are more likely to be represented while specialized artifacts and works from non-Western cultures will be less so. That’s because those works are held by museums that have enough money to digitize their collections, such as the Metropolitan Museum or the British Museum. If you have a computer trained on these datasets, they are more able to discern patterns and recurring themes in paintings that are Impressionist or Baroque, where they have lots of data points to pull from.

Professor Frohnert also reminds me that another issue is not just the quantity and diversity of data but ensuring its standardization and comparability. AI models rely on various types of data, which can introduce inconsistencies in image quality, color accuracy, and metadata completeness, all of which can impact results. 

Color discrepancies in digital reproductions may lead to misinterpretations of an artwork’s materials or even its authenticity. While researchers are working to address these limitations, transparency in AI processes remains an ongoing concern.

What about the use of AI in authenticating a work as being created by a specific artist?
At our panel on AI and art in February one of the questions that kept coming up was about using AI to establish if a work was from a specific artist. If you have a big enough data set of Renaissance paintings, for example, then it becomes easier to assess the probability of a painting being 80 percent likely to be Michelangelo’s work. But these tasks embrace subjectivity, an area where AI currently falls short. For example, AI may fail to recognize the varying cultural significance of color, instead seeing only the literal spectral hue. And AI struggles with the sensory and emotive aspects of art. While it can detect stylistic connections and material patterns, it cannot replicate the nuanced, sensory-driven human experience.

What other techniques are being developed with the help of AI?
My colleague Lisa Conte is very interested in how AI is allowing us to analyze a painting, or any artwork, without being invasive. Currently, we use techniques that are like taking a biopsy. But when there’s actual contact with a delicate object, it’s intrusive, so part of the excitement about AI is around the idea of being able to avoid that.

Tell us about “Conservation of AI-Based Artworks,” the course being offered this semester by the IFA and the Tandon School of Engineering in Brooklyn. What are the benefits of bringing engineers, art historians, and computer scientists together to explore these topics?
This is the first time a course is being offered and cross-listed between the Institute of Fine Arts and the Tandon School of Engineering. It is founded on the belief that bringing together diverse skill sets in an educational setting can foster new ideas and innovation in the fast-paced and ever-evolving field of time-based media conservation.

The course is being co-taught by Professor Deena Engel, who is a computer scientist, and Professor Thiago Hersan, an artist and AI developer. It uses six works by media artist Rafael Lozano-Hemmer as case studies, allowing students to conduct source code analysis and risk assessments. The course introduces the essential tools used in time-based media conservation and examines the programming languages behind these artworks. Students assess the artworks from both computer science and art conservation perspectives and develop documentation and conservation strategies to ensure their long-term preservation. The course emphasizes the integration of expertise from art conservation, art history, computer science, and engineering, an interdisciplinary approach that is essential to the future. We are also building a new Conservation Technologies program with our colleagues at Tandonto to collaborate with museum scientists and institutions with ongoing research projects.