Artists Emerge from their Paintings: Museum AR Experiences of Art and Artifacts

Artists Emerge from their Paintings is an augmented reality application where a museum visitor viewing a painting can have a direct interactive experience with the artist, the world of the painting, and the artwork’s context. Users wearing a Hololens head-mounted display suddenly experience a whole 3D scene when the hardware and application detects a VanGogh painting hanging on the wall.  Upon approaching the painting, the users encounter a lifesize VanGogh who speaks about his art using the actual words of Van Gogh. The 3D scene around the user includes the lifesize reproductions of the furniture in the last room in the asylum before the end of his life by suicide.

In the Artists Emerge from their Paintings augmented reality application, virtual characters, objects, and information can appear in the room in front of the painting, on the same wall beside or over the canvas, and the application can even create a 3D space behind the wall of the room where the user is standing. 

The application includes several interactive features. On the table are letters. When the user picks a letter, a replica of the physical letter appears in the visitor’s hand. Van Gogh voices the letter to Theo, Van Gogh’s brother.  When the user picks up the sketchbook, Van Gogh’s preliminary sketches magically appear superimposed on the real painting. 

The application was developed at the NJIT M.I.N.D. Lab (http://www.nymindlab.com) by Dr. Kum-Biocca, a team of student artists and programmers including NJIT MIND Lab Students team_ Brandon Kong (Digital Design, 4th year), June Enriquez (Digital Design, 4th year), and Abhijit Sprijanth (Programmer, SketchUp Trimble).

An augmented reality application, Artists Emerge from their Paintings, won two top awards at the international EuroXR conference held in Valencia, Spain.  Dr. Hannah Kum-Biocca and the team at her M.I.N.D. Lab received the Best XR application award and a top XR Demonstration award.  XR is a widely accepted term that covers Virtual Reality, Augmented Reality, and Mixed Reality (VR/AR/MR)