The latest word tour by Korean artists, BTS, was conceived to be one that communicates with the fans present at the performances and concerts. For this, AR was used in specific situations and Live-Lab, a disguise studio partner based in Seoul, recommended the use of disguise to power the experience.
Born out of conversations between management company Big Hit, creative directors Plan A, and production designers FragmentNine came the idea of using augmented reality for the song “Trivia: Love”.
Tasked with turning these ideas into reality, FragmentNine, consisting of principal designers Jeremy Lechterman and Jackson Gallagher, reached out to All of it Now [AOIN] to begin exploring possibilities. The AR content ultimately comprised words, shapes, and a heart motif that the artist could interact with.
AOIN were tasked with integrating AR into the stadium-scale tour using the existing camera and video system.
Kevin Zhu, one of AOIN’s AR content designers, said: “Creatively we needed to help the artists, management and production understand what AR content entailed and how it would look. This involved a lot of discussion and client education on the variables that impact the end product: lighting, size,scale, colour, front plate compositing and more. This kind of discussion was necessary because the technology and implementation of AR graphics was so new to the major stakeholders. Ultimately, it was a good learning experience all around, and we will be taking some of those lessons forward into projects.”
On the technical side, AOIN had to overcome the unknowns associated with the integration of AR workflows. Bero Mora, producer and AR content designer, noted: “We have done many tests within our studio where everything works in a controlled environment, but when you are on-site you have to deal with real-world issues. Since a lot of this technology is relatively new, we have to deal with a lot of unknown issues and unwritten features. Working closely with our partners, such as disguise, allows us to accomplish what we need to make these workflows operational in the real world.”
According to AOIN executive producer, Danny Firpo: “Understanding the performance on a stadium scale was an important factor. The sheer scale of a stadium setting meant that the performer was almost too small to see in person except for those in the front row. A significant percentage of the audience would experience the show and watch the performer through the IMAG screens with the AR effects.”
The disguise workflow was vital in making creative content decisions without having the full AR rig set up. Firpo explained: “We used the spatial mapping feature on the gx 2 pretty heavily, which allowed us to virtually block out all the camera positions and AR effects on stage from within our studio. Management could approve the changes and revisions remotely using disguise’s stage render video as a virtual proxy of the actual performance.”
The integration of disguise with the stYpe camera tracking system allowed AOIN to link virtual camera objects in disguise’s software GUI with real-time lens and 6D positional data the stYpe unit received from a camera and broadcasting output. stYpe’s Redspy system was used for the handheld camera, and a Vinten Tracking head was used for tracking a stationary camera at the FOH camera riser position.
Neil Carman, touring AR engineer, explained: “We had to make a virtual stage line up perfectly with the real stage, and we were only able to see this alignment through the live camera feed into the gx 2. So, disguise became a powerful real-time compositing tool that helped us understand adjustments we needed to make for perspective and position. The virtual stage assets that FragmentNine delivered to us for the project were scaled perfectly for the real-world space. Being able to verify scale within the disguise software made aligning the real world and the virtual world much simpler.”
Once disguise had all the connected Stype systems agree on a universal zero point, from which all graphics would emanate, the server allowed AOIN to precisely move around individual elements of the graphics from different sections during rehearsals. This aided AOIN in choreographing interactive movements between the performers and the graphics.
Alvin Chu, CEO of Live-Lab, said: “The stability of the hardware and flexibility of the software on the disguise server contributes greatly to the success of a long tour.”