By Mona Lalwani From engadget

It took several weeks for Intel to design Gaga’s David Bowie tribute at Monday night’s award show.

When Lady Gaga started singing “Space Oddity” at the Grammys on Monday night, virtual drops of red paint dripped down her face to form a lightning bolt. The image — a meticulously planned facial projection — instantly evoked David Bowie’s face on the cover of Aladdin Sane. For the six-minute tribute to the legend who passed away a little over a month ago, Gaga employed an assortment of cutting-edge tools to create a kaleidoscopic visual treat.

In a way that’s true to her brand of music and eccentricity, Gaga brought together art, technology and fashion to create this spectacular homage on national television. While Bowie fans continue to weigh in on whether the tribute did justice to his legacy, the fact that she introduced a new dimension to live performances is undeniable.

Intel and Haus of Gaga came together to test technologies that could translate Gaga’s unconventional ideas into a transformative live performance. The supergroup of technologists, engineers and designers set up a pop-up laboratory in Los Angeles to work on iterations months before the final show. But when Bowie passed away, Gaga and her team reshaped their ideas and techniques to create a tribute within just a few weeks.

From the projection that gave Gaga a “digital skin” to a holographic tornado of fluorescent colors toward the end, the performance integrated music and technology in a way that hadn’t been done before in a live setting of this magnitude. I caught up with Paul Tapp, the director of technology at Intel, and Nancy Tilbury, the creative director at Haus of Gaga, for a complete breakdown of the technology that powered the performance.

How did the collaboration between Lady Gaga and Intel come about? And how long was it in the making?

Paul Tapp: Gaga is someone who has always taken pride in pushing the boundaries but has always embraced technology as well. From Intel’s perspective, we’ve been involved in the music industry, creating music and performances for over 15 years at this point. As an innovation company, we love to be pushed and we love hearing firsthand how we can make the impossible possible.

When we first started talking to Gaga nearly a year ago, she had a whole bunch of problem statements for us. She asked, “Can you help me make my environment come alive? Can my instrument take on a new dimension? What about standard constraints for how I express myself through music and makeup and clothing? Can we take it to a new dimension?” She was clear on what she wanted to do. It helped us to immediately think about how to solve those problems from an engineering perspective.

We started figuring out what sort of technologies to use and what she would do with those onstage. When Bowie unfortunately passed, she said, “Guys, I have to do this, Bowie is critical to who I am; he’s influenced me hugely in my career. I know we don’t have much time, but can you help?” We rose to the occasion because we fully support that from a creative perspective but also Bowie himself is someone who pushed the boundaries. He was unafraid of taking risks and making new statements.

What was Gaga’s vision for the performance? What did she hope to achieve?

Nancy Tilbury: In everything that we do there’s an element of magic with Gaga. We knew that we wanted to use technologies like holography and to show visualizations of galaxies — the relationship between creativity and science. And, of course, the wondrous vision of Bowie in the end shot, looking at the stars and looking at him. There’s a level of spirituality in all of her performances. We push to find visualizations that make people think deeper than just using standard screens and colors; the content has to be inspiring too. Ruth [Hogben, the director of Haus of Gaga] worked tirelessly on that. There was hardware and software working together. In terms of wearable tech — the ring she was wearing — we worked to build the narrative and the technology together, so when the ring moved, the ambience changed.

Gaga seemed to control parts of the performance with those orb-shaped rings on her hands. Was that wearable technology created specifically for this performance?

Tapp: It came about when Gaga said,”I’m a musician. I can sing, I love fashion but how can you help me express myself beyond my traditional constraints — beyond my body?” This is where the Intel Curie ring came in. We have a really small piece of silicone that has the capabilities of a computer processor. It’s got sensors like a mobile phone and it’s got radio-transmitting capabilities. So we could send the positional data from three dimensions directly from Gaga’s hands through a network to our [Intel] Xeon servers at the back that controlled the stage elements. So Gaga’s movements literally caused the stage to respond in real time. It was robust: When she moved, it moved.

Tilbury: My business partner Benjamin [Males] at XO [a London-based fashion and technology studio] is an engineer, and I’m a designer. We came on board as partners of Haus of Gaga. I think we pushed the Curie team quite hard from a design and tech point of view. Our starting point was this tiny chip that I had to design around. I knew that because of the distance and the fact that it was televised, we would need to have some scale and depth so people could see what the ring was doing. That’s why we made it larger: It was not to do with the size of the technology but more so people could see the magic of the ring connecting to the environment. I worked with an engineer in the UK to produce the ring. There’s so much detailing that went into it. It’s like designing a piece of couture, except it has this complex piece of technology inside. Gaga was supposed to wear only one, but when she showed up she said, ‘”I love this so much I have to have two.”

What visual aspects of the performance did the Curie ring control?

Tapp: When she was singing “Rebel Rebel,” the back screen, which was a rich LED display that the Grammys provided, had Gaga in a rotational ghosting effect. As she moved around, the colors and the ghosting on-screen were responding to her positions in the choreography. The rings on her hands were controlling that effect. Then in the end for the big finale, where she was with Nile Rodgers belting out “Heroes,” she had this florescent pink smoke emanating from the ring. Nile had a band with the same Curie chip on the end of his guitar, so as he was playing there was a green cyan kind of smoke coming out from his guitar. The smoke [effects] combined in a holographic effect to produce this apparition of David Bowie’s face.

What about the facial projection at the beginning? How did you employ the “living canvases” technology to project visuals onto Lady Gaga’s face to create the digital skin?

Tapp: On the stage, there were two parts. There was tracking of Gaga’s facial expressions and movements in real time. The second was being able to take the graphical 3D makeup, twist and distort it based on the position of her face that we were tracking and project it accordingly.

We had 10 infrared markers that were designed specifically to add to her looks. So they were discrete and beautiful rather than distracting. We had five cameras mounted around her. They were infrared sensors that picked up on the infrared light coming from the markers. Those cameras were sending digital data through an Intel Core i7 processor–based system, which then told the projector exactly where Gaga’s face was and what the expression was because she was singing. The processor calculated what the image should be and how to distort it so the makeup looks correct. There was a huge amount of processing and sending of data.

Another highlight of the show was the “holographic” explosion of colors at the end. Right from Pepper’s ghost illusions to projections on mesh screens, the word “hologram” has become a buzzword for many variations of 3D projections. What kind of technology did you use to create that effect?

Tapp: I’ll be very transparent about that: Holograms don’t really exist. There’s an ongoing joke in the display industry: If you go back 60, 70 years and if you’d asked anyone in the industry, “When are we going to have holograms?” they would say, “In about 10 years from now.” But if you ask people in the industry the same question today, they will say the same thing.

That said, one of the things we talked about with Gaga was that we can sense you in three dimensions. We have Intel Curie and RealSense, both of these are working in a new dimension, so why don’t we use that three-dimensional input to create a display? While working with Benjamin, we figured that the best way to give that 3D effect in a live stage environment was to use this material called Holo-Gauze. It has a special silver substrate on it, which catches the light much more intensely than a mesh would. We had to make a lot of decisions going into it. There’s so much lighting that you actually don’t have any control over [at the Grammys]. And Gaga wanted the tribute to be so magnificent in terms of live musicians and dancers, we had to factor that into where we position things on the stage. Holo-Gauze was a great way of putting up what we did. From the cameras and the audience perspective, I think it looked nice.

The 58th GRAMMY Awards – Show
How much of Intel’s RealSense technology was involved in the show?

Tapp: It was used in the making of the performance but not onstage. The only way we ended up using RealSense was for 3D scanning [Gaga’s face] to help us build the facial-projection technology.

What were the biggest challenges with the performance?

Tapp: Gaga and Intel had a healthy pushing of each other to go beyond the comfort zones. To be frank about it, there were things she asked for that we couldn’t do in the time frame. But we were also pushing her because this was entirely live. The technology was so integrated in the performance that we couldn’t have anything screw up.

When David Bowie unfortunately passed and Gaga said, “I have to do this,” we were nervous about the development timelines at that point. We had to start with a completely new creative one month before going live. We’ve never done projection in a live performance on a singing face before. And that content can take three, four months to develop. We didn’t have that time. It was a bit of a roller coaster.

“What she put on was about the infusion of creativity and technology. It’s the combination of an excessively creative mind and engineers who are willing to make the technology to meet the demands of that creative mind.”

Tilbury: The facial projection came with so many risks. It’s a piece of R&D technology, and I believe we’ve pushed it to the point that you’ll see many artists use these kind of technologies in live performances in the next couple of years. But she was the first, and it’s really a testament to her. She takes those risks. To be able to be in an environment with brand-new technologies and to allow millions of people to see it through popular culture is fantastic.

Tapp: Even the Curie wasn’t a slam dunk. Nancy and Ben put challenges to Intel. They wanted the ring to change color and have different colors within the same ring at the same time. Then there were FCC regulations and the very fact that we were in an auditorium with 12,000 mobile phones. We had to make sure there was no radio magnetic interference through the performance.

What does this integration of sensors and processors mean for the future of music?

Tapp: What she put on was about the infusion of creativity and technology. It’s the combination of an excessively creative mind and engineers who are willing to make the technology to meet the demands of that creative mind. I think you can do innovative things without necessarily inventing the future. We didn’t have to invent anything new, but we had to apply technology in a completely new way. Hopefully we’ve given a message that the sky’s the limit. We have the tools to free your creativity from traditional constraints. I hope we’ll see the bar raised higher and higher, but to do that you have to take risks. And that’s where Gaga stands out.

Tilbury: For us, the future is here. Some of the things that people have been dreaming up in comic books and science fiction are really beginning to come to life with some of the miniaturization of technologies that companies like Intel are creating. The performance was a step forward in understanding how to put the technology in people’s hands.

[Image credit: Lester Cohen/WireImage (top); Jeff Kravitz via Getty Images (bottom); gifs from Intel’s behind-the-scenes footage]

For more on this story and video go to: http://www.engadget.com/2016/02/18/inside-lady-gagas-high-tech-grammy-performance/