Welcome to The Creative + Tech Orbit. If you’re reading this, then you either subscribed or someone forwarded it to you. It’s free to subscribe. If you are not already part of our community, welcome, please join us and subscribe by clicking here. LINK
This final story in The Creative + Tech Orbit’s series of Gaussian splatting (Part 1, Part 2, Part 3) examines how splats were used to create finished shots in DC Studios’ tentpole Superman, directed by James Gunn and released by Warner Bros., as well as its potential as part of actor-friendly techniques.
In the film, splats were part of a cutting-edge method used to create roughly 40 shots of Superman’s Kryptonian parents, played by Bradley Cooper and Angela Sarafyan, who appear in a corrupted holographic message to their newborn.
The filmmakers wanted to depict the holographic message in a unique way, but it was also an important story point. “They wanted it to be emotional and not look like a 3D visual effect,” Kevin Sears, CG supervisor out of Framestore Montreal, tells the Orbit. “They wanted it to be real people performing, but they wanted it to be 3D and have all this control.”
They needed control and flexibility because many creative decisions including camera moves would not be made until after principal photography. “It was also going to be a challenge because the holograms were not just human scale. They were going to be an unknown size and there were also going to be multiple iterations of the hologram message inside the Fortress of Solitude,” Sears relates, adding that a final requirement was that they would need to make the holographic message appear to be corrupted.
For these reasons, overall VFX supervisor Stephane Ceretti and London-headquartered VFX studio Framestore teamed with Infinite Realities, a UK-based volumetric capture company that introduced an inventive approach to using Gaussian splatting, which was a relatively new concept at the time. After a series of tests, they captured the actors in costume on Infinite Realities’ spherical Deus Capture Stage with 192 cameras at 24 frames per second. “Having two performers next to each other was very important, and that was the first time they had done that,” Sears relates. “There was a lot of planning for the camera coverage and getting the right lighting onto all the frames that we’re going to need from all the different angles. The actors performed facing forward, because that’s what the story dictated.” Gunn watched remotely and directed the actors.
They did a few takes of the two-minute performance and then employed machine learning to create Gaussian splats of the performance at each angle. “It ended up taking over five days to take all the video feeds from each the 192 cameras. And what we’re doing with machine learning [to create the splat] is on every point of their bodies, there’s a color value, but from every angle, these cameras are being trained to determine what that value is from every different angle.” That data was then provided to Framestore to get to work on the actual shots.
Sears reports that in the end, they had the freedom to decide where to put the camera, focus lengths, zooms — “and do different takes and stage their performance in a way that we wanted, to completely control after the fact. But then, when you’re rendering it, it looks just like real footage. It’s not a digital person walking around. It’s not AI, it’s not us changing the performance in any way. We just use machine learning to train that data set, and then it is your performance as a human being just being ‘refilmed’ from any creative angle.”
(Pictured: Deus Capture Stage at Infinite Realities. All images courtesy of Framestore)
Initially, “it was quite heavy in terms of the data set and Infinite Realities did a lot of the heavy lifting,” Sears says, noting that “ultimately the data set that came to us was several hundred gigabytes of the performance and nowadays, not really that big of a deal. … It wasn’t cumbersome. It wasn’t slow. We had this kind of one set that we could move a camera around and play with it and then test things out. So it was quite quick to make iterations.” The final images were rendered at 4K resolution and put into the plate photography.
Sears explains that at the point when the message is corrupted and unstable, they were able to manipulate the points to “offset them and rotate them and make them kind of jittery and shake it round. That was really cool that we had all the photography from every angle. Because when we broke the points apart and made them shake and have the message corrupted, you could see the same lighting information properly from the right angles all stored in that data set.”
He adds, “the main advantage was the quality assurance that James Gunn wanted this important story point to be humans giving a real performance.”
Looking Ahead
Sears see potential for use of Gaussian splatting in visual effects and he describes this potential as “groundbreaking” for human performances. “It’s kind of like theater in the round. In no way we’re trying to replace an actor’s performance,” he says. “When it comes to Gaussian splats, we’re just setting up all the tools to be able to capture this moment in time and and so we’re not going to change your arm or your face or performance. We’re not changing you. This is you. We’re just able to now decide how we want to stage it later in post and I think that was really exciting to see the actors embracing that.”