Gaussian Splatting: Creator Joshua Kerr Tests Potential For Indie Filmmaking, Virtual Production (Part 3 in a Series)
Indie creator Joshua Kerr, whose YouTube channel explores virtual production and has an estimated 41,000 subscribers, recently experimented with Gaussian splatting (more on this subject can be found in Part 1 and Part 2 of the series), AI and virtual production by reimagining one of his childhood Super 8 zombie shoots. Here’s his experience and takeaways.
The Process
Kerr began by traveling to Newcastle upon Tyne, UK [where he grew up] to take iPhone photos in order to capture a 360-degree image of the area. He loaded the image into Google Gemini to start generating apocalyptic variations, including busted cars and crumbling houses. After a few edits in Photoshop, he loaded his image into World Labs’ Marble to generate a Gaussian splat.
He then used the Gaussian splats with Lightcraft Jetset to create an on-set virtual production with his iPhone. “We wanted to go out and be in the environment, as opposed to shooting on a greenscreen,” Kerr relates. “I really wanted to be on the street and kind of see what it looked like with the Jetset monitor, because we could load the splat right in and visualize the environment almost one to one with the real place.
“What was sort of striking about it was the speed that the Gaussians ran on set in terms of how well they render quickly in the phone,” he continues. “We didn’t really need a proxy. It’s showing this splat as it would be represented on a desktop computer. So there’s a real jump in quality using those [in this workflow] and the performance was really fast.”
Kerr adds that he didn’t have control of the lighting with this method, and so for relighting, he turned to Beeble AI. From there, everything was rendered in Unreal Engine before finishing (final compositing and grading) with DaVinci Resolve.
(Image courtesy of Joshua Kerr)
“There so much potential,” Kerr suggests. “As far as filmmakers go, I honestly think that there’s so much that they can do now, even with just an iPhone and an idea.”
Limitations
At the moment, Kerr finds that the key limitation to use of Gaussian splatting is flexibility. “They’re not flexible, as opposed to the way you might think of a CGI or virtual environment. … in terms of its lighting and the position of objects,” he explains. “They’re very much how you capture them. You need to either scan them or generate them in the exact way that you want to visualize them. So if you’re going out and scanning something [for] golden hour, then they need to be there at golden hour. There’s some tools that allow a level of lighting tweaking, but it’s not what I’d call physically accurate, so that’s probably where their biggest limitation is.”
Opportunities
“I don’t think that this is an inaccessible thing,” Kerr contends. “I think that Gaussians, if they’re not already, are going to be a very key part of virtual production, especially because of how useful they are and how fast they are.”
He adds that Gaussian splatting may support additional applications in VR and immersive content production. “It seems like this kind of technology would work really well in a headset,” he says, citing virtual scouting as an example of a possible use.
Kerr concludes, “my whole goal [of getting into virtual production] was to make sure that everything that could be done on a high-end studio virtual production shoot with an LED wall could be done in a bedroom with a little bit of know how and the software necessary.” Kerr opines that this is “hugely possible … There’s a lot of YouTubers like myself who do exactly this thing and try to teach it out to people, and it’s a mega opportunity.”
Check back tomorrow for the final story in this series.