If you’ve heard the term Gaussian splatting—a subject that rapidly gained attention in VFX and AI circles—but are not really sure what it means, you’re not alone.
The Creative + Tech Orbit has put together a multi-part look at this topic, including views on its potential and use cases. In this story, Chris Nichols, director of special projects at the Chaos Innovation Lab, explains Gaussian splatting with some history. Below are edited excerpts from Chris’ conversation with the Orbit’s Carolyn Giardina.
Background
Nichols: For years we have looked at 3D data in computers through “meshes” that are made through a series of polygons, and the way that polygons are defined in a computer is that there’s a series of points, and then you connect a minimum of three points and you create a triangle. That triangle is a surface, and on that surface you basically decide its color. And there could be many different colors within an area, which is usually done as a texture. The complicated part about it is that colors of surfaces are different depending on what angle you’re looking at. But the thing that’s unique about the polygons is that we assign certain parameters to that surface that are called shaders, and they change colors, depending on the camera angle and angle of the light that is hitting it.
In terms of collecting 3D data, one of the ways that we’ve been doing that is through photogrammetry, meaning we take a bunch of photos, and then we derive a mesh and a texture from that system. The problem with photogrammetry is that the color is static. It doesn’t change depending on the angle that we do. So we have to do all kinds of tricks to make it more functional.
A decade ago, light fields became more commonly known and began to be used in a practical way. What a light field does is that it sort of looks at photographs not just from a single point of view, but from multiple points of view. The problem with light fields was that it required, like, a thousand images to make it, and so much information within those images is redundant information. So you create an enormous amount of data.
Along came the “AI revolution.” The thing that’s really good about AI is that if you give it enough sparse information—meaning, for instance, instead of using one thousand points of view, one might take ten points of view—the AI can figure out the rest. The evolution was use of a neural network of these “radiance” fields, or “Nerfs,” which effectively allowed a CG artist to work with a more sparse amount of data. Very cool, also extremely complicated data that takes a lot of energy to display that and to render it, essentially to see it from all those points of view. So this solved some issue but was still cumbersome to look at.
(Photo of Chris Nichols, courtesy of subject)
Gaussian Splatting
And then a research paper was published [introducing] these things called Gaussian splats [suggesting that] instead of polygons, we just need the points—point clouds—and the thing that’s nice about points is you can create a whole lot of them. It doesn’t have the same weight as triangle systems. But instead of defining a single color on that point, which is what we normally did with points, we’re going to put a different system on that point, called a Gaussian, which is like a bell curve of color. So that’s the Gaussian part, and they call it a splat, because it’s really just a splat of color.
The cool thing about it is that it’s super lightweight, like you can actually render them on your phone. The challenge with them is how to create them. Currently, that process of training a Gaussian splat is generally done through a series of AI tools. The training of the data requires the AI but then once it’s done, it’s rendered the way you would normally render things. There’s some drawbacks, for instance, the current way of implementing a Gaussian splat doesn’t allow one to change the lighting, so stakeholders are working on approaches to address this.
I’m almost certain that almost every [company in VFX] is at least looking into it. They’re just so practical to use. [They could be used for] anything that requires 3D data.
Check back tomorrow for the next story in this series.