Relighting 3DGS
Resulting in 3D reconstructions that manage to be lightweight, yet can be packed with the small details that can make them indistinguishable from a high resolution photo or video of the same thing, and with SideFX including 3DGS support in the latest update to Houdini, it seems like new pathways to using this powerful technology in VFX are opening up every day.
But there is one glaring limitation, Gaussian splats are very tricky to relight.
The issue is that, unlike traditional 3D rendering, there’s no physically-based material and shader that is interacting with the CG lighting within the scene. Nothing saying “this surface is shiny, it should catch the light in this way ” or “this surface is slightly translucent so it should allow some light through”, only a true-to-life, but ultimately baked-in reconstruction of how the light interacted with the subject when the dataset of images was captured.
With mesh-based photogrammetry processes, you might want to capture the subject matter with very flat, neutral lighting, in order to reduce the appearance of shadows. This is so you would have maximum flexibility to relight the resulting photogrammetry model. However, it could be that the best approach with 3DGS is to try and match the lighting conditions when capturing to the lighting in your intended use, since you won’t have that same flexibility.
Of course, just because it’s tricky to relight 3DGS, doesn’t mean it’s impossible. Many of the tools for working with splats have ways to make your models respond to the lighting in the scene. Here’s an example where we’ve created a 3DGS model from a drone video, and then placed the model into a scene in Unreal where it’s lit by the scene’s lighting. Keeping the changes subtle goes a long way to selling the relighting.
Drone footage sourced from Pexels under Creative Commons license for the purposes of this demonstration
I’m sure many more tools for working with 3DGS are coming our way, with more work being done to help artists mitigate this limitation.
