Apple Sharp

Recently, researchers from Apple published “Sharp Monocular View Synthesis in Less Than a Second”, a research paper detailing their approach to producing a 3D Gaussian model from a single image. In addition, Apple made the neural network model and other project files available for others to evaluate.

To test Sharp, I found a photo on Pexels with a good amount of depth and ran it through Sharp, then I took the resulting splat model into OctaneRender. In this test you can see how I’m not only able to move around the image and readjust focus, but it's really simple to place 3D objects into what is effectively a 2D image. By matching the scene lighting to the lighting in the photo, I can have 3D objects cast shadows onto the splats.

There are some considerations; for one, this isn’t a generative AI process, there’s no inference of the rest of the scene outside of what was captured in the original photograph. As you see later in the video, if we try to look at the scene from another angle, it very quickly breaks down. This can affect the ability of that scene to cast shadows and appear in reflections, as only the parts that appear in the original image actually exist to do those things.

On reflections; unlike full Gaussian splat models created from a range of images, no “view-dependent” effects are recreated with Sharp. That means that reflections don’t shift as we change perspective on reflective surfaces such as the car body.

All that said, I do think this could evolve into a really compelling way to composite CGI elements into footage. So whilst Sharp is currently only available under a non-commercial licence, it offers a glimpse into the future of working across 2D and 3D content.