3D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models
Gimin Nam
Mariem Khlifi
Andrew Rodriguez
Alberto Tono
Linqi Zhou
Paul Guerrero
[Paper]
[GitHub]
Figure 1. We propose 3D-LDM as a diffusion model that generates neural implicit representations of 3D shapes. Our approach applies a diffusion model to the latent space of an autodecoder for neural implicit shapes, allowing us to generate 3D shapes with continuous surfaces using a diffusion model.

Abstract

Diffusion models have shown great promise for image generation, beating GANs in terms of generation diversity, with comparable image quality. However, their application to 3D shapes has been limited to point or voxel representations that can in practice not accurately represent a 3D surface. We propose a diffusion model for neural implicit representations of 3D shapes that operates in the latent space of an auto-decoder. This allows us to generate diverse and high quality 3D surfaces. We additionally show that we can condition our model on images or text to enable image-to-3D generation and text-to-3D generation using CLIP embeddings. Furthermore, adding noise to the latent codes of existing shapes allows us to explore shape variations.

Unconditional sampling result comparison


Figure 2. Result comparison between 3D-LDM auto-decoder (left) with 3D-LDM + DeepSDF.

Figure 3. Result comparison between SPAGHETTI (Left) with 3D-LDM + DeepSDF.

Model Architecture
Figure . Overview of our architecture. In the first Stage, an auto-decoder is pre-trained to represent a set of 3D shapes; in the second stage, a diffusion model is trained on the latent space of the auto-decoder. At generation time, the diffusion model creates latent shape codes that are decoded into neural implicit 3D shapes by the pre-trained auto-decoder.

Acknowledgements

This template was originally made by Phillip Isola and Richard Zhang for a colorful ECCV project; the code can be found here.
This project was possible thanks to the support from the backers at MIT's Summer Geometry Initiative 2022.