A fantastic proof-of-concept for using a normal pass, a depth pass, an alpha channel, and a material ID pass to generate a plausible image!
The task of generating natural images from 3D scenes has been a long standing
goal in computer graphics. On the other hand, recent developments in deep
neural networks allow for trainable models that can produce natural-looking
images with little or no knowledge about the scene structure. While the
generated images often consist of realistic looking local patterns, the overall
structure of the generated images is often inconsistent. In this work we
propose a trainable, geometry-aware image generation method that leverages
various types of scene information, including geometry and segmentation, to
create realistic looking natural images that match the desired scene structure.
Our geometrically-consistent image synthesis method is a deep neural network,
called Geometry to Image Synthesis (GIS) framework, which retains the
advantages of a trainable method, e.g., differentiability and adaptiveness,
but, at the same time, makes a step towards the generalizability, control and
quality output of modern graphics rendering engines. We utilize the GIS
framework to insert vehicles in outdoor driving scenes, as well as to generate
novel views of objects from the Linemod dataset. We qualitatively show that our
network is able to generalize beyond the training set to novel scene
geometries, object shapes and segmentations. Furthermore, we quantitatively
show that the GIS framework can be used to synthesize large amounts of training
data which proves beneficial for training instance segmentation models.
Have you seen some amazing research which should be covered here? Contact us or let us know in the comments section below!