Production Deployment

You’ve carefully formulated and specified a target problem, collected and processed training data, picked a familiar framework (or learned a new one) and through a process of research, intuition, and luck chosen an initial network architecture to represent your problem. You’ve methodically run experiments and tuned hyperparameters, getting side-tracked along the way to write tools…Read More

Mesoscopic Facial Geometry Inference Using Deep Neural Networks

We present a learning-based approach for synthesizing facial geometry at medium and fine scales from diffusely-lit facial texture maps. When applied to an image sequence, the synthesized detail is temporally coherent. Unlike current state-of-the-art methods, which assume ”dark is deep”, our model is trained with measured facial detail collected using polarized gradient illumination in a…Read More

High-Fidelity Facial Reflectance and Geometry Inference From an Unconstrained Image

We present a deep learning-based technique to infer high-quality facial reflectance and geometry given a single unconstrained image of the sub- ject, which may contain partial occlusions and arbitrary illumination conditions. The reconstructed high-resolution textures, which are generated in only a few seconds, include high-resolution skin surface reflectance maps, representing both the diffuse and specular…Read More

Human Motion Modeling using DVGANs

We present a novel generative model for human motion modeling using Generative Adversarial Networks (GANs). We formulate the GAN discriminator using dense validation at each time-scale and perturb the discriminator input to make it translation invariant. Our model is capable of motion generation and completion. We show through our evaluations the resiliency to noise, generalization…Read More

Direction-aware Spatial Context Features for Shadow Detection and Removal

Shadow detection and shadow removal are fundamental and challenging tasks, requiring an understanding of the global image semantics. This paper presents a novel deep neural network design for shadow detection and removal by analyzing the image context in a direction-aware manner. To achieve this, we first formulate the direction-aware attention mechanism in a spatial recurrent…Read More


It’s always fun to see machine learning challenges posed as well as the competing solutions, but even more so when they’re relevant to VFX. PoseTrack is a new large-scale benchmark for human pose estimation and tracking in video. We provide a publicly available training and validation set as well as an evaluation server for benchmarking…Read More

Learning to See in the Dark

Imaging in low light is challenging due to low photon count and low SNR. Short-exposure images suffer from noise, while long exposure can induce blur and is often impractical. A variety of denoising, deblurring, and enhancement techniques have been proposed, but their effectiveness is limited in extreme conditions, such as video-rate imaging at night. To…Read More

From Faces to Outdoor Light Probes

Image-based lighting has allowed the creation of photo-realistic computer-generated content. However, it requires the accurate capture of the illumination conditions, a task neither easy nor intuitive, especially to the average digital photography enthusiast. This paper presents an approach to directly estimate an HDR light probe from a single LDR photograph, shot outdoors with a consumer…Read More

DeepStereo: Learning to Predict New Views from the World’s Imagery

Deep networks have recently enjoyed enormous success when applied to recognition and classification problems in computer vision, but their use in graphics problems has been limited. In this work, we present a novel deep architecture that performs new view synthesis directly from pixels, trained from a large number of posed image sets. In contrast to…Read More