High-Fidelity Facial Reflectance and Geometry Inference From an Unconstrained Image

We present a deep learning-based technique to infer high-quality facial reflectance and geometry given a single unconstrained image of the sub- ject, which may contain partial occlusions and arbitrary illumination conditions. The reconstructed high-resolution textures, which are generated in only a few seconds, include high-resolution skin surface reflectance maps, representing both the diffuse and specular…Read More

Direction-aware Spatial Context Features for Shadow Detection and Removal

Shadow detection and shadow removal are fundamental and challenging tasks, requiring an understanding of the global image semantics. This paper presents a novel deep neural network design for shadow detection and removal by analyzing the image context in a direction-aware manner. To achieve this, we first formulate the direction-aware attention mechanism in a spatial recurrent…Read More


It’s always fun to see machine learning challenges posed as well as the competing solutions, but even more so when they’re relevant to VFX. PoseTrack is a new large-scale benchmark for human pose estimation and tracking in video. We provide a publicly available training and validation set as well as an evaluation server for benchmarking…Read More

From Faces to Outdoor Light Probes

Image-based lighting has allowed the creation of photo-realistic computer-generated content. However, it requires the accurate capture of the illumination conditions, a task neither easy nor intuitive, especially to the average digital photography enthusiast. This paper presents an approach to directly estimate an HDR light probe from a single LDR photograph, shot outdoors with a consumer…Read More

DeepStereo: Learning to Predict New Views from the World’s Imagery

Deep networks have recently enjoyed enormous success when applied to recognition and classification problems in computer vision, but their use in graphics problems has been limited. In this work, we present a novel deep architecture that performs new view synthesis directly from pixels, trained from a large number of posed image sets. In contrast to…Read More

DeepWarp: DNN-based Nonlinear Deformation

DeepWarp is an efficient and highly re-usable deep neural network (DNN) based nonlinear deformable simulation framework. Unlike other deep learning applications such as image recognition, where different inputs have a uniform and consistent format (e.g. an array of all the pixels in an image), the input for deformable simulation is quite variable, high-dimensional, and parametrization-unfriendly….Read More

Unsupervised Geometry-Aware Representation for 3D Human Pose Estimation

A great example of how domain-specific knowledge can help design network architecture, in this case helping them make the jump from supervised learning (where training data may be difficult or time-consuming to acquire) to unsupervised learning (where training data is often plentiful). Modern 3D human pose estimation techniques rely on deep networks, which require large…Read More

Deep Unsupervised Intrinsic Image Decomposition by Siamese Training

Intrinsic image decomposition means splitting the observed color of a scene into its underlying components, such as illumination and reflectance. Once this process has been performed the layers can be manipulated independently before being recomposed to recreate a modified scene. What’s particularly interesting about this work is that it uses unsupervised training, which by definition…Read More

Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation

Although this technique to estimate camera motion and decompose the scene into rigid/dynamic motion (a potential aid to segmentation) relies on a depth channel in the input images, it may not be long until new approaches are developed which can operate on RGB only. Estimation of 3D motion in a dynamic scene from a temporal…Read More

Learning Category-Specific Mesh Reconstruction from Image Collections

Some truly remarkable results given the dataset from which this model was trained. Code on github will be available in the future. We present a learning framework for recovering the 3D shape, camera, and texture of an object from a single image. The shape is represented as a deformable 3D mesh model of an object…Read More