Published on Mon Aug 24 2020

Monocular Reconstruction of Neural Face Reflectance Fields

Mallikarjun B R., Ayush Tewari, Tae-Hyun Oh, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Mohamed Elgharib, Christian Theobalt

The reflectance field of a face describes the reflectance properties responsible for complex lighting effects including diffuse, specular, inter-reflection and self shadowing. Most existing methods for estimating the face reflectance from a monocular image assume faces to be diffuse. We show that our method outperforms

0
0
0
Abstract

The reflectance field of a face describes the reflectance properties responsible for complex lighting effects including diffuse, specular, inter-reflection and self shadowing. Most existing methods for estimating the face reflectance from a monocular image assume faces to be diffuse with very few approaches adding a specular component. This still leaves out important perceptual aspects of reflectance as higher-order global illumination effects and self-shadowing are not modeled. We present a new neural representation for face reflectance where we can estimate all components of the reflectance responsible for the final appearance from a single monocular image. Instead of modeling each component of the reflectance separately using parametric models, our neural representation allows us to generate a basis set of faces in a geometric deformation-invariant space, parameterized by the input light direction, viewpoint and face geometry. We learn to reconstruct this reflectance field of a face just from a monocular image, which can be used to render the face from any viewpoint in any light condition. Our method is trained on a light-stage training dataset, which captures 300 people illuminated with 150 light conditions from 8 viewpoints. We show that our method outperforms existing monocular reflectance reconstruction methods, in terms of photorealism due to better capturing of physical premitives, such as sub-surface scattering, specularities, self-shadows and other higher-order effects.

Fri May 14 2021
Computer Vision
Predicting Surface Reflectance Properties of Outdoor Scenes Under Unknown Natural Illumination
The paper proposes a framework to predict surface reflectance properties of outdoor scenes under unknown natural illumination. The model is based on a combination of phenomenological and physics-based scattering models.
1
0
0
Sat Dec 02 2017
Computer Vision
SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild
SfSNet is designed to reflect a physical lambertian rendering model. It learns from a mixture of labeled synthetic and unlabeled real world images. This allows the network to capture low frequency variations.
0
0
0
Mon Dec 07 2020
Computer Vision
NeRD: Neural Reflectance Decomposition from Image Collections
Decomposing a scene into its shape, reflectance, and illumination is a challenging but essential problem in computer vision and graphics. NeRD is a method that achieves this decomposition by introducing physically-based rendering to neurological radiance fields.
2
0
0
Wed Jun 27 2012
Machine Learning
Deep Lambertian Networks
Illumination variations can be explained by changing only the lighting latent variable in our model. By transferring learned knowledge from similar objects, albedo and surface normals estimation from a single image is possible.
0
0
0
Mon Mar 29 2021
Computer Vision
Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing
Robust face reconstruction from monocular image in general lighting conditions is challenging. Methods combining deep neural network encoders with differentiable rendering have opened up the path for very fast monocular reconstruction of geometry and lighting.
1
0
0
Sun Mar 27 2016
Computer Vision
DeLight-Net: Decomposing Reflectance Maps into Specular Materials and Natural Illumination
This is a notoriously difficult problem,yet key to various re-rendering applications. We propose a Convolutional Neural Network (CNN) architecture to construct both material parameters (i.e. Phong) as well as illumination.
0
0
0