Published on Tue Mar 31 2020

StyleRig: Rigging StyleGAN for 3D Control over Portrait Images

Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt

StyleGAN generates photorealistic portrait images of faces with eyes, teeth, hair and context. 3D morphable face models offer control over the semantic parameters, but lack photorealism when rendered. RigNet is trained between the 3DMM's semantic parameters and StyleGAN's input.

0
0
0
Abstract

StyleGAN generates photorealistic portrait images of faces with eyes, teeth, hair and context (neck, shoulders, background), but lacks a rig-like control over semantic face parameters that are interpretable in 3D, such as face pose, expressions, and scene illumination. Three-dimensional morphable face models (3DMMs) on the other hand offer control over the semantic parameters, but lack photorealism when rendered and only model the face interior, not other parts of a portrait image (hair, mouth interior, background). We present the first method to provide a face rig-like control over a pretrained and fixed StyleGAN via a 3DMM. A new rigging network, RigNet is trained between the 3DMM's semantic parameters and StyleGAN's input. The network is trained in a self-supervised manner, without the need for manual annotations. At test time, our method generates portrait images with the photorealism of StyleGAN and provides explicit control over the 3D semantic parameters of the face.

Thu Nov 26 2020
Computer Vision
Lifting 2D StyleGAN for 3D-Aware Face Generation
LiftedGAN is able to disentangle the latent space of StyleGAN2 into texture, shape, viewpoint, lighting and generate 3D components for synthetic images. Unlike most previous methods, our method is completely self-supervised.
0
0
0
Sun Feb 24 2019
Computer Vision
3D Guided Fine-Grained Face Manipulation
We present a method for fine-grained face manipulation. Given a face image with an arbitrary expression, our method can synthesize another arbitrary expression by the same person. This is achieved by first fitting a 3D face model and then disentangling the face into a texture and a shape.
0
0
0
Sun Sep 20 2020
Computer Vision
PIE: Portrait Image Embedding for Semantic Control
Portrait editing is a popular and important research topic with a large variety of applications. For ease of use, control should be provided via a semantically meaningful parameterization. We present the first approach for embedding real portrait images in the latent space of StyleGAN.
0
0
0
Sat Mar 13 2021
Machine Learning
PhotoApp: Photorealistic Appearance Editing of Head Portraits
0
0
0
Tue Jul 14 2020
Computer Vision
JNR: Joint-based Neural Rig Representation for Compact 3D Face Modeling
0
0
0
Thu Jan 07 2021
Computer Vision
GAN-Control: Explicitly Controllable GANs
We present a framework for training GANs with explicit control over generated images. We are able to control the generated image by settings exact attributes such as age, pose, expression, etc. Using contrastive learning, we obtain GAN's with an explicitlydisentangled latent space.
1
0
1