Published on Wed Feb 08 2017

Video Frame Synthesis using Deep Voxel Flow

Ziwei Liu, Raymond A. Yeh, Xiaoou Tang, Yiming Liu, Aseem Agarwala

We address the problem of synthesizing new video frames in an existing video. Traditional optical-flow-based solutions often fail, while newer neural-network-based methods often produce blurry results. Our method requires no human supervision, andany video can be used as training data.

0
0
0
Abstract

We address the problem of synthesizing new video frames in an existing video, either in-between existing frames (interpolation), or subsequent to them (extrapolation). This problem is challenging because video appearance and motion can be highly complex. Traditional optical-flow-based solutions often fail where flow estimation is challenging, while newer neural-network-based methods that hallucinate pixel values directly often produce blurry results. We combine the advantages of these two methods by training a deep network that learns to synthesize video frames by flowing pixel values from existing ones, which we call deep voxel flow. Our method requires no human supervision, and any video can be used as training data by dropping, and then learning to predict, existing frames. The technique is efficient, and can be applied at any video resolution. We demonstrate that our method produces results that both quantitatively and qualitatively improve upon the state-of-the-art.

Fri Sep 21 2018
Computer Vision
Temporal Interpolation as an Unsupervised Pretraining Task for Optical Flow Estimation
The difficulty of annotating training data is a major obstacle to using CNNs for low-level tasks in video. We train a CNN unsupervised for temporal interpolation. By fine-tuning on small amounts of ground truth flow, the network can learn to fill in homogeneous regions
0
0
0
Tue Dec 25 2018
Machine Learning
Motion Selective Prediction for Video Frame Synthesis
A dual network can use in a flexible way both dynamic and static convolutional motion kernels, to predict future frames. The construct of our model gives us the the means to efficiently analyze its functioning and interpret its output.
0
0
0
Fri Oct 16 2020
Computer Vision
Vid-ODE: Continuous-Time Video Generation with Neural Ordinary Differential Equation
Video generation models often operate under the assumption of fixed frame rates. Vid-ODE combines neural ODE with pixel-level video processing techniques. It is the first work successfully performing continuous-time video generation using real-world videos.
0
0
0
Wed Mar 03 2021
Computer Vision
Neural 3D Video Synthesis
We propose a novel approach for 3D video synthesis that is able to represent multi-view video recordings of a dynamic real-world scene. At the core of our approach is a novel time-conditioned neural radiance fields that represents scene dynamics using a set of compact latent codes.
0
0
0
Thu Dec 07 2017
Computer Vision
Multi-Scale Video Frame-Synthesis Network with Transitive Consistency Loss
Traditional approaches to interpolate/extrapolate frames in a video sequence require accurate pixel correspondences between images. We propose a unified network to parameterize the interest frame position and infer frames within the same framework. Our approach avoids expensive global optimization of optical flow methods.
0
0
0
Fri Jul 16 2021
Computer Vision
CCVS: Context-aware Controllable Video Synthesis
This presentation introduces a self-supervised learning approach. It conditions the synthesis process on contextual information for temporal continuity and ancillary information for fine control. The prediction model is doubly autoregressive, in the latent space of an autoencoder for forecasting, and in image
1
9
36