Continuous Parametric Optical Flow

Thirty-seventh Neural Information Processing Systems (NeurIPS 2023)

Jianqin Luo*, Zhexiong Wan*, Yuxin Mao, Bo Li, Yuchao Dai#

Northwestern Polytechnical University  
Shaanxi Key Laboratory of Information Acquisition and Processing
* Equal Contribution   # corresponding author
daiyuchao@nwpu.edu.cn

Paper

Code

Abstract

In this paper, we present continuous parametric optical flow, a parametric representation of dense and continuous motion over arbitrary time interval. In contrast to existing discrete-time representations (i.e.flow in between consecutive frames), this new representation transforms the frame-to-frame pixel correspondences to dense continuous flow. In particular, we present a temporal-parametric model that employs B-splines to fit point trajectories using a limited number of frames. To further improve the stability and robustness of the trajectories, we also add an encoder with a neural ordinary differential equation (ODE) to represent features associated with specific times. We also contribute a synthetic dataset and introduce two evaluation perspectives to measure the accuracy and robustness of continuous flow estimation. Benefiting from the combination of explicit parametric modeling and implicit feature optimization, our model focuses on motion continuity and outperforms than the flow-based and point-tracking approaches for fitting long-term and variable sequences.


Overall Pipeline

pipeline

Our model focuses on the representation and optimization of continuous motion, which explicitly describes continuous flow trajectory by cubic B-splines with learnable control points and implicitly aggregates spatio-temporal information by Neural ODE with ConvGRU.


Pixel-wise Continuous Flow Visualization

visualization

We evaluate our algorithm on synthetic dataset and real-world benchmark. The 24-frame visualization reveals that our continuous flow could handle with relatively complex rotation motion and keep long-term tracking to suppress drift.


Citation


 @InProceedings{Luo_CPFlow_NeurIPS_2023,
  author    = {Luo, Jianqin and Wan, Zhexiong and Mao, Yuxin and Zhang, Jing and Dai, Yuchao},
  title     = {Continuous Parametric Optical Flow},
  booktitle = {Thirty-seventh Conference on Neural Information Processing Systems},
  year      = {2023},
}

Acknowledgments

This research was supported in part by the National Natural Science Foundation of China (62271410, 62001394), the Fundamental Research Funds for the Central Universities, and the Innovation Foundation for Doctor Dissertation of Northwestern Polytechnical University (CX2023013).

Thanks the ACs and the reviewers for their comments, which is very helpful to improve our paper.

Thanks for the following helpful open source projects: Vid-ODE, RAFT, Kubric, TAP-Vid-benchmark, PIPs.