Abstract

The vast majority of modern consumer-grade cameras employ the Rolling Shutter (RS) mechanism, which often produces geometrically distorted images due to inter-row delay during imaging, that needs to be properly handled. While several models and solutions have been proposed for the motion estimation problem of the monocular RS camera, the corresponding motion estimation model and algorithm have not been developed for the standard RS stereo setup. This is primarily due to the temporal-dynamic nature of the RS camera, which results in scanline-varying camera poses. This paper fills in this gap by developing two tractable RS stereo models under constant velocity motion and constant acceleration motion respectively, which only need to simply scale the optical flow vector. We further propose two corresponding RS stereo motion estimation algorithms to accurately compute the relative pose of the RS stereo camera between two consecutive frames. As a result of utilizing these RS stereo models associated with the parallel visual rays, our proposed methods offer better disambiguation of the translation and rotation, hence leading to significantly improved performance compared with the Global Shutter (GS) stereo model. Furthermore, we develop an RS stereo image correction method to remove RS distortions and recover high-quality GS images by conducting motion compensation. Experimental results on both simulated data and real RS stereo images show the superiority of our proposed models and methods.

Contribution

• We propose two tractable RS stereo models under constant velocity motion and constant acceleration motion respectively, which can compensate the RS intra-frame motion effectively.
• We develop two efficient RS-stereo-aware motion estimation algorithms to obtain the subtle RS stereo full motion of each scanline.
• We present an RS-stereo-aware image correction algorithm to remove the geometric distortion in RS stereo images.
• Experiments on both simulated data and real RS stereo images show the effectiveness of our proposed algorithms.

Illustration of the standard RS stereo configuration

Given multiple pairs of point correspondences like $({p}_l,{p}_r)$ and their optical flow values, the relative motion $({t}, {\omega})$ of the stereo center $O$ between two consecutive frames can be estimated by our proposed algorithms. Note that each pair of point correspondences is generated by parallel projection rays (red) and therefore shares the same image coordinates.

Overview of our RS-stereo-aware motion estimation and image correction pipeline

From two consecutive RS stereo images, we establish RS stereo models to robustly recover the RS stereo motion and then achieve high-quality image correction (e.g., the red dotted pole in the foreground is repaired) combined with the RS depth map estimated by the GS stereo matching method, such as the SGM method.

Quantitative comparison

Quantitative error analysis based on the constant velocity model under various settings: (a) translation size, (b) rotation size, (c) optical flow noise, (d) readout time ratio, (e) baseline length, (f) number of correspondences. Note that the x-axes of (a) and (b) represent the multiples of translation and rotation, respectively.

Rolling shutter correction results on real data

(a) The original RS left images. (b-c) The corrected left images by the constant velocity model and constant acceleration model, respectively. (d) Our experimental platform. Our model and method are effective to remove the RS distortion in the real-world RS stereo images.