Multimodal Variational Auto-encoder based Audio-Visual Segmentation

IEEE International Conference on Computer Vision (ICCV 2023)

Yuxin Mao¹, Jing Zhang², Mochu Xiang¹ Yiran Zhong³ Yuchao Dai^1#

¹Northwestern Polytechnical University ²Australian National University ²Shanghai AI Laboratory
^# corresponding author
maoyuxin@mail.nwpu.edu.cn, daiyuchao@nwpu.edu.cn

arXiv (Comming soon)

Code

Abstract

We propose an Explicit Conditional Multimodal Variational Auto-Encoder (ECMVAE) for audio-visual segmentation (AVS), aiming to segment sound sources in the video sequence. Existing AVS methods focus on implicit feature fusion strategies, where models are trained to fit the discrete samples in the dataset. With a limited and less diverse dataset, the resulting performance is usually unsatisfactory. In contrast, we address this problem from an effective representation learning perspective, aiming to model the contribution of each modality explicitly. Specifically, we find that audio contains critical category information of the sound producers, and visual data provides candidate sound producer(s). Their shared information corresponds to the target sound producer(s) shown in the visual data. In this case, cross-modal shared representation learning is especially important for AVS. To achieve this, our ECMVAE factorizes the representations of each modality with a modality-shared representation and a modality-specific representation. An orthogonality constraint is applied between the shared and specific representations to maintain the exclusive attribute of the factorized latent code. Further, a mutual information maximization regularizer is introduced to achieve extensive exploration of each modality. Quantitative and qualitative evaluations on the AVSBench demonstrate the effectiveness of our approach, leading to a new state-of-the-art for AVS, with a 3.84 mIOU performance leap on the challenging MS3 subset for multiple sound source segmentation.

MMVAE model framework

Overview of the proposed ECMVAE for audio-visual segmentation. The feature extractors are used to extract backbone features for the two modalities. We also design three latent encoders to achieve latent space factorization and obtain both task-driven shared representation and modality-related specific representation, achieving explicit multimodal representation learning. The decoder is introduced to obtain the final segmentation maps, indicating the sound producers of the audio-visual data.