Segment object in video from text description using inter-frame and vision-language with IFIRVOS
Segment object in video from text description using inter-frame and vision-language with IFIRVOS
Referring Video Object Segmentation with Inter-Frame Interaction and Cross-Modal Correlation
arXiv paper abstract https://arxiv.org/abs/2307.00536
arXiv PDF paper https://arxiv.org/pdf/2307.00536.pdf
Referring video object segmentation (RVOS) aims to segment the target object in a video sequence described by a language expression.
Typical query-based methods process the video sequence in a frame-independent manner to reduce the high computational cost, which however affects the performance due to the lack of inter-frame interaction for temporal coherence modeling and spatio-temporal representation learning of the referred object.
Besides, they directly adopt the raw and high-level sentence feature as the language queries to decode the visual features, where the weak correlation between visual and linguistic features also increases the difficulty of decoding the target information and limits the performance of the model.
… proposes a novel RVOS framework, dubbed IFIRVOS, to address these issues … design … inter-frame interaction module in the Transformer decoder to … learn the spatio-temporal features of the referred object, so as to decode the object information in the video sequence more precisely and generate more accurate segmentation results.
… devise the vision-language interaction module before the multimodal Transformer to enhance the correlation between the visual and linguistic features, thus facilitating the process of decoding object information from visual features by language queries in Transformer decoder and improving the segmentation performance.
… validate the superiority of … IFIRVOS over state-of-the-art methods and the effectiveness of … proposed modules.
Stay up to date. Subscribe to my posts https://morrislee1234.wixsite.com/website/contact
Web site with my other posts by category https://morrislee1234.wixsite.com/website
LinkedIn https://www.linkedin.com/in/morris-lee-47877b7b
