Hits:

DOI number:10.1109/TMM.2023.3295731

Affiliation of Author(s):Institute of Electrical and Electronics Engineers Inc.

Journal:IEEE Transactions on Multimedia

Abstract:Few-shot learning (FSL) is a challenging task that aims to train a classifier to recognize novel categories, where only a few annotated examples are available in each category. Recently, many FSL approaches have been proposed based on the meta-learning paradigm, which attempts to learn transferable knowledge from similar tasks by designing a meta-learner. However, most of these approaches only exploit the information from visual modality and do not utilize ones from additional modalities (<italic>e.g.</italic>, textual description). Since the labeled examples in FSL are limited, increasing the information on the examples is a probable solution to improve the classification performance. This motivates us to propose a novel meta-learning method, termed textual enhanced adaptive meta-fusion FSL (TAMF-FSL), which leverages both the visual information from the visual image and semantic information from language supervision. Specifically, TAMF-FSL exploits the semantic information of textual description to improve the visual-based models. We first employ a text encoder to learn the semantic features of each visual category, and then design a modality alignment module and meta-fusion module to align and fuse the visual and semantic features for final prediction. Extensive experiments show that the proposed method outperforms many recent or competitive FSL counterparts on two popular datasets.

Co-author:Zhan Yibing,Luo Yong,Hu Han,Du Bo

First Author:Han Mengya

Indexed by:Journal paper

Correspondence Author:Su Kehua

Page Number:1-11

ISSN No.:1520-9210

Translation or Not:no

Date of Publication:2023-03-01

Included Journals:EI