Hits:

Impact Factor:10.6

DOI number:10.1109/TIP.2023.3261743

Affiliation of Author(s):IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC

Journal:IEEE TRANSACTIONS ON IMAGE PROCESSING

Place of Publication:445 HOES LANE, PISCATAWAY, NJ 08855-4141

Key Words:VisualizationTask analysis;Semantics;Feature extraction;Training;Image segmentation;Image color analysis;Visual intention understanding;cross modality;hierarchical relation

Abstract:Visual intention understanding is the task of exploring the potential and underlying meaning expressed in images. Simply modeling the objects or backgrounds within the image content leads to unavoidable comprehension bias. To alleviate this problem, this paper proposes a Cross-modality Pyramid Alignment with Dynamic optimization (CPAD) to enhance the global understanding of visual intention with hierarchical modeling. The core idea is to exploit the hierarchical relationship between visual content and textual intention labels. For visual hierarchy, we formulate the visual intention understanding task as a hierarchical classification problem, capturing multiple granular features in different layers, which corresponds to hierarchical intention labels. For textual hierarchy, we directly extract the semantic representation from intention labels at different levels, which supplements the visual content modeling without extra manual annotations. Moreover, to further narrow the domain gap between different modalities, a cross-modality pyramid alignment module is designed to dynamically optimize the performance of visual intention understanding in a joint learning manner. Comprehensive experiments intuitively demonstrate the superiority of our proposed method, outperforming existing visual intention understanding methods.

Co-author:Shi Qinghongya,Du Bo

First Author:Ye Mang

Indexed by:Article

Correspondence Author:Su Kehua

Document Type:J

Volume:32

Page Number:2190-2201

ISSN No.:1057-7149

Translation or Not:no

Date of Publication:2023-05-05

Included Journals:SCI、EI