Hits:

Impact Factor:10.6

DOI number:10.1109/TIP.2023.3299791

Affiliation of Author(s):IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC

Journal:IEEE TRANSACTIONS ON IMAGE PROCESSING

Place of Publication:445 HOES LANE, PISCATAWAY, NJ 08855-4141

Key Words:Cross Relation;Image Retrieval;Transformer

Abstract:Composing Text and Image to Image Retrieval (CTI-IR) aims at finding the target image, which matches the query image visually along with the query text semantically. However, existing works ignore the fact that the reference text usually serves multiple functions, e.g., modification and auxiliary. To address this issue, we put forth a unified solution, namely Hierarchical Aggregation Transformer incorporated with Cross Relation Network (CRN). CRN unifies modification and relevance manner in a single framework. This configuration shows broader applicability, enabling us to model both modification and auxiliary text or their combination in triplet relationships simultaneously. Specifically, CRN includes: 1) Cross Relation Network comprehensively captures the relationships of various composed retrieval scenarios caused by two different query text types, allowing a unified retrieval model to designate adaptive combination strategies for flexible applicability; 2) Hierarchical Aggregation Transformer aggregates top-down features with Multi-layer Perceptron (MLP) to overcome the limitations of edge information loss in a window-based multi-stage Transformer. Extensive experiments demonstrate the superiority of the proposed CRN over all three fashion-domain datasets. Code is available at github.com/yan9qu/crn.

Co-author:Ye Mang,Cai Zhaohui,Du Bo

First Author:Yang Qu

Indexed by:Article

Correspondence Author:Su Kehua

Document Type:J

Volume:32

Page Number:4543-4554

ISSN No.:1057-7149

Translation or Not:no

Date of Publication:2023-08-24

Included Journals:SCI、EI