TY - GEN

T1 - A Multimodal Approach for Semantic Patent Image Retrieval

AU - Pustu-Iren, Kader

AU - Bruns, Gerrit

AU - Ewerth, Ralph

N1 - Funding Information: We would like to sincerely thank the reviewers for their valuable and comprehensive comments. This work is financially supported by the Federal Ministry of Education and Research (BMBF, Bundesmin-isterium für Bildung und Forschung, project reference 01IO2004A).

PY - 2021

Y1 - 2021

N2 - Patent images such as technical drawings contain valuable information and are frequently used by experts to compare patents. However, current approaches to patent information retrieval are largely focused on textual information. Consequently, we review previous work on patent retrieval with a focus on illustrations in figures. In this paper, we report on work in progress for a novel approach for patent image retrieval that uses deep multimodal features. Scene text spotting and optical character recognition are employed to extract numerals from an image to subsequently identify references to corresponding sentences in the patent document. Furthermore, we use a neural state-of-the-art CLIP model to extract structural features from illustrations and additionally derive textual features from the related patent text using a sentence transformer model. To fuse our multimodal features for similarity search we apply re-ranking according to averaged or maximum scores. In our experiments, we compare the impact of different modalities on the task of similarity search for patent images. The experimental results suggest that patent image retrieval can be successfully performed using the proposed feature sets, while the best results are achieved when combining the features of both modalities.

AB - Patent images such as technical drawings contain valuable information and are frequently used by experts to compare patents. However, current approaches to patent information retrieval are largely focused on textual information. Consequently, we review previous work on patent retrieval with a focus on illustrations in figures. In this paper, we report on work in progress for a novel approach for patent image retrieval that uses deep multimodal features. Scene text spotting and optical character recognition are employed to extract numerals from an image to subsequently identify references to corresponding sentences in the patent document. Furthermore, we use a neural state-of-the-art CLIP model to extract structural features from illustrations and additionally derive textual features from the related patent text using a sentence transformer model. To fuse our multimodal features for similarity search we apply re-ranking according to averaged or maximum scores. In our experiments, we compare the impact of different modalities on the task of similarity search for patent images. The experimental results suggest that patent image retrieval can be successfully performed using the proposed feature sets, while the best results are achieved when combining the features of both modalities.

KW - Deep learning

KW - Mulitmodal feature representations

KW - Patent image similarity search

KW - Scene text spotting

UR - http://www.scopus.com/inward/record.url?scp=85111008683&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85111008683

T3 - CEUR Workshop Proceedings

SP - 45

EP - 49

BT - Patent Text Mining and Semantic Technologies 2021

T2 - 2nd Workshop on Patent Text Mining and Semantic Technologies, PatentSemTech 2021

Y2 - 15 July 2021

ER -