TY - JOUR

T1 - ADDRESSING CLASS IMBALANCE IN MULTI-CLASS IMAGE CLASSIFICATION BY MEANS OF AUXILIARY FEATURE SPACE RESTRICTIONS

AU - Dorozynski, M.

AU - Rottensteiner, F.

N1 - Funding Information: The research leading to these results is in the context of the ”SILKNOW. Silk heritage in the Knowledge Society: from punched cards to big data, deep learning and visual/tangible simulations” project, which has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 769504.

PY - 2022/5/30

Y1 - 2022/5/30

N2 - Learning from imbalanced class distributions generally leads to a classifier that is not able to distinguish classes with few training examples from the other classes. In the context of cultural heritage, addressing this problem becomes important when existing digital online collections consisting of images depicting artifacts and assigned semantic annotations shall be completed automatically; images with known annotations can be used to train a classifier that predicts missing information, where training data is often highly imbalanced. In the present paper, combining a classification loss with an auxiliary clustering loss is proposed to improve the classification performance particularly for underrepresented classes, where additionally different sampling strategies are applied. The proposed auxiliary loss aims to cluster feature vectors with respect to the semantic annotations as well as to visual properties of the images to be classified and thus, is supposed to help the classifier in distinguishing individual classes. We conduct an ablation study on a dataset consisting of images depicting silk fabrics coming along with annotations for different silk-related classification tasks. Experimental results show improvements of up to 10.5% in average F1-score and up to 20.8% in the F1-score averaged over the underrepresented classes in some classification tasks.

AB - Learning from imbalanced class distributions generally leads to a classifier that is not able to distinguish classes with few training examples from the other classes. In the context of cultural heritage, addressing this problem becomes important when existing digital online collections consisting of images depicting artifacts and assigned semantic annotations shall be completed automatically; images with known annotations can be used to train a classifier that predicts missing information, where training data is often highly imbalanced. In the present paper, combining a classification loss with an auxiliary clustering loss is proposed to improve the classification performance particularly for underrepresented classes, where additionally different sampling strategies are applied. The proposed auxiliary loss aims to cluster feature vectors with respect to the semantic annotations as well as to visual properties of the images to be classified and thus, is supposed to help the classifier in distinguishing individual classes. We conduct an ablation study on a dataset consisting of images depicting silk fabrics coming along with annotations for different silk-related classification tasks. Experimental results show improvements of up to 10.5% in average F1-score and up to 20.8% in the F1-score averaged over the underrepresented classes in some classification tasks.

KW - auxiliary clustering loss

KW - class imbalances

KW - convolutional neural networks

KW - Deep learning

KW - image classification

KW - silk heritage

UR - http://www.scopus.com/inward/record.url?scp=85132030391&partnerID=8YFLogxK

U2 - 10.5194/isprs-archives-XLIII-B2-2022-777-2022

DO - 10.5194/isprs-archives-XLIII-B2-2022-777-2022

M3 - Conference article

AN - SCOPUS:85132030391

VL - 43

SP - 777

EP - 785

JO - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives

JF - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives

SN - 1682-1750

IS - B2-2022

T2 - 2022 24th ISPRS Congress on Imaging Today, Foreseeing Tomorrow, Commission II

Y2 - 6 June 2022 through 11 June 2022

ER -