TY - JOUR

T1 - Multi-task deep learning with incomplete training samples for the image-based prediction of variables describing silk fabrics

AU - Dorozynski, M.

AU - Clermont, D.

AU - Rottensteiner, F.

N1 - Funding Information: The research leading to these results is in the frame of the ”SIL-KNOW. Silk heritage in the Knowledge Society: from punched cards to big data, deep learning and visual/tangible simulations” project, which has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 769504. We would also like to thank IMA-TEX for providing the annotated images for this research. Funding Information: The research leading to these results has received funding from the Research and Innovation Framework Programme (Marie Curie Actions) of the European Union's Horizon 2020 Framework Programme H2020-MSCA-IF-2016, project no. 747046.

PY - 2019/8/21

Y1 - 2019/8/21

N2 - This paper presents a method for the classification of images of silk fabrics with the aim to predict properties such as the placeand time of origin and the production technique. The proposed method was developed in the context of the EU project SILKNOW(http://silknow.eu/). In the context of classification, we address the problem of limited as well as not fully labelled data andinvestigate the connection between the distinct variables. A pre-trained Convolutional Neural Network (CNN) is used for thefeature extraction and a classification network realizing Multi-task learning (MTL) is trained based on these features. The trainingprocedure is adapted to enable the consideration of images that do not have a label for all tasks. Additionally, MTL with fullylabeled training data is investigated for the classification of silk fabrics. The impact of both MTL approaches is compared to singletask learning based on two different class structures. We achieve overall accuracies of 92-95% and average F1-scores of 88-90% inour best experiments.

AB - This paper presents a method for the classification of images of silk fabrics with the aim to predict properties such as the placeand time of origin and the production technique. The proposed method was developed in the context of the EU project SILKNOW(http://silknow.eu/). In the context of classification, we address the problem of limited as well as not fully labelled data andinvestigate the connection between the distinct variables. A pre-trained Convolutional Neural Network (CNN) is used for thefeature extraction and a classification network realizing Multi-task learning (MTL) is trained based on these features. The trainingprocedure is adapted to enable the consideration of images that do not have a label for all tasks. Additionally, MTL with fullylabeled training data is investigated for the classification of silk fabrics. The impact of both MTL approaches is compared to singletask learning based on two different class structures. We achieve overall accuracies of 92-95% and average F1-scores of 88-90% inour best experiments.

KW - Convolutional Neural Networks

KW - Cultural heritage

KW - Incomplete training samples

KW - Multi-task learning

KW - Silk fabrics

UR - http://www.scopus.com/inward/record.url?scp=85073786382&partnerID=8YFLogxK

U2 - 10.5194/isprs-annals-IV-2-W6-47-2019

DO - 10.5194/isprs-annals-IV-2-W6-47-2019

M3 - Conference article

AN - SCOPUS:85073786382

VL - 4

SP - 47

EP - 54

JO - ISPRS Journal of Photogrammetry and Remote Sensing

JF - ISPRS Journal of Photogrammetry and Remote Sensing

SN - 0924-2716

IS - 2/W6

T2 - 27th CIPA International Symposium on Documenting the Past for a Better Future

Y2 - 1 September 2019 through 5 September 2019

ER -