TY - GEN

T1 - Unsupervised Training Data Generation of Handwritten Formulas using Generative Adversarial Networks with Self-Attention

AU - Springstein, Matthias

AU - Müller-Budack, Eric

AU - Ewerth, Ralph

PY - 2021/8/27

Y1 - 2021/8/27

N2 - The recognition of handwritten mathematical expressions in images and video frames is a difficult and unsolved problem yet. Deep convectional neural networks are basically a promising approach, but typically require a large amount of labeled training data. However, such a large training dataset does not exist for the task of handwritten formula recognition. In this paper, we introduce a system that creates a large set of synthesized training examples of mathematical expressions which are derived from LaTeX documents. For this purpose, we propose a novel attention-based generative adversarial network to translate rendered equations to handwritten formulas. The datasets generated by this approach contain hundreds of thousands of formulas, making it ideal for pretraining or the design of more complex models. We evaluate our synthesized dataset and the recognition approach on the CROHME 2014 benchmark dataset. Experimental results demonstrate the feasibility of the approach.

AB - The recognition of handwritten mathematical expressions in images and video frames is a difficult and unsolved problem yet. Deep convectional neural networks are basically a promising approach, but typically require a large amount of labeled training data. However, such a large training dataset does not exist for the task of handwritten formula recognition. In this paper, we introduce a system that creates a large set of synthesized training examples of mathematical expressions which are derived from LaTeX documents. For this purpose, we propose a novel attention-based generative adversarial network to translate rendered equations to handwritten formulas. The datasets generated by this approach contain hundreds of thousands of formulas, making it ideal for pretraining or the design of more complex models. We evaluate our synthesized dataset and the recognition approach on the CROHME 2014 benchmark dataset. Experimental results demonstrate the feasibility of the approach.

KW - datasets

KW - formula recognition

KW - generative adversarial network

UR - http://www.scopus.com/inward/record.url?scp=85114808291&partnerID=8YFLogxK

U2 - https://doi.org/10.48550/arXiv.2106.09432

DO - https://doi.org/10.48550/arXiv.2106.09432

M3 - Conference contribution

AN - SCOPUS:85114808291

T3 - MMPT 2021 - Proceedings of the 2021 Workshop on Multi-Modal Pre-Training for Multimedia Understanding

SP - 46

EP - 54

BT - MMPT 2021

T2 - 1st International Joint Workshop on Multi-Modal Pre-Training for Multimedia Understanding, MMPT 2021

Y2 - 21 August 2021

ER -