TY - GEN

T1 - On the Impact of Dataset Size

T2 - 2021 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, WI-IAT 2021

AU - Nguyen, Thi Huyen

AU - Nguyen, Hoang H.

AU - Ahmadi, Zahra

AU - Hoang, Tuan Anh

AU - Doan, Thanh Nam

N1 - Funding Information: This work is supported by the DFG Grant (NI-1760/1-1) Managed Forgetting, the European Union’s Horizon 2020 research and innovation program under grant agreement No. 832921 (project MIRROR), and No. 833635 (project ROXANNE).

PY - 2022/4/13

Y1 - 2022/4/13

N2 - The recent advent and evolution of deep learning models and pre-trained embedding techniques have created a breakthrough in supervised learning. Typically, we expect that adding more labeled data improves the predictive performance of supervised models. On the other hand, collecting more labeled data is not an easy task due to several difficulties, such as manual labor costs, data privacy, and computational constraint. Hence, a comprehensive study on the relation between training set size and the classification performance of different methods could be essentially useful in the selection of a learning model for a specific task. However, the literature lacks such a thorough and systematic study. In this paper, we concentrate on this relationship in the context of short, noisy texts from Twitter. We design a systematic mechanism to comprehensively observe the performance improvement of supervised learning models with the increase of data sizes on three well-known Twitter tasks: sentiment analysis, informativeness detection, and information relevance. Besides, we study how significantly better the recent deep learning models are compared to traditional machine learning approaches in the case of various data sizes. Our extensive experiments show (a) recent pre-trained models have overcome big data requirements, (b) a good choice of text representation has more impact than adding more data, and (c) adding more data is not always beneficial in supervised learning.

AB - The recent advent and evolution of deep learning models and pre-trained embedding techniques have created a breakthrough in supervised learning. Typically, we expect that adding more labeled data improves the predictive performance of supervised models. On the other hand, collecting more labeled data is not an easy task due to several difficulties, such as manual labor costs, data privacy, and computational constraint. Hence, a comprehensive study on the relation between training set size and the classification performance of different methods could be essentially useful in the selection of a learning model for a specific task. However, the literature lacks such a thorough and systematic study. In this paper, we concentrate on this relationship in the context of short, noisy texts from Twitter. We design a systematic mechanism to comprehensively observe the performance improvement of supervised learning models with the increase of data sizes on three well-known Twitter tasks: sentiment analysis, informativeness detection, and information relevance. Besides, we study how significantly better the recent deep learning models are compared to traditional machine learning approaches in the case of various data sizes. Our extensive experiments show (a) recent pre-trained models have overcome big data requirements, (b) a good choice of text representation has more impact than adding more data, and (c) adding more data is not always beneficial in supervised learning.

KW - dataset size

KW - empirical study

KW - extrapolation methods

KW - machine learning

KW - neural network

KW - Twitter classification

UR - http://www.scopus.com/inward/record.url?scp=85128592543&partnerID=8YFLogxK

U2 - 10.1145/3486622.3493960

DO - 10.1145/3486622.3493960

M3 - Conference contribution

AN - SCOPUS:85128592543

T3 - ACM International Conference Proceeding Series

SP - 210

EP - 217

BT - WI-IAT '21

PB - Association for Computing Machinery (ACM)

Y2 - 14 December 2021 through 17 December 2021

ER -