TY - GEN
T1 - Adversarial Autoencoder and Multi-Task Semi-Supervised Learning for Multi-stage Process
AU - Mendes, Andre
AU - Togelius, Julian
AU - dos Santos Coelho, Leandro
N1 - Publisher Copyright:
© Springer Nature Switzerland AG 2020.
PY - 2020
Y1 - 2020
N2 - In selection processes, decisions follow a sequence of stages. Early stages have more applicants and general information, while later stages have fewer applicants but specific data. This is represented by a dual funnel structure, in which the sample size decreases from one stage to the other while the information increases. Training classifiers for this case is challenging. In the early stages, the information may not contain distinct patterns to learn, causing underfitting. In later stages, applicants have been filtered out and the small sample can cause overfitting. We redesign the multi-stage problem to address both cases by combining adversarial autoencoders (AAE) and multi-task semi-supervised learning (MTSSL) to train an end-to-end neural network for all stages together. The AAE learns the representation of the data and performs data imputation in missing values. The generated dataset is fed to an MTSSL mechanism that trains all stages together, encouraging related tasks to contribute to each other using a temporal regularization structure. Using real-world data, we show that our approach outperforms other state-of-the-art methods with a gain of 4x over the standard case and a 12% improvement over the second-best method.
AB - In selection processes, decisions follow a sequence of stages. Early stages have more applicants and general information, while later stages have fewer applicants but specific data. This is represented by a dual funnel structure, in which the sample size decreases from one stage to the other while the information increases. Training classifiers for this case is challenging. In the early stages, the information may not contain distinct patterns to learn, causing underfitting. In later stages, applicants have been filtered out and the small sample can cause overfitting. We redesign the multi-stage problem to address both cases by combining adversarial autoencoders (AAE) and multi-task semi-supervised learning (MTSSL) to train an end-to-end neural network for all stages together. The AAE learns the representation of the data and performs data imputation in missing values. The generated dataset is fed to an MTSSL mechanism that trains all stages together, encouraging related tasks to contribute to each other using a temporal regularization structure. Using real-world data, we show that our approach outperforms other state-of-the-art methods with a gain of 4x over the standard case and a 12% improvement over the second-best method.
KW - Autoencoder
KW - Multi-stage
KW - Multi-task
UR - http://www.scopus.com/inward/record.url?scp=85085727629&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85085727629&partnerID=8YFLogxK
U2 - 10.1007/978-3-030-47436-2_1
DO - 10.1007/978-3-030-47436-2_1
M3 - Conference contribution
AN - SCOPUS:85085727629
SN - 9783030474355
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 3
EP - 16
BT - Advances in Knowledge Discovery and Data Mining - 24th Pacific-Asia Conference, PAKDD 2020, Proceedings
A2 - Lauw, Hady W.
A2 - Lim, Ee-Peng
A2 - Wong, Raymond Chi-Wing
A2 - Ntoulas, Alexandros
A2 - Ng, See-Kiong
A2 - Pan, Sinno Jialin
PB - Springer
T2 - 24th Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2020
Y2 - 11 May 2020 through 14 May 2020
ER -