A Benchmark of Activation Functions in Extreme Learning Machine for High-Dimensional Low-Sample-Size Microarray Classification

Inggih Permana; Shir Li Wang; Febi Nur Salisah; Sanusi; Febi Yanto

doi:10.31572/inotera.Vol10.Iss2.2025.ID593

Inggih Permana Faculty of Computing and Meta-Technology, Universiti Pendidikan Sultan Idris, Malaysia
Shir Li Wang Faculty of Computing and Meta-Technology, Universiti Pendidikan Sultan Idris, Malaysia
Febi Nur Salisah Faculty of Science and Technology, Universitas Islam Negeri Sultan Syarif Kasim Riau
Sanusi Teknologi Informasi, Fakultas Teknik, Universitas Teuku Umar
Febi Yanto Faculty of Science and Technology, Universitas Islam Negeri Sultan Syarif Kasim Riau

DOI: https://doi.org/10.31572/inotera.Vol10.Iss2.2025.ID593

Keywords: Activation function, Extreme learning machine, High-dimensional, Low-sample-size, Microarray

Abstract

Microarray data classification presents significant challenges in machine learning due to its high-dimensionality, low-sample-size (HDLSS) characteristics and class imbalance. These conditions often lead to overfitting and result in models with low performance stability. Extreme Learning Machine Learning (ELM) is an efficient method, but its performance is highly dependent on the choice of activation function that determines the model's non-linearity. Therefore, this study aims to conduct a comparative benchmarking of eight activation functions (sigmoid, tanh, relu, hardlimit, identity, swish, cosine, and softsign) within the ELM framework on 11 cancer-related microarray datasets. Testing was conducted through 30 independent runs to ensure statistical robustness, and performance was evaluated using accuracy, F1-score, precision, and recall metrics. Experimental results show that the choice of activation function has a significant impact on microarray data classification performance. The sigmoid function consistently provides superior and most stable results across various datasets, achieving a global average accuracy above 70% for all evaluation metrics. The advantage of the sigmoid lies not only in its high average performance but also in its stability, as evidenced by its low standard deviation. These findings provide strong practical guidance, recommending the use of the sigmoid activation function for robust ELM implementations in microarray data classification.

Downloads

Download data is not yet available.

References

Alimi, F., Khan, A., Ali, H., Bouzidi, M., Alshammari, A. O., & Ahmad, B. (2025). M-estimation activation functions for high-performance extreme learning machine ensemble classification. Scientific Reports, 15(1), 32154. https://doi.org/10.1038/s41598-025-16798-5.

Begum, S., Sarkar, R., Chakraborty, D., Sen, S., & Maulik, U. (2021). Application of active learning in DNA microarray data for cancerous gene identification. Expert Systems with Applications, 177, 114914. https://doi.org/10.1016/j.eswa.2021.114914.

Cao, J., Zhang, K., Yong, H., Lai, X., Chen, B., & Lin, Z. (2018). Extreme learning machine with affine transformation inputs in an activation function. IEEE transactions on neural networks and learning systems, 30(7), 2093-2107. https://doi.org/10.1109/TNNLS.2018.2877468.

Das, D., Nayak, D. R., Dash, R., & Majhi, B. (2019). An empirical evaluation of extreme learning machine: application to handwritten character recognition. Multimedia Tools and Applications, 78(14), 19495-19523.

Deng, X., Li, M., Deng, S., & Wang, L. (2022). Hybrid gene selection approach using XGBoost and multi-objective genetic algorithm for cancer classification. Medical & Biological Engineering & Computing, 60(3), 663-681. https://doi.org/10.1007/s11517-021-02476-x.

Dokeroglu, T., & Sevinc, E. (2019). Evolutionary parallel extreme learning machines for the data classification problem. Computers & Industrial Engineering, 130, 237-249. https://doi.org/10.1016/j.cie.2019.02.024.

Hastie, T., Tibshirani, R., Friedman, J., & Franklin, J. (2005). The elements of statistical learning: data mining, inference and prediction. The Mathematical Intelligencer, 27(2), 83-85. https://doi.org/10.1007/978-0-387-84858-7.

Hira, S., & Bai, A. (2022). A novel map reduced based parallel feature selection and extreme learning for micro array cancer data classification. Wireless Personal Communications, 123(2), 1483-1505. https://doi.org/10.1007/s11277-021-09196-3.

Huang, G. B., Zhu, Q. Y., & Siew, C. K. (2004, July). Extreme learning machine: a new learning scheme of feedforward neural networks. In 2004 IEEE international joint conference on neural networks (IEEE Cat. No. 04CH37541) (Vol. 2, pp. 985-990). Ieee. https://doi.org/10.1109/IJCNN.2004.1380068.

Huang, G. B., Zhu, Q. Y., & Siew, C. K. (2006). Extreme learning machine: theory and applications. Neurocomputing, 70(1-3), 489-501. https://doi.org/10.1016/j.neucom.2005.12.126.

Ismail, N., Othman, Z. A., & Samsudin, N. A. (2019). Regularization activation function for extreme learning machine. International Journal of Advanced Computer Science and Applications, 10(3), 240-247. https://dx.doi.org/10.14569/IJACSA.2019.0100331.

Kumar, C. A., & Ramakrishnan, S. (2014, December). Binary classification of cancer microarray gene expression data using extreme learning machines. In 2014 IEEE International Conference on Computational Intelligence and Computing Research (pp. 1-4). IEEE. https://doi.org/10.1109/ICCIC.2014.7238297.

Li, B., Li, Y., & Rong, X. (2013). The extreme learning machine learning algorithm with tunable activation function. Neural Computing and Applications, 22(3), 531-539. https://doi.org/10.1007/s00521-012-0858-9.

Liu, S., Feng, L., Xiao, Y., & Wang, H. (2014). Robust activation function and its application: Semi-supervised kernel extreme learning method. Neurocomputing, 144, 318-328. https://doi.org/10.1016/j.neucom.2014.04.041.

Ramachandran, P., Zoph, B., & Le, Q. V. (2017). Swish: a self-gated activation function. arXiv preprint arXiv:1710.05941, 7(1), 5.

Ratnawati, D. E., Marjono, Widodo, & Anam, S. (2020, September). Comparison of activation function on extreme learning machine (ELM) performance for classifying the active compound. In AIP Conference Proceedings (Vol. 2264, No. 1, p. 140001). AIP Publishing LLC. https://doi.org/10.1063/5.0023872.

Salisah, F. N., Permana, I., & Wang, S. L. (2025). Evaluating the Impact of Data Balancing Techniques on the k-Nearest Neighbors Algorithm for Microarray Data Classification. Jurnal Inotera, 10(2), 261-271. https://doi.org/10.31572/inotera.Vol10.Iss2.2025.ID497.

Sucharita, S., Sahu, B., Swarnkar, T., & Meher, S. K. (2024). Classification of cancer microarray data using a two-step feature selection framework with moth-flame optimization and extreme learning machine. Multimedia Tools and Applications, 83(7), 21319-21346. https://doi.org/10.1007/s11042-023-16353-2.

Tripathi, D., Edla, D. R., Kuppili, V., & Bablani, A. (2020). Evolutionary extreme learning machine with novel activation function for credit scoring. Engineering Applications of Artificial Intelligence, 96, 103980. https://doi.org/10.1016/j.engappai.2020.103980.

Vogiatzis, D., & Tsapatsoulis, N. (2008). Active learning for microarray data.International Journal of Approximate Reasoning, 47(1), 85-96. https://doi.org/10.1016/j.ijar.2007.03.009.

Zhang, R., Huang, G. B., Sundararajan, N., & Saratchandran, P. (2007). Multicategory classification using an extreme learning machine for microarray gene expression cancer diagnosis. IEEE/ACM transactions on Computational Biology and Bioinformatics, 4(3), 485-495. https://doi.org/10.1109/tcbb.2007.1012.

Zhu, Z., Ong, Y. S., & Dash, M. (2007). Markov blanket-embedded genetic algorithm for gene selection. Pattern Recognition, 40(11), 3236-3248. https://doi.org/10.1016/j.patcog.2007.02.007.