Leveraging extensive feature modeling for facial emotion recognition

  • Milica Tufegdzic orcid

    Academy of Professional Studies Sumadija, Kragujevac 34000, Serbia
  • Nevena Tufegdzic orcid

    SaTCIP Publisher Ltd, Vrnjacka Banja 36210, Serbia
  • Marija Mojsilovic orcid

    Academy of Professional Studies Sumadija, Kragujevac 34000, Serbia
Article ID: 4397
DOI: https://doi.org/10.59400/cai4397
Keywords: machine learning, action units, emotion classification, image processing, feature extraction, FACS (facial action coding system), explainable AI

Abstract

Facial emotion recognition (FER) is an important area of affective computing with applications in human–computer interaction, healthcare, education, and intelligent systems. Although recent FER research is largely dominated by deep learning and transformer-based approaches, handcrafted feature modeling remains attractive due to its interpretability and lower computational requirements. This study proposes an Action Unit–based machine learning (AU-ML) framework for recognizing basic emotions from lateral facial expressions using the Karolinska Directed Emotional Faces (KDEF) dataset. Facial Action Units (AUs) were extracted, and manual feature selection was performed to retain only AU intensity and presence information relevant to emotion recognition. This process significantly reduced the original feature vector, improving computational efficiency while preserving classification performance. To compensate for the reduced dataset size after extracting lateral images, data augmentation techniques, including horizontal flipping, shifting, scaling, rotation, and brightness and contrast adjustments, were applied prior to AU extraction. Several machine learning algorithms were evaluated, including K-Nearest Neighbors, Support Vector Classifier, Decision Tree, Naïve Bayes, Random Forest, AdaBoost, Bagging, Voting, and Stacking Classifiers, CatBoost, and Extreme Gradient Boosting. The results demonstrated that ensemble methods generally outperformed simpler classifiers, with CatBoost achieving the best classification performance. The findings indicate that extensive feature modeling remains a reliable approach to emotion recognition, and that AU-based representations provide an interpretable and computationally efficient alternative to deep learning approaches.

Published
2025-12-19
How to Cite
Tufegdzic, M., Tufegdzic, N., & Mojsilovic, M. (2025). Leveraging extensive feature modeling for facial emotion recognition. Computing and Artificial Intelligence, 3(4). https://doi.org/10.59400/cai4397
Issue
Vol. 3 No. 4 (2025)
Section
Article

Copyright (c) 2025 Milica Tufegdzic, Nevena Tufegdzic, Marija Mojsilovic

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

References

[1]Vaswani A, Shazeer N, Parmar N, et al. Attention Is All You Need. arXiv preprint. 2023. doi: 10.48550/arXiv.1706.03762

[2]Deepika, Vashisth S, Sharma P. Effectiveness of Focal Loss for Traffic Sign Detection Using Deep Neural Networks. International Journal of convergence in healthcare. 2023; 3(2). doi: 10.55487/9rtyzm18

[3]Xu Y, Lin YS, Zhou X, et al. Utilizing emotion recognition technology to enhance user experience in real-time. Computing and Artificial Intelligence. 2024; 2(1): 1388. doi: 10.59400/cai.v2i1.1388

[4]Dhuheir M, Albaseer A, Baccour E, et al. Emotion Recognition for Healthcare Surveillance Systems Using Neural Networks: A Survey. arXiv preprint. 2021. doi: 10.48550/arXiv.2107.05989

[5]Lima ES, Perico CP, Nichio BTL, et al. Emotional recognition technologies applied to health: Review and challenges. Brazilian Journal of Psychiatry. 2025; 47: e20243963. doi: 10.47626/1516-4446-2024-3963

[6]Wu Y, Mi Q, Gao T. A Comprehensive Review of Multimodal Emotion Recognition: Techniques, Challenges, and Future Directions. Biomimetics. 2025; 10(7): 418. doi: 10.3390/biomimetics10070418

[7]Kalateh S, Estrada-Jimenez LA, Nikghadam-Hojjati S, et al. A Systematic Review on Multimodal Emotion Recognition: Building Blocks, Current State, Applications, and Challenges. IEEE Access. 2024; 12: 103976–104019. doi: 10.1109/ACCESS.2024.3430850

[8]Kumar MJD, Rao MS, Narendra KC. Multimodal Emotion Recognition: A Comprehensive Survey of Datasets, Methods, and Applications. IEEE Access. 2025; 13: 201067–201097. doi: 10.1109/ACCESS.2025.3636186

[9]Gautam C, Seeja KR. Facial emotion recognition using Handcrafted features and CNN. Procedia Computer Science. 2023; 218: 1295–1303. doi: 10.1016/j.procs.2023.01.108

[10]Nithin DK, Sivakumar PB. Generic Feature Learning in Computer Vision. Procedia Computer Science. 2015; 58: 202–209. doi: 10.1016/j.procs.2015.08.054

[11]Zhalgas A, Amirgaliyev B, Sovet A. Robust Face Recognition Under Challenging Conditions: A Comprehensive Review of Deep Learning Methods and Challenges. Applied Sciences. 2025; 15(17): 9390. doi: 10.3390/app15179390

[12]Viola P, Jones M. Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition; 8–14 December 2001; Kauai, HI, USA. doi: 10.1109/CVPR.2001.990517

[13]Dalal N, Triggs B. Histograms of Oriented Gradients for Human Detection. In: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05); 20–25 June 2005; San Diego, CA, USA. pp. 886–893. doi: 10.1109/CVPR.2005.177

[14]Ali A, Nasir JA, Ahmed MM, et al. Machine Learning Based Statistical Analysis of Emotion Recognition using Facial Expression. RADS Journal of Biological Research & Applied Sciences. 2020; 11(1): 39–46. doi: 10.37962/jbas.v11i1.262

[15]Ahonen T, Hadid A, Pietikäinen M. Face Recognition with Local Binary Patterns. In: Computer Vision - ECCV 2004, Lecture Notes in Computer Science. Springer; 2004. pp. 469–481. doi: 10.1007/978-3-540-24670-1_36

[16]Asad M, Gilani SO, Jamil M. Emotion Detection through Facial Feature Recognition. International Journal of Multimedia and Ubiquitous Engineering. 2017; 12(11): 21–30. doi: 10.14257/ijmue.2017.12.11.03

[17]Deepika, Vashisth S, Saurav S. Histogram of Oriented Gradients Based Reduced Feature for Traffic Sign Recognition. In: Proceedings of the 2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI); 19–22 September 2018; Bangalore, India. pp. 2206–2212. doi: 10.1109/ICACCI.2018.8554624

[18]Weng CH, Lai SH. Online facial expression recognition based on combining texture and geometric information. In: Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP); 27–30 October 2014; Paris, France. pp. 5976–5980. doi: 10.1109/ICIP.2014.7026206

[19]Tico M, Haverinen T, Kuosmanen P. A Method of Color Histogram Creation for Image Retrieval. In: Proceedings of the NORSIG 2000 Nordic Signal Processing Symposium; 13–15 June 2000; Kolmården, Sweden. pp. 157–160. Available online: https://www.researchgate.net/publication/244449350_A_method_of_color_histogram_creation_for_image_retrieval

[20]Lindeberg T. Orientation selectivity properties for the affine Gaussian derivative and the affine Gabor models for visual receptive fields. Journal of Computational Neuroscience. 2025; 53(1): 61–98. doi: 10.1007/s10827-024-00888-w

[21]Sagonas C, Tzimiropoulos G, Zafeiriou S, et al. 300 Faces in-the-Wild Challenge: The First Facial Landmark Localization Challenge. In: Proceedings of the 2013 IEEE International Conference on Computer Vision Workshops; 2–8 December 2013; Sydney, Australia. pp. 397–403. doi: 10.1109/ICCVW.2013.59

[22]Liu C, Hirota K, Ma J, et al. Facial Expression Recognition Using Hybrid Features of Pixel and Geometry. IEEE Access. 2021; 9: 18876–18889. doi: 10.1109/ACCESS.2021.3054332

[23]Haghpanah MA, Saeedizade E, Masouleh MT, et al. Real-Time Facial Expression Recognition using Facial Landmarks and Neural Networks. In: Proceedings of the 2022 International Conference on Machine Vision and Image Processing (MVIP); 23–24 February 2022; Ahvaz, Iran. pp. 1–7. doi: 10.1109/MVIP53647.2022.9738754

[24]Kim KI, Kim J, Jung K. Recognition of facial images using support vector machines. In: Proceedings of the 11th IEEE Signal Processing Workshop on Statistical Signal Processing; 8 August 2001; Singapore. pp. 468–471. doi: 10.1109/SSP.2001.955324

[25]Anggo M, Arapu L. Face Recognition Using Fisherface Method. Journal of Physics: Conference Series. 2018; 1028: 012119. doi: 10.1088/1742-6596/1028/1/012119

[26]Kshirsagar VP, Baviskar MR, Gaikwad ME. Face recognition using Eigenfaces. In: Proceedings of the 2011 3rd International Conference on Computer Research and Development; 11–13 March 2011; Shanghai, China. pp. 302–306. doi: 10.1109/ICCRD.2011.5764137

[27]Wagner U, Dürrschmid K, Pauser S. Emotion Recognition—Recent Advances and Applications in Consumer Behavior and Food Sciences with an Emphasis on Facial Expressions. In: Emotion Recognition—Recent Advances, New Perspectives and Applications. IntechOpen; 2023. doi: 10.5772/intechopen.110581

[28]Shahzad HM, Bhatti SM, Jaffar A, et al. Hybrid Facial Emotion Recognition Using CNN-Based Features. Applied Sciences. 2023; 13(9): 5572. doi: 10.3390/app13095572

[29]Kumar R, Corvisieri G, Fici TF, et al. Transfer Learning for Facial Expression Recognition. Information. 2025; 16(4): 320. doi: 10.3390/info16040320

[30]Zhang K, Zhang Z, Li Z, et al. Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks. IEEE Signal Processing Letters. 2016; 23(10): 1499–1503. doi: 10.1109/LSP.2016.2603342

[31]Garg D, Goel P, Pandya S, et al. A Deep Learning Approach for Face Detection using YOLO. In: Proceedings of the 2018 IEEE Punecon; 30 November–2 December 2018; Pune, India. pp. 1–4. doi: 10.1109/PUNECON.2018.8745376

[32]Deng J, Guo J, Zhou Y, et al. RetinaFace: Single-stage Dense Face Localisation in the Wild. arXiv preprint. 2019. doi: 10.48550/arXiv.1905.00641

[33]Barrett LF, Lewis M, Haviland-Jones JM. Handbook of Emotions, 4th ed. Guilford Press; 2016.

[34]Paul Ekman Group. Facial Action Coding System. Paul Ekman Group; n.d. Available online: https://www.paulekman.com/facial-action-coding-system/

[35]KDEF & AKDEF. About KDEF. KDEF & AKDEF; n.d. Available online: https://kdef.se/home/aboutKDEF

[36]Baltrusaitis T, Robinson P, Morency LP. OpenFace: An open source facial behavior analysis toolkit. In: Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV); 7–10 March 2016; Lake Placid, NY, USA. pp. 1–10. doi: 10.1109/WACV.2016.7477553

[37]Baltrusaitis T, Robinson P, Morency LP. Constrained Local Neural Fields for Robust Facial Landmark Detection in the Wild. In: Proceedings of the 2013 IEEE International Conference on Computer Vision Workshops; 2–8 December 2013; Sydney, Australia. pp. 354–361. doi: 10.1109/ICCVW.2013.54

[38]CatBoost. CatBoost Is a High-Performance Open Source Library for Gradient Boosting on Decision Trees. CatBoost; n.d. Available online: https://catboost.ai

[39]Aly M, Alotaibi NS. A comprehensive deep learning framework for real time emotion detection in online learning using hybrid models. Scientific Reports. 2025; 15(1): 42012. doi: 10.1038/s41598-025-26381-7

[40]Abdeldayem M, Badawy W, F. A. Hamed H, et al. Improving Facial Expression Recognition in real-world Environments. Statistics, Optimization & Information Computing. 2025; 14(6): 3546–3564. doi: 10.19139/soic-2310-5070-3171

[41]Khan T, Yasir M, Choi C. Attention-enhanced optimized deep ensemble network for effective facial emotion recognition. Alexandria Engineering Journal. 2025; 119: 111–123. doi: 10.1016/j.aej.2025.01.078

[42]Arslanoğlu MC, Acar H, Albayrak A. Face Expression Recognition via transformer-based classification models. Balkan Journal of Electrical and Computer Engineering. 2024; 12(3): 214–223. doi: 10.17694/bajece.1486140