Customer’s Spontaneous Facial Expression Recognition



In the field of consumer science, customer facial expression is often categorized either as negative or positive. Customer who portrays negative emotion to a specific product mostly means they reject the product while a customer with positive emotion is more likely to purchase the product. To observe customer emotion, many researchers have studied different perspectives and methodologies to obtain high accuracy results. This paper aims to recognize customer spontaneous expressions while the customer observed certain products. We have developed a customer service system using a Conventional Neural Network which is trained to detect three types of facial expression, i.e. happy, sad, and neutral. Facial features are extracted together with its Histogram of Gradient and sliding window.  The results are then compared with the existing works and it shows an achievement of 82.9% success rate on average.


Facial Expressions; Face Detection; Customer’s Emotion; CNN


Zhi, R., Hu, X., Wang, C., & Liu, S. (2020). Development of a Direct Mapping Model between Hedonic Rating and Facial Responses by Dynamic Facial Expression Representation. Food Research International, 137, 109411.

Samant, S. S., Chapko, M. J., Seo, H. (2017). Predicting consumer liking and preference based on emotional responses and sensory perception: A study with basic taste solutions. Food Research International, 100: 325-334

Moore, Sarah G. (2015), “Attitude Predictability and Helpfulness in Online Reviews: The Role of Explained Actions and Reactions,” Journal of Consumer Research, 42 (1), 30–44.

Rocklage, M. D., & Fazio, R. H. (2020). The Enhancing Versus Backfiring Effects of Positive Emotion in Consumer Reviews. Journal of Marketing Research, 57(2), 332-352.

Hussain, N., Ujir, H., Hipiny, I. and Minoi, J.L. 2019. 3D Facial Action Units Recognition for Emotional Expression, International Journal of Recent Technology and Engineering (IJRTE), Volume-8 Issue-28, pp. 1317-1323.

Harrison-Walker, L. J. (2019). The effect of consumer emotions on outcome behaviors following service failure. Journal of Services Marketing, 33(3), 285–302.

Ujir, H. and Spann, M. 2014. Surface Normals with Modular Approach and Weighted Voting Scheme in 3D Facial Expression Classification. International Journal of Computer and Information Technology (ISSN: 2279 – 0764) Volume 03 – Issue 05, September 2014, pp. 909-918.

Tarnowski, P., Kolodziej, M., Majkowski, A., & Rak, R. J. (2017, June). Emotion recognition using facial expressions. In ICCS, pp. 1175-1184.

Yuan, J., Mcdonough, S., You, Q., & Luo, J. (2013, August). Sentribute: image sentiment analysis from a mid-level perspective. In Proceedings of the Second International Workshop on Issues of Sentiment Discovery and Opinion Mining (pp. 1-8).

Yolcu, G., Oztel, I., Kazan, S., Oz, C., & Bunyak, F. (2020). Deep learning-based face analysis system for monitoring customer interest. Journal of Ambient Intelligence and Humanized Computing, 11(1), 237-248.

Zhang, H., Jolfaei, A., & Alazab, M. (2019). A face emotion recognition method using convolutional neural network and image edge computing. IEEE Access, 7, 159081-159089.

Desmet P. M. A. 2012. Faces of Product Pleasure: 25 Positive Emotions in Human-Product Interactions. International Journal of Design, 6, 2 (Dec. 2012) 2012.

Balaji, M. S., Roy, S. K., & Quazi, A. (2017). Customers’ emotion regulation strategies in service failure encounters. European Journal of Marketing.

Gavrilov, A. D., Jordache, A., Vasdani, M., & Deng, J. (2018). Preventing model overfitting and underfitting in convolutional neural networks. International Journal of Software Science and Computational Intelligence (IJSSCI), 10(4), 19-28.

Mohan, K., Seal, A., Krejcar, O., & Yazidi, A. FER-net: facial expression recognition using deep neural net. Neural Computing and Applications, 1-12.

Vijayakumar, T. (2019). Comparative study of capsule neural network in various applications. Journal of Artificial Intelligence, 1(01), 19-27

Sampaio, C. H., Ladeira, W. J., & Santini, F. D. O. (2017). Apps for Mobile Banking and Customer Satisfaction: A Cross-Cultural Study. International Journal of Bank Marketing, 35(7), 1133–1153.

An, K., Hui, M. K., & Leung, K. (2001). Who Should Be Responsible? Effects of Voice and Compensation on Responsibility Attribution, Perceived Justice, and Post‐Complaint Behaviors Across Cultures. International Journal of Conflict Management, 12(4), 350–364.

Lu, S., Xiao, L., & Ding, M. (2016). A Video-Based Automated Recommender (VAR) System for Garments. Marketing Science, 35(3), 484-510.

Deshpande, N. T., & Ravishankar, S. (2017). Face Detection and Recognition using Viola-Jones algorithm and Fusion of PCA and ANN. Advances in Computational Sciences and Technology,10(5), 1173-1189.

Kumar, S., Singh, S., & Kumar, J. (2019). Multiple Face Detection using Hybrid Features with SVM Classifier. In Data and Communication Networks (pp. 253-265). Springer, Singapore.

Le, H. T., & Vea, L.A (2016, January). A Customer Emotion Recognition through Facial Expression using Kinect Sensor v1 and v2: A Comperative Analysis. In Proceedings of the 10th International Conference on Ubiquitous Information Management and Communication (pp. 1-7).

Nakano, T., & Kato, S. (2017, October). Potentiality of 3D Convolutional Neural Networks to Estimate Customer Expectation and Satisfaction. In Consumer Electronics (GCCE), 2017 IEEE 6th Global Conference on (pp. 1-4). IEEE.

Bouzakraoui, M. S., Sadiq, A., & Enneya, N. (2017, March). A Customer Emotion Recognition through Facial Expression using POEM Descriptor und SVM Classifier. In Proceedings of the 2nd international Conference on Big Data, Cloud and Applications (p. 80). ACM.

Lyons, M. J., Kamachi, M., & Gyoba, J. (1997). Japanese Female Facial Expressions (JAFFE). Database of digital images, 3.

Ramani, S. K. (2018, February). Facial Expression Detection Using Neural Network for Customer Based Service. In 2018 International Conference on Computer, Communication, and Signal Processing (ICCCSP) (pp. 1-4). IEEE.

Caroppo, A., Leone, A., & Siciliano, P. (2020). Comparison Between Deep Learning Models and Traditional Machine Learning Approaches for Facial Expression Recognition in Ageing Adults. Journal of Computer Science and Technology, 35(5), 1127-1146.

M. F. Valstar and M. Pantic. Induced Disgust, Happiness and Surprise: an addition to the MMI Facial Expression Database. In Proceedings of International Conference Language Resources and Evaluation, Workshop on EMOTION, pages 65–70, Malta, May 2010.

Qi, X., Wang, T., & Liu, J. (2017). Comparison of Support Vector Machine and Softmax Classifiers in Computer Vision. In 2017 Second International Conference on Mechanical, Control and Computer Engineering (ICMCCE) (pp. 151-155). IEEE.

Shao, J., & Qian, Y. (2019). Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing, 355, 82-92.

Giannopoulos, Panagiotis, Isidoros Perikos, and Ioannis Hatzilygeroudis. Deep Learning Approaches for Facial Emotion Recognition: A Case Study on FER-2013. Advances in Hybridization of Intelligent Methods. Springer, Cham, 1-16, 2018.

Van Kleef, J. (2016). Towards Human-like Performance Face Detection: A Convolutional Neural Network Approach. University of Twente.

Lin, G., & Shen, W. (2018). Research on convolutional neural network based on improved Relu piecewise activation function. Procedia computer science, 131, 977-984.

Ma, N., Zhang, X., Zheng, H. T., & Sun, J. (2018). Shufflenet v2: Practical Guidelines for Efficient CNN Architecture Design. In Proceedings of the European conference on computer vision (ECCV) (pp. 116-131).

Ren, S., He, K., Girshick, R., Zhang, X., & Sun, J. (2016). Object Detection Networks on Convolutional Feature Maps. IEEE transactions on pattern analysis and machine intelligence, 39(7), 1476-1481.

Agarap, A. F. (2018). Deep Learning using Rectified Linear Units (Relu). arXiv preprint arXiv:1803.08375.

C. Cortes and V. Vapnik. 1995. Support-vector Networks. Machine Learning 20.3 (1995), 273–297.

Zhang, T., Zheng, W., Cui, Z., Zong, Y., Yan, J., & Yan, K. (2016). A Deep Neural Network-Driven Feature Learning Method for Multi-View Facial Expression Recognition. IEEE Transactions on Multimedia, 18(12), 2528-2536.

Chen, L., Zhou, M., Su, W., Wu, M., She, J., & Hirota, K. (2018). Softmax Regression Based Deep Sparse Autoencoder Network for Facial Emotion Recognition in Human-Robot Interaction. Information Sciences, 428, 49-61.



  • There are currently no refbacks.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

shopify stats IJEECS visitor statistics