New blender-based augmentation method with quantitative evaluation of CNNs for hand gesture recognition

Abstract

In this study, we extensively analyze and evaluate the performance of recent deep neural networks (DNNs) for hand gesture recognition and static gestures in particular. To this end, we captured an unconstrained hand dataset with complex appearances, shapes, scales, backgrounds, and viewpoints. We then deployed some new trending convolution neuron networks (CNNs) for gesture classification. We arrived at three major conclusions: i) DenseNet121 architecture is the best recognition rate through almost evaluated red, green, blue (RGB) and augmentation datasets. Its performance is outstanding in most original works; ii) blender-based augmentation help to significantly increase 9% of accuracy, compared to the use of a RGB cues; iii) most CNNs can achieve impressive results at 97% accuracy when the training and testing datasets come from the same lab-based or constrained environment. Their performance is drastically reduced when dealing with gestures collected in unconstrained environments. In particular, we validated the best CNN on a new unconstrained dataset. We observed a significant reduction with an accuracy of only 74.55%. This performance can be improved up to 80.59% by strategies such as blender-based and/or GAN-based data augmentations to obtain an acceptable result of 83.17%. These findings contribute crucial factors and make fruitful recommendations for the development of a robust hand-based interface in practice