Reconocimiento de Gestos para HoloLens2 con Puntos 3D de las Manos Bajo Limitaciones de Datos
DOI:
https://doi.org/10.65234/interaccion.120Palabras clave:
Aprendizaje automático, Realidad mixta, Reconocimiento de gestos de la mano, Interacción Persona-Ordenador, Inteligencia Artificial, Aprendizaje con pocos Ejemplos, Microsoft HoloLens 2Resumen
En la actualidad, la interacción persona-ordenador busca ser cada vez más intuitiva. En el caso de la realidad mixta, el uso de gestos emerge como una solución factible para lograr interacciones más naturales y fluidas. En este trabajo, implementamos un sistema completo de reconocimiento de gestos de las manos para las Microsoft HoloLens 2, basado en un par de clasificadores en cascada. Para el entrenamiento de los modelos, utilizamos la parte disponible públicamente del conjunto de datos SHREC22, que cuenta con un número limitado de muestras, convirtiendo esta tarea en un problema de aprendizaje con pocos ejemplos, ya que solo disponemos de 36 muestras por clase. Exploramos diversas arquitecturas de redes neuronales para identificar la más adecuada en este contexto. Al evaluar el sistema en su conjunto, logramos una tasa de error de gestos (GER) del 9.6%, lo que demuestra el potencial del enfoque propuesto, si bien su rendimiento podría optimizarse con futuros ajustes y más datos de entrenamiento.
Referencias
Asadi, A. R., & Hemadi, R. (2024). Towards Mixed Reality as the Everyday Computing Paradigm: Challenges & Design Recommendations. https://doi.org/10.48550/arxiv.2402.15974 DOI: https://doi.org/10.1007/978-3-031-73128-0_37
Avola, D., Bernardi, M., Cinque, L., Foresti, G. L., & Massaroni, C. (2018). Exploiting Recurrent Neural Networks and Leap Motion Controller for Sign Language and Semaphoric Gesture Recognition. IEEE Transactions on Multimedia, 21(1), 234-245. https://doi.org/10.1109/tmm.2018.2856094 DOI: https://doi.org/10.1109/TMM.2018.2856094
Balakrishnan, P., & Guo, H.-J. (2024). HoloLens 2 Technical Evaluation as Mixed Reality Guide. Lecture Notes in Computer Science, 145-165. https://doi.org/10.1007/978-3-031-61041-7_10 DOI: https://doi.org/10.1007/978-3-031-61041-7_10
Bannon, L. (2011). Reimagining HCI: toward a more human-centered perspective. Interactions, 18(4), 50. https://doi.org/10.1145/1978822.1978833 DOI: https://doi.org/10.1145/1978822.1978833
Benedict, J. D., Guliuzo, J. D., & Chaparro, B. S. (2019). The Intuitiveness of Gesture Control with a Mixed Reality Device. Proceedings of the Human Factors and Ergonomics Society, 63(1), 1435-1439. https://doi.org/10.1177/1071181319631403 DOI: https://doi.org/10.1177/1071181319631403
Caputo, A., Giachetti, A., Giannini, F., Lupinetti, K., Monti, M., Pegoraro, M., & Ranieri, A. (2020). SFINGE 3D: A novel benchmark for online detection and recognition of heterogeneous hand gestures from 3D fingers’ trajectories. Computers & Graphics, 91, 232-242. https://doi.org/10.1016/J.CAG.2020.07.014 DOI: https://doi.org/10.1016/j.cag.2020.07.014
Caputo, A., Giachetti, A., Soso, S., Pintani, D., D’Eusanio, A., Pini, S., Borghi, G., Simoni, A., Vezzani, R., Cucchiara, R., Ranieri, A., Giannini, F., Lupinetti, K., Monti, M., Maghoumi, M., LaViola, J. J., Le, M. Q., Nguyen, H. D., & Tran, M. T. (2021). SHREC 2021: Skeleton-based hand gesture recognition in the wild. Computers and Graphics (Pergamon), 99, 201-211. https://doi.org/10.1016/j.cag.2021.07.007 DOI: https://doi.org/10.1016/j.cag.2021.07.007
Chakraborty, B. K., Sarma, D., Bhuyan, M. K., & MacDorman, K. F. (2018). Review of constraints on vision-based gesture recognition for human-computer interaction. IET Computer Vision, 12(1), 3-15. https://doi.org/10.1049/IET-CVI.2017.0052 DOI: https://doi.org/10.1049/iet-cvi.2017.0052
De Smedt, Q., Wannous, H., & Vandeborre, J. P. (2016). Skeleton-Based Dynamic Hand Gesture Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1206-1214. https://doi.org/10.1109/CVPRW.2016.153 DOI: https://doi.org/10.1109/CVPRW.2016.153
Dipesh Gyawali. (2023). Mixed Reality: The Interface of the Future. https://doi.org/10.48550/arxiv.2309.00819
Emporio, M., Caputo, A., & Giachetti, A. (2021). STRONGER: Simple TRajectory-based ONline GEsture Recognizer. Eurographics Italian Chapter Proceedings - Smart Tools and Applications in Graphics, STAG, 109-117. https://doi.org/10.2312/stag.20211481
Emporio, M., Caputo, A., Giachetti, A., Cristani, M., Borghi, G., D’Eusanio, A., Le, M. Q., Nguyen, H. D., Tran, M. T., Ambellan, F., Hanik, M., Nava-Yazdani, E., & von Tycowicz, C. (2022). SHREC 2022 track on online detection of heterogeneous gestures. Computers and Graphics (Pergamon), 107, 241-251. https://doi.org/10.1016/j.cag.2022.07.015 DOI: https://doi.org/10.1016/j.cag.2022.07.015
Guo, L., Lu, Z., & Yao, L. (2021). Human-Machine Interaction Sensing Technology Based on Hand Gesture Recognition: A Review. IEEE Transactions on Human-Machine Systems, 51(4), 300-309. https://doi.org/10.1109/THMS.2021.3086003 DOI: https://doi.org/10.1109/THMS.2021.3086003
He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-December, 770-778. https://doi.org/10.1109/CVPR.2016.90 DOI: https://doi.org/10.1109/CVPR.2016.90
Köpüklü, O., Gunduz, A., Kose, N., & Rigoll, G. (2019). Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks. Proceedings - 14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019. https://doi.org/10.1109/FG.2019.8756576 DOI: https://doi.org/10.1109/FG.2019.8756576
Maghoumi, M., & LaViola, J. J. (2018). DeepGRU: Deep Gesture Recognition Utility. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11844 LNCS, 16-31. https://doi.org/10.1007/978-3-030-33720-9_2 DOI: https://doi.org/10.1007/978-3-030-33720-9_2
Marin, G., Dominio, F., & Zanuttigh, P. (2016). Hand gesture recognition with jointly calibrated Leap Motion and depth sensor. Multimedia Tools and Applications, 75(22), 14991-15015. https://doi.org/10.1007/S11042-015-2451-6 DOI: https://doi.org/10.1007/s11042-015-2451-6
Milgram, P., & Kishino, F. (1994). A Taxonomy of Mixed Reality Visual Displays. IEICE Transactions on Information and Systems, 77, 1321-1329.
Milgram, P., Takemura, H., Utsumi, A., & Kishino, F. (1995). Augmented reality: a class of displays on the reality-virtuality continuum. Telemanipulator and Telepresence Technologies, 2351. https://doi.org/10.1117/12.197321 DOI: https://doi.org/10.1117/12.197321
Murtagh, F. (1991). Multilayer perceptrons for classification and regression. Neurocomputing, 2(5-6), 183-197. https://doi.org/10.1016/0925-2312(91)90023-5 DOI: https://doi.org/10.1016/0925-2312(91)90023-5
Parcheta, Z., & Martínez-Hinarejos, C. D. (2017). Sign language gesture recognition using HMM. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 10255 LNCS, 419-426. https://doi.org/10.1007/978-3-319-58838-4_46 DOI: https://doi.org/10.1007/978-3-319-58838-4_46
Ribeiro, R., Ramos, J., Safadinho, D., Reis, A., Rabadão, C., Barroso, J., & Pereira, A. (2021). Web AR Solution for UAV Pilot Training and Usability Testing. Sensors, 21(4), 1456. https://doi.org/10.3390/s21041456 DOI: https://doi.org/10.3390/s21041456
Rokhsaritalemi, S., Sadeghi-Niaraki, A., & Choi, S. -M. (2020). A Review on Mixed Reality: Current Trends, Challenges and Prospects. Applied Sciences, 10(2), 636. https://doi.org/10.3390/app10020636 DOI: https://doi.org/10.3390/app10020636
Soler-Dominguez, J. L., Navas-Medrano, S., & Pons, P. (2024). ARCADIA: A Gamified Mixed Reality System for Emotional Regulation and Self-Compassion. https://doi.org/10.1145/3613904.3642123 DOI: https://doi.org/10.1145/3613904.3642123
Spittle, B., Frutos-Pascual, M., Creed, C., & Williams, I. (2022). A Review of Interaction Techniques for Immersive Environments. IEEE Transactions on Visualization and Computer Graphics, 1-1. https://doi.org/10.1109/TVCG.2022.3174805 DOI: https://doi.org/10.1109/TVCG.2022.3174805
Yang, F., Sakti, S., Wu, Y., & Nakamura, S. (2019). Make Skeleton-based Action Recognition Model Smaller, Faster and Better. http://arxiv.org/abs/1907.09658 DOI: https://doi.org/10.1145/3338533.3366569
Zaccardi, S., Frantz, T., Beckwée, D., Swinnen, E., & Jansen, B. (2023). On-Device Execution of Deep Learning Models on HoloLens2 for Real-Time Augmented Reality Medical Applications. Sensors 2023, Vol. 23, Page 8698, 23(21), 8698. https://doi.org/10.3390/S23218698 DOI: https://doi.org/10.3390/s23218698
Zaman Khan, R., & Ibraheem, N. (2012). Hand Gesture Recognition: A Literature Review. International Journal of Artificial Intelligence & Applications, 3(4), 161-174. https://doi.org/10.5121/ijaia.2012.3412 DOI: https://doi.org/10.5121/ijaia.2012.3412
Descargas
Publicado
Número
Sección
Licencia
Derechos de autor 2026 Mario Andreu Villar, Carlos David Martínez Hinarejos, Patricia Pons Tomas, Jose Luis Soler Domínguez, Samuel Navas Medrano, Vicent Ortiz Castelló, Marta García Ballesteros
Esta obra está bajo una licencia internacional Creative Commons Atribución-NoComercial 4.0.
