Abstract
The preservation and transmission of Intangible Cultural Heritage (ICH) face significant challenges in today’s globalised world, mainly when teaching complex traditional handicraft techniques that involve repetitive and precise hand movements. This study explores the application of Mixed Reality (MR) technology combined with gesture recognition to enhance the teaching and training of these repetitive movements, focusing on the thread-winding process in Yunnan-style kite making as a case study. MR enables learners to interact with virtual materials in a simulated environment, while gesture recognition, implemented using Google’s MediaPipe, captures and evaluates hand movements in real time.
By comparing learners’ gestures with the movements of skilled artisans, the MR system provides immediate feedback to correct technique and improve accuracy. The research demonstrates that MR-assisted gesture recognition significantly enhances the standardisation of repetitive movements, improves teaching quality, and reduces training time and material costs. Importantly, repetitive actions—essential to mastering traditional handcrafts—benefit from the immersive, interactive feedback that MR provides, helping learners refine their movements through continuous practice.
This study contributes to the growing field of digital heritage education by showcasing how MR can modernise the transmission of traditional craft skills. In addition, it highlights the potential of gesture recognition technology in advancing the teaching and training of intricate motor skills in ICH practices. The experimental results suggest that MR-based systems could be valuable in preserving and promoting craftsmanship by offering scalable, cost-effective training solutions.
Keywords: Mixed Reality; Gesture Recognition; MediaPipe; Intangible Cultural Heritage; Traditional Handicrafts
References
- Azuma RT. “A survey of augmented reality”. Presence: Teleoperators & Virtual Environments 6.4 (1997): 355-385.
- Billinghurst M and Kato H. “Collaborative mixed reality. In Proceedings of the First International Symposium on Mixed and Augmented Reality”. IEEE (2002): 2-9.
- Bronstein MM., et al. “Data fusion through Euclidean and non-Euclidean manifold modeling: Applications to sensor networks”. IEEE Transactions on Signal Processing 56.7 (2008): 2678-2689.
- Chen H and Hu W. “The impact of online learning platforms on traditional craft transmission”. Journal of Cultural Heritage Studies 12.3 (2021): 85-97.
- Chen CH and Wang JJ. “Virtual Reality in Education: Mixed Reality Applications for Vocational Training”. Journal of Educational Technology 38.2 (2021): 125-145.
- Chen HT and Xu J. “Gesture recognition using multi-modal sensor fusion: A study in handcraft preservation”. Journal of Human-Computer Interaction 37.9 (2021): 1132-1145.
- Liu Z and Zhang L. “Challenges in the preservation of intangible cultural heritage in the digital age”. International Journal of Cultural Studies 23.2 (2020): 176-189.
- Lugaresi C., et al. “MediaPipe: A framework for building perception pipelines”. arXiv preprint arXiv:1906.08172 (2019).
- Tatzgern M., et al. “Adaptive and robust hand gesture recognition for interactive systems”. IEEE Transactions on Visualization and Computer Graphics 21.5 (2015): 659-668.
- Taylor AM and Kumar S. “Preserving cultural heritage through immersive technologies: A mixed reality approach to craft education”. Virtual Reality & Intelligent Hardware 3.2 (2021): 145-157.
- Xu Z, Chen P and Qian S. “Advances in gesture recognition systems: Applications to vocational training and traditional handcrafts”. International Journal of Human-Computer Studies 157 (2022): 102770.
- Yang J. “The challenges of globalisation in preserving local cultural practices: A case study of bamboo crafts”. International Journal of Heritage Studies 25.1 (2019): 55-72.
- Zhang Y, Wang Z and Chen X. “Multimodal Gesture Recognition: Combining EMG and Computer Vision”. Journal of Neuro Engineering and Rehabilitation 17.5 (2020): 178-190.