Issue

Vol. 18 (2026)

TENG-Based Self-Powered Silent Speech Recognition Interface: from Assistive Communication to Immersive AR/VR Interaction

https://doi.org/10.1007/s40820-025-01982-z
Shuai Lin
School of Physics and Information Engineering, Fuzhou University, Fuzhou 350108, People’s Republic of China
Yanmin Guo
School of Physics and Information Engineering, Fuzhou University, Fuzhou 350108, People’s Republic of China
Xiangyao Zeng
School of Physics and Information Engineering, Fuzhou University, Fuzhou 350108, People’s Republic of China
Xiongtu Zhou
School of Physics and Information Engineering, Fuzhou University, Fuzhou 350108, People’s Republic of China
Yongai Zhang
School of Physics and Information Engineering, Fuzhou University, Fuzhou 350108, People’s Republic of China
Chengda Li
School of Artificial Intelligence, Xiamen City University, Xiamen 361008, People’s Republic of China
Chaoxing Wu
School of Physics and Information Engineering, Fuzhou University, Fuzhou 350108, People’s Republic of China
Shuai Lin
School of Physics and Information Engineering, Fuzhou University, Fuzhou 350108, People’s Republic of China

Corresponding Author: Chaoxing Wu

chaoxing_wu@fzu.edu.cn

Nano-Micro Letters, Vol. 18 (2026), Article Number: 143

Abstract

Lip language provides a silent, intuitive, and efficient mode of communication, offering a promising solution for individuals with speech impairments. Its articulation relies on complex movements of the jaw and the muscles surrounding it. However, the accurate and real-time acquisition and decoding of these movements into reliable silent speech signals remains a significant challenge. In this work, we propose a real-time silent speech recognition system, which integrates a triboelectric nanogenerator-based flexible pressure sensor (FPS) with a deep learning framework. The FPS employs a porous pyramid–structured silicone film as the negative triboelectric layer, enabling highly sensitive pressure detection in the low-force regime (1 V N− 1 for 0–10 N and 4.6 V N− 1 for 10–24 N). This allows it to precisely capture jaw movements during speech and convert them into electrical signals. To decode the signals, we proposed a convolutional neural network-long short-term memory (CNN–LSTM) hybrid network, combining CNN and LSTM model to extract both local spatial features and temporal dynamics. The model achieved 95.83% classification accuracy in 30 categories of daily words. Furthermore, the decoded silent speech signals can be directly translated into executable commands for contactless and precise control of the smartphone. The system can also be connected to AR glasses, offering a novel human–machine interaction approach with promising potential in AR/VR applications.


Highlights:
1 A porous pyramid-structured triboelectric nanogenerator sensor is designed for self-powered silent speech signal acquisition.
2 A hybrid neural network that combines convolutional neural network with long short-term memory is proposed to accurately decode silent speech signals.
3 Silent speech commands enable real-time, contactless control of smartphones and immersive AR/VR interaction.

Keywords

Flexible pressure sensor Silent speech recognition Triboelectric nanogenerator Deep learning AR/VR interaction
Lin, Shuai, Yanmin Guo, Xiangyao Zeng, Xiongtu Zhou, Yongai Zhang, Chengda Li, Chaoxing Wu, and Shuai Lin. 2026. “TENG-Based Self-Powered Silent Speech Recognition Interface: From Assistive Communication to Immersive AR/VR Interaction”. Nano-Micro Letters 18 (January):143. https://doi.org/10.1007/s40820-025-01982-z.
Download Citation
Endnote/Zotero/Mendeley (RIS)
BibTeX
References
  1. B.M. Jangabaevna, The significant role of language in the expression and transmission of emotions. Am. J. Philol. Sci. 5(4), 93–95 (2025). https://doi.org/10.37547/ajps/Volume05Issue04-23
  2. L. Wong, G. Grand, A.K. Lew, N.D. Goodman, V.K. Mansinghka, J. Andreas, J.B. Tenenbaum, from word models to world models: Translating from natural language to the probabilistic language of thought. arXiv preprint arXiv:2306.12672 (2023). https://doi.org/10.48550/arXiv.2306.12672
  3. S.A. Teli, A.M. Sheikh, S. Jan, J.Y. Pala. Mir Language use during chatting and expressing emotions by Kashmiri speakers using whatsapp. J. South Asian Exch. 1(1) (2024). https://doi.org/10.21659/jsae/v1n1/v1n102
  4. P. Hecker, N. Steckhan, F. Eyben, B.W. Schuller, B. Arnrich, Voice analysis for neurological disorder recognition-a systematic review and perspective on emerging trends. Front. Digit. Health 4, 842301 (2022). https://doi.org/10.3389/fdgth.2022.842301
  5. Y. Shevchenko, S. Dubiaha, O. Kovalova, H. Varina, H. Svyrydenko, Neuropsychological peculiarities of cognitive functions of speech-impaired junior pupils. Conhecimento Divers 15(40), 322–339 (2023). https://doi.org/10.18316/rcd.v15i40.11252
  6. A. Favaro, C. Motley, T. Cao, M. Iglesias, A. Butala et al., A multi-modal array of interpretable features to evaluate language and speech patterns in different neurological disorders. 2022 IEEE spoken language technology workshop (SLT), 532–539. IEEE (2023). https://doi.org/10.1109/SLT54892.2023.10022435
  7. G.P. Usha, J.S.R. Alex, Speech assessment tool methods for speech impaired children: a systematic literature review on the state-of-the-art in speech impairment analysis. Multimed. Tools Appl. (2023). https://doi.org/10.1007/s11042-023-14913-0
  8. C. Alighieri, K. De Maere, G. Poncelet, L. Willekens, C. Vander Linden et al., Occurrence of speech-language disorders in the acute phase following pediatric acquired brain injury: results from the Ghent university hospital. Brain Inj. 35(8), 907–921 (2021). https://doi.org/10.1080/02699052.2021.1927185
  9. A. Barman, A. Chatterjee, R. Bhide, Cognitive impairment and rehabilitation strategies after traumatic brain injury. Indian J. Psychol. Med. 38(3), 172–181 (2016). https://doi.org/10.4103/0253-7176.183086
  10. J. Leblanc, E. De Guise, M. Feyz, J. Lamoureux, Early prediction of language impairment following traumatic brain injury. Brain Inj. 20(13–14), 1391–1401 (2006). https://doi.org/10.1080/02699050601081927
  11. M.M. Smith, Simply a speech impairment? Literacy challenges for individuals with severe congenital speech impairments. Int. J. Disabil. Dev. Educ. 48(4), 331–353 (2001). https://doi.org/10.1080/10349120120094257
  12. C.P. Barnett, B.W.M. van Bon, Monogenic and chromosomal causes of isolated speech and language impairment. J. Med. Genet. 52(11), 719–729 (2015). https://doi.org/10.1136/jmedgenet-2015-103161
  13. K.P. Connaghan, C. Baylor, M. Romanczyk, J. Rickwood, G. Bedell, Communication and social interaction experiences of youths with congenital motor speech disorders. Am. J. Speech Lang. Pathol. 31(6), 2609–2627 (2022). https://doi.org/10.1044/2022_AJSLP-22-00034
  14. A. Dahlgren Sandberg, Phonological recoding problems in children with severe congenital speech impairments: the importance of productive speech. In: basic functions of language, reading and reading disability, pp. 315–327. Springer US (2002). https://doi.org/10.1007/978-1-4615-1011-6_19
  15. R.K. Kadu, A.V. Chandak, P.J. Assudani, A. Tiwari, N. Jaurkar, Sign language recognition by hand gesture for deaf and speech impaired community using ML. 2024 OITS International conference on information technology (OCIT)., 542–547. IEEE (2025). https://doi.org/10.1109/OCIT65031.2024.00100
  16. J.R. Green, C.A. Moore, K.J. Reilly, The sequential development of jaw and lip control for speech. J. Speech Lang. Hear. Res. 45(1), 66–79 (2002). https://doi.org/10.1044/1092-4388(2002/005)
  17. K.S. Talha, K. Wan, S.K. Za’ba, Z.M. Razlan, A.B. Shahriman, Speech analysis based on image information from lip movement. IOP Conf. Ser. Mater. Sci. Eng. 53, 012016 (2013). https://doi.org/10.1088/1757-899x/53/1/012016
  18. M. Bourguignon, M. Baart, E.C. Kapnoula, N. Molinaro, Lip-reading enables the brain to synthesize auditory features of unknown silent speech. J. Neurosci. 40(5), 1053–1065 (2020). https://doi.org/10.1523/JNEUROSCI.1101-19.2019
  19. M. Oghbaie, A. Sabaghi, K. Hashemifard, M. Akbari, When deep learning deciphers silent video: a survey on automatic deep lip reading. Multimed. Tools Appl. 84(32), 40363–40405 (2025). https://doi.org/10.1007/s11042-024-20156-4
  20. C. Yu, X. Wang, Z. Qian, Silent speech recognition using visual cascading fusion of tongue-lip movements based on pre-trained and fine-tuned model. EURASIP J. Audio Speech Music Process. 2025(1), 16 (2025). https://doi.org/10.1186/s13636-025-00403-8
  21. X. Wang, Z. Su, J. Rekimoto, Y. Zhang, Watch your mouth: silent speech recognition with depth sensing. Proceedings of the CHI conference on human factors in computing systems. Honolulu HI USA. ACM, 1–15. https://doi.org/10.1145/3613904.3642092
  22. B. Huang, Y. Shao, H. Zhang, P. Wang, X. Chen et al., Design and implementation of a silent speech recognition system based on sEMG signals: a neural network approach. Biomed. Signal Process. Control 92, 106052 (2024). https://doi.org/10.1016/j.bspc.2024.106052
  23. Z. Li, B. Ma, W. Mao, J. Zhang, Z. Yu et al., SVIT-SSR: a sEMG-based vision transformer approach for silent speech recognition. Electron. Lett. 60(21), e13285 (2024). https://doi.org/10.1049/ell2.13285
  24. J. Menezes, C. Wagner, P. Steiner, P. Schaffer, D. Plettemeier et al., Non-invasive speaker-dependent continuous phoneme recognition with a radar-based silent speech interface. ICASSP 2025 - 2025 IEEE international conference on acoustics, speech and signal processing (ICASSP), 1–5. IEEE (2025). https://doi.org/10.1109/ICASSP49660.2025.10887719
  25. J. Menezes, M. Schütze, P. Schaffer, D. Plettemeier, P. Birkholz, Exploring antenna placement configurations with a radar-based silent speech interface. ICASSP 2025 - 2025 IEEE international conference on acoustics, speech and signal processing (ICASSP), 1–5. IEEE (2025). https://doi.org/10.1109/ICASSP49660.2025.10890284
  26. G. Zhu, C. Pan, W. Guo, C.-Y. Chen, Y. Zhou et al., Triboelectric-generator-driven pulse electrodeposition for micropatterning. Nano Lett. 12(9), 4960–4965 (2012). https://doi.org/10.1021/nl302560k
  27. F.-R. Fan, Z.-Q. Tian, Z.L. Wang, Flexible triboelectric generator. Nano Energy 1(2), 328–334 (2012). https://doi.org/10.1016/j.nanoen.2012.01.004
  28. S. Fu, W. He, H. Wu, C. Shan, Y. Du et al., High output performance and ultra-durable DC output for triboelectric nanogenerator inspired by primary cell. Nano-Micro Lett. 14(1), 155 (2022). https://doi.org/10.1007/s40820-022-00898-2
  29. K.-H. Lee, M.-G. Kim, W. Kang, H.-M. Park, Y. Cho et al., Pulse-charging energy storage for triboelectric nanogenerator based on frequency modulation. Nano-Micro Lett. 17(1), 210 (2025). https://doi.org/10.1007/s40820-025-01714-3
  30. G. Du, J. Zhao, Y. Shao, T. Liu, B. Luo et al., A self-damping triboelectric tactile patch for self-powered wearable electronics. eScience 5(2), 100324 (2025). https://doi.org/10.1016/j.esci.2024.100324
  31. B. Shi, Q. Wang, H. Su, J. Li, B. Xie et al., Progress in recent research on the design and use of triboelectric nanogenerators for harvesting wind energy. Nano Energy 116, 108789 (2023). https://doi.org/10.1016/j.nanoen.2023.108789
  32. Z. Zhao, Z. Quan, H. Tang, Q. Xu, H. Zhao et al., A broad range triboelectric stiffness sensor for variable inclusions recognition. Nano-Micro Lett. 15(1), 233 (2023). https://doi.org/10.1007/s40820-023-01201-7
  33. Z. Xu, D. Li, K. Wang, Y. Liu, J. Wang et al., Stomatopod-inspired integrate-and-fire triboelectric nanogenerator for harvesting mechanical energy with ultralow vibration speed. Appl. Energy 312, 118739 (2022). https://doi.org/10.1016/j.apenergy.2022.118739
  34. K. Wang, Y. Weng, G. Chen, C. Wu, J.H. Park et al., Coupling electrostatic induction and global electron circulation for constant-current triboelectric nanogenerators. Nano Energy 85, 105929 (2021). https://doi.org/10.1016/j.nanoen.2021.105929
  35. Y. Wang, Z. Gao, W. Wu, Y. Xiong, J. Luo et al., TENG-boosted smart sports with energy autonomy and digital intelligence. Nano-Micro Lett. 17(1), 265 (2025). https://doi.org/10.1007/s40820-025-01778-1
  36. W. Chen, J. Kang, J. Zhang, Y. Zhang, X. Zhou et al., An information display and encrypted transmission system based on a triboelectric nanogenerator and a cholesteric liquid crystal. Nano Energy 134, 110594 (2025). https://doi.org/10.1016/j.nanoen.2024.110594
  37. W. Dong, K. Sheng, B. Huang, K. Xiong, K. Liu et al., Stretchable self-powered TENG sensor array for human-robot interaction based on conductive ionic gels and LSTM neural network. IEEE Sens. J. 24(22), 37962–37969 (2024). https://doi.org/10.1109/JSEN.2024.3464633
  38. S. Jiang, X. Liu, J. Liu, D. Ye, Y. Duan et al., Flexible metamaterial electronics. Adv. Mater. 34(52), 2200070 (2022). https://doi.org/10.1002/adma.202200070
  39. F.-R. Fan, L. Lin, G. Zhu, W. Wu, R. Zhang et al., Transparent triboelectric nanogenerators and self-powered pressure sensors based on micropatterned plastic films. Nano Lett. 12(6), 3109–3114 (2012). https://doi.org/10.1021/nl300988z
  40. Y. Qin, X. Ma, Z. Ruan, X. Xiang, Z. Shi et al., Improvement of thermal stability of charges in polylactic acid electret films for biodegradable electromechanical sensors. ACS Appl. Mater. Interfaces 16(45), 62680–62692 (2024). https://doi.org/10.1021/acsami.4c13772
  41. P. Zhang, Y. Ma, H. Zhang, L. Deng, High-performance triboelectric nanogenerators based on foaming agent-modified porous PDMS films with multiple pore sizes. ACS Appl. Energy Mater. 6(12), 6598–6606 (2023). https://doi.org/10.1021/acsaem.3c00633
Read More
Author biographies are not available.

Most read articles by the same author(s)