Issue

Vol. 15 (2023)

Humanoid Intelligent Display Platform for Audiovisual Interaction and Sound Identification

https://doi.org/10.1007/s40820-023-01199-y
Yang Wang
School of Physical Science and Technology, ShanghaiTech University, 393 Middle Huaxia Road, Shanghai 201210, People’s Republic of China
Wenli Gao
School of Physical Science and Technology, ShanghaiTech University, 393 Middle Huaxia Road, Shanghai 201210, People’s Republic of China
Shuo Yang
School of Physical Science and Technology, ShanghaiTech University, 393 Middle Huaxia Road, Shanghai 201210, People’s Republic of China
Qiaolin Chen
State Key Laboratory of Molecular Engineering of Polymers, Department of Macromolecular Science, Laboratory of Advanced Materials, Fudan University, Shanghai 200433, People’s Republic of China
Chao Ye
School of Textile and Clothing, Yancheng Institute of Technology, Jiangsu 224051, People’s Republic of China
Hao Wang
School of Physical Science and Technology, ShanghaiTech University, 393 Middle Huaxia Road, Shanghai 201210, People’s Republic of China
Qiang Zhang
School of Textile Science and Engineering, Wuhan Textile University, Wuhan 430200, People’s Republic of China
Jing Ren
School of Physical Science and Technology, ShanghaiTech University, 393 Middle Huaxia Road, Shanghai 201210, People’s Republic of China
Zhijun Ning
School of Physical Science and Technology, ShanghaiTech University, 393 Middle Huaxia Road, Shanghai 201210, People’s Republic of China
Xin Chen
State Key Laboratory of Molecular Engineering of Polymers, Department of Macromolecular Science, Laboratory of Advanced Materials, Fudan University, Shanghai 200433, People’s Republic of China
Zhengzhong Shao
State Key Laboratory of Molecular Engineering of Polymers, Department of Macromolecular Science, Laboratory of Advanced Materials, Fudan University, Shanghai 200433, People’s Republic of China
Jian Li
School of Physical Science and Technology, ShanghaiTech University, 393 Middle Huaxia Road, Shanghai 201210, People’s Republic of China
Yifan Liu
School of Physical Science and Technology, ShanghaiTech University, 393 Middle Huaxia Road, Shanghai 201210, People’s Republic of China
Shengjie Ling
School of Physical Science and Technology, ShanghaiTech University, 393 Middle Huaxia Road, Shanghai 201210, People’s Republic of China

Corresponding Author: Shengjie Ling

lingshj@shanghaitech.edu.cn

Nano-Micro Letters, Vol. 15 (2023), Article Number: 221

Abstract

This study proposes a rational strategy for the design, fabrication and system integration of the humanoid intelligent display platform (HIDP) to meet the requirements of highly humanized mechanical properties and intelligence for human–machine interfaces. The platform's sandwich structure comprises a middle light-emitting layer and surface electrodes, which consists of silicon elastomer embedded with phosphor and silk fibroin ionoelastomer, respectively. Both materials are highly stretchable and resilient, endowing the HIDP with skin-like mechanical properties and applicability in various extreme environments and complex mechanical stimulations. Furthermore, by establishing the numerical correlation between the amplitude change of animal sounds and the brightness variation, the HIDP realizes audiovisual interaction and successful identification of animal species with the aid of Internet of Things (IoT) and machine learning techniques. The accuracy of species identification reaches about 100% for 200 rounds of random testing. Additionally, the HIDP can recognize animal species and their corresponding frequencies by analyzing sound characteristics, displaying real-time results with an accuracy of approximately 99% and 93%, respectively. In sum, this study offers a rational route to designing intelligent display devices for audiovisual interaction, which can expedite the application of smart display devices in human–machine interaction, soft robotics, wearable sound-vision system and medical devices for hearing-impaired patients.


Highlights:
1 A humanoid intelligent display platform (HIDP) is created using stretchable and resilient ionotronic materials, and can be applicable in extreme environments and complex mechanical stimulations.
2 HIDP links sound amplitude and brightness through machine learning for audiovisual interaction.
3 HIDP identifies and displays animal species and corresponding frequencies in real-time.

Keywords

Silk fibroin Ionoelastomer Luminescent display Machine learning Audiovisual interaction
Wang, Yang, Wenli Gao, Shuo Yang, Qiaolin Chen, Chao Ye, Hao Wang, Qiang Zhang, Jing Ren, Zhijun Ning, Xin Chen, Zhengzhong Shao, Jian Li, Yifan Liu, and Shengjie Ling. 2023. “Humanoid Intelligent Display Platform for Audiovisual Interaction and Sound Identification”. Nano-Micro Letters 15 (October):221. https://doi.org/10.1007/s40820-023-01199-y.
Download Citation
Endnote/Zotero/Mendeley (RIS)
BibTeX
References
  1. C. Larson, B. Peele, S. Li, S. Robinson, M. Totaro et al., Highly stretchable electroluminescent skin for optical signaling and tactile sensing. Science 351, 1071–1074 (2016). https://doi.org/10.1126/science.aac5082
  2. H. Yin, Y. Zhu, K. Youssef, Z. Yu, Q. Pei, Structures and materials in stretchable electroluminescent devices. Adv. Mater. 34, 2106184 (2022). https://doi.org/10.1002/adma.202106184
  3. Z. Yang, W. Wang, J. Pan, C. Ye, Alternating current electroluminescent devices with inorganic phosphors for deformable displays. Cell Rep. Phys. Sci. 1, 100213 (2020). https://doi.org/10.1016/j.xcrp.2020.100213
  4. Z. Zhang, Light-emitting materials for wearable electronics. Nat. Rev. Mater. 7, 839–840 (2022). https://doi.org/10.1038/s41578-022-00502-4
  5. Y. Zhang, Y. Fang, J. Li, Q. Zhou, Y. Xiao et al., Dual-mode electronic skin with integrated tactile sensing and visualized injury warning. ACS Appl. Mater. Interfaces 9, 37493–37500 (2017). https://doi.org/10.1021/acsami.7b13016
  6. S.A. Morin, R.F. Shepherd, S.W. Kwok, A.A. Stokes, A. Nemiroski et al., Camouflage and display for soft machines. Science 337, 828–832 (2012). https://doi.org/10.1126/science.1222149
  7. T. Yokota, P. Zalar, M. Kaltenbrunner, H. Jinno, N. Matsuhisa et al., Ultraflexible organic photonic skin. Sci. Adv. 2, e1501856 (2016). https://doi.org/10.1126/sciadv.1501856
  8. B. Lee, J.-Y. Oh, H. Cho, C.W. Joo, H. Yoon et al., Ultraflexible and transparent electroluminescent skin for real-time and super-resolution imaging of pressure distribution. Nat. Commun. 11, 663 (2020). https://doi.org/10.1038/s41467-020-14485-9
  9. Y. Wang, J. Ren, Z. Lv, L. Cao, S. Lin et al., Direct functionalization of natural silks through continuous force-reeling technique. Chem. Eng. J. 435, 134901 (2022). https://doi.org/10.1016/j.cej.2022.134901
  10. J.H. Koo, S. Jeong, H.J. Shim, D. Son, J. Kim et al., Wearable electrocardiogram monitor using carbon nanotube electronics and color-tunable organic light-emitting diodes. ACS Nano 11, 10032–10041 (2017). https://doi.org/10.1021/acsnano.7b04292
  11. X. Shi, Y. Zuo, P. Zhai, J. Shen, Y. Yang et al., Large-area display textiles integrated with functional systems. Nature 591, 240–245 (2021). https://doi.org/10.1038/s41586-021-03295-8
  12. Y. Wu, S.S. Mechael, C. Lerma, R.S. Carmichael, T.B. Carmichael, Stretchable ultrasheer fabrics as semitransparent electrodes for wearable light-emitting e-textiles with changeable display patterns. Matter 2, 882–895 (2020). https://doi.org/10.1016/j.matt.2020.01.017
  13. C. Dai, Y. Wang, Y. Shan, C. Ye, Z. Lv et al., Cytoskeleton-inspired hydrogel ionotronics for tactile perception and electroluminescent display in complex mechanical environments. Mater. Horiz. 10, 136–148 (2023). https://doi.org/10.1039/D2MH01034H
  14. Y. Lee, J.W. Chung, G.H. Lee, H. Kang, J.-Y. Kim et al., Standalone real-time health monitoring patch based on a stretchable organic optoelectronic system. Sci. Adv. 7, eabg9180 (2021). https://doi.org/10.1126/sciadv.abg9180
  15. A. Robinson, A. Aziz, Q. Liu, Z. Suo, S.P. Lacour, Hybrid stretchable circuits on silicone substrate. J. Appl. Phys. 115, 143511 (2014). https://doi.org/10.1063/1.4871279
  16. S.-I. Park, Y. Xiong, R.-H. Kim, P. Elvikis, M. Meitl et al., Printed assemblies of inorganic light-emitting diodes for deformable and semitransparent displays. Science 325, 977–981 (2009). https://doi.org/10.1126/science.1175690
  17. T. Kim, H. Lee, W. Jo, T.-S. Kim, S. Yoo, Realizing stretchable oleds: a hybrid platform based on rigid island arrays on a stress-relieving bilayer structure. Adv. Mater. Technol. 5, 2000494 (2020). https://doi.org/10.1002/admt.202000494
  18. J. Liang, L. Li, X. Niu, Z. Yu, Q. Pei, Elastomeric polymer light-emitting devices and displays. Nat. Photon. 7, 817–824 (2013). https://doi.org/10.1038/nphoton.2013.242
  19. J. Wang, C. Yan, K.J. Chee, P.S. Lee, Highly stretchable and self-deformable alternating current electroluminescent devices. Adv. Mater. 27, 2876–2882 (2015). https://doi.org/10.1002/adma.201405486
  20. C.H. Yang, B. Chen, J. Zhou, Y.M. Chen, Z. Suo, Electroluminescence of giant stretchability. Adv. Mater. 28, 4480–4484 (2016). https://doi.org/10.1002/adma.201504031
  21. F. Stauffer, K. Tybrandt, Bright stretchable alternating current electroluminescent displays based on high permittivity composites. Adv. Mater. 28, 7200–7203 (2016). https://doi.org/10.1002/adma.201602083
  22. Z. Yu, X. Niu, Z. Liu, Q. Pei, Intrinsically stretchable polymer light-emitting devices using carbon nanotube-polymer composite electrodes. Adv. Mater. 23, 3989–3994 (2011). https://doi.org/10.1002/adma.201101986
  23. X. Wang, J. Sun, L. Dong, C. Lv, K. Zhang et al., Stretchable and transparent electroluminescent device driven by triboelectric nanogenerator. Nano Energy 58, 410–418 (2019). https://doi.org/10.1016/j.nanoen.2019.01.058
  24. H. Shin, B.K. Sharma, S.W. Lee, J.-B. Lee, M. Choi et al., Stretchable electroluminescent display enabled by graphene-based hybrid electrode. ACS Appl. Mater. Interfaces 11, 14222–14228 (2019). https://doi.org/10.1021/acsami.8b22135
  25. J. Ge, H.-B. Yao, X. Wang, Y.-D. Ye, J.-L. Wang et al., Stretchable conductors based on silver nanowires: improved performance through a binary network design. Angew. Chem. Int. Ed. 52, 1654–1659 (2013). https://doi.org/10.1002/anie.201209596
  26. S. Choi, S.I. Han, D. Jung, H.J. Hwang, C. Lim et al., Highly conductive, stretchable and biocompatible ag–au core–sheath nanowire composite for wearable and implantable bioelectronics. Nat. Nanotechnol. 13, 1048–1056 (2018). https://doi.org/10.1038/s41565-018-0226-8
  27. B. You, Y. Kim, B.-K. Ju, J.-W. Kim, Highly stretchable and waterproof electroluminescence device based on superstable stretchable transparent electrode. ACS Appl. Mater. Interfaces 9, 5486–5494 (2017). https://doi.org/10.1021/acsami.6b14535
  28. L. Cai, S. Zhang, Y. Zhang, J. Li, J. Miao et al., Direct printing for additive patterning of silver nanowires for stretchable sensor and display applications. Adv. Mater. Technol. 3, 1700232 (2018). https://doi.org/10.1002/admt.201700232
  29. Y. Chen, Y. Wu, S.S. Mechael, T.B. Carmichael, Heterogeneous surface orientation of solution-deposited gold films enables retention of conductivity with high strain—a new strategy for stretchable electronics. Chem. Mater. 31, 1920–1927 (2019). https://doi.org/10.1021/acs.chemmater.8b04487
  30. D. Son, J. Kang, O. Vardoulis, Y. Kim, N. Matsuhisa et al., An integrated self-healable electronic skin system fabricated via dynamic reconstruction of a nanostructured conducting network. Nat. Nanotechnol. 13, 1057–1065 (2018). https://doi.org/10.1038/s41565-018-0244-6
  31. J.I. Lee, H. Choi, S.H. Kong, S. Park, D. Park et al., Visco-poroelastic electrochemiluminescence skin with piezo-ionic effect. Adv. Mater. 33, 2100321 (2021). https://doi.org/10.1002/adma.202100321
  32. C. Li, Q. He, Y. Wang, Z. Wang, Z. Wang et al., Highly robust and soft biohybrid mechanoluminescence for optical signaling and illumination. Nat. Commun. 13, 3914 (2022). https://doi.org/10.1038/s41467-022-31705-6
  33. D.-K. Kwon, J.-M. Myoung, Wearable and semitransparent pressure-sensitive light-emitting sensor based on electrochemiluminescence. ACS Nano 14, 8716–8723 (2020). https://doi.org/10.1021/acsnano.0c03186
  34. S.W. Lee, S.H. Cho, H.S. Kang, G. Kim, J.S. Kim et al., Electroluminescent pressure-sensing displays. ACS Appl. Mater. Interfaces 10, 13757–13766 (2018). https://doi.org/10.1021/acsami.8b01790
  35. X.Y. Wei, X. Wang, S.Y. Kuang, L. Su, H.Y. Li et al., Dynamic triboelectrification-induced electroluminescence and its use in visualized sensing. Adv. Mater. 28, 6656–6664 (2016). https://doi.org/10.1002/adma.201600604
  36. Y. Wang, H. Wu, L. Xu, H. Zhang, Y. Yang et al., Hierarchically patterned self-powered sensors for multifunctional tactile sensing. Sci. Adv. 6, eabb9083 (2020). https://doi.org/10.1126/sciadv.abb9083
  37. X. Zhou, X. Xu, Y. Zuo, M. Liao, X. Shi et al., A fiber-shaped light-emitting pressure sensor for visualized dynamic monitoring. J. Mater. Chem. C 8, 935–942 (2020). https://doi.org/10.1039/C9TC05653J
  38. L. Su, Z. Jiang, Z. Tian, H. Wang, H. Wang et al., Self-powered, ultrasensitive, and high-resolution visualized flexible pressure sensor based on color-tunable triboelectrification-induced electroluminescence. Nano Energy 79, 105431 (2021). https://doi.org/10.1016/j.nanoen.2020.105431
  39. T. Kaneko, K. Tanaka, H. Kameoka, S. Seki, presented at ICASSP 2022-2022 IEEE International Conference, iSTFTNet: Fast and lightweight mel-spectrogram vocoder incorporating inverse short-time Fourier transform, Japan, March, 2022.
  40. H. Zhang, L. Cao, J. Li, Y. Liu, Z. Lv et al., Dual physically crosslinked silk fibroin ionoelastomer with ultrahigh stretchability and low hysteresis. Chem. Mater. 4, 1752–1761 (2023). https://doi.org/10.1021/acs.chemmater.2c03536
  41. Q. Liu, S. Yang, J. Ren, S. Ling, Flame-retardant and sustainable silk ionotronic skin for fire alarm systems. ACS Mater. Lett. 2, 712–720 (2020). https://doi.org/10.1021/acsmaterialslett.0c00062
  42. Q. Liu, X. Li, H. Zhang, J. Ren, S. Yang et al., Intellisense silk fibroin ionotronic batteries for wildfire detection and alarm. Nano Energy 101, 107630 (2022). https://doi.org/10.1016/j.nanoen.2022.107630
  43. S. Yang, Q. Liu, J. Ren, S. Ling, Influence of hydrated protons on temperature and humidity responsiveness of silk fibroin hydrogel ionotronics. Giant 5, 100044 (2021). https://doi.org/10.1016/j.giant.2020.100044
  44. C. Dai, C. Ye, J. Ren, S. Yang, L. Cao et al., Humanoid ionotronic skin for smart object recognition and sorting. ACS Mater. Lett. 5, 189–201 (2023). https://doi.org/10.1021/acsmaterialslett.2c00783
  45. Y. Zhu, Y. Xia, M. Wu, W. Guo, C. Jia et al., Wearable, freezing-tolerant, and self-powered electroluminescence system for long-term cold-resistant displays. Nano Energy 98, 107309 (2022). https://doi.org/10.1016/j.nanoen.2022.107309
  46. J. Liu, Q. Chen, Q. Liu, B. Zhao, S. Ling et al., Intelligent silk fibroin ionotronic skin for temperature sensing. Adv. Mater. Technol. 5, 2000430 (2020). https://doi.org/10.1002/admt.202000430
  47. R. Long, C.Y. Hui, Fracture toughness of hydrogels: Measurement and interpretation. Soft Matter 12, 8069–8086 (2016). https://doi.org/10.1039/c6sm01694d
  48. Z. Chen, T. Zhang, C.-T. Chen, S. Yang, Z. Lv et al., Mechanically and electrically biocompatible hydrogel ionotronic fibers for fabricating structurally stable implants and enabling noncontact physioelectrical modulation. Mater. Horiz. 9, 1735–1749 (2022). https://doi.org/10.1039/D2MH00296E
  49. S. Lin, J. Liu, X. Liu, X. Zhao, Muscle-like fatigue-resistant hydrogels by mechanical training. Proc. Natl. Acad. Sci. USA 116, 10244–10249 (2019). https://doi.org/10.1073/pnas.1903019116
  50. F.H. Silver, J.W. Freeman, D. DeVore, Viscoelastic properties of human skin and processed dermis. Skin Res. Technol. 7, 18–23 (2001). https://doi.org/10.1034/j.1600-0846.2001.007001018.x
  51. J. Ankersen, A.E. Birkbeck, R.D. Thomson, P. Vanezis, Puncture resistance and tensile strength of skin simulants. Proc. Inst. Mech. Eng. Part H 213, 493–501 (1999). https://doi.org/10.1243/0954411991535103
  52. H. Oxlund, J. Manschot, A. Viidik, The role of elastin in the mechanical properties of skin. J. Biomech. 21, 213–218 (1988). https://doi.org/10.1016/0021-9290(88)90172-8
  53. J. Liu, J. Wang, Z. Zhang, F. Molina-Lopez, G.-J.N. Wang et al., Fully stretchable active-matrix organic light-emitting electrochemical cell array. Nat. Commun. 11, 3362 (2020). https://doi.org/10.1038/s41467-020-17084-w
  54. C.-C. Jao, J.-R. Chang, C.-Y. Ya, W.-C. Chen, C.-J. Cho et al., Novel stretchable light-emitting diodes based on conjugated-rod block elastic-coil copolymers. Polym. Int. 70, 426–431 (2021). https://doi.org/10.1002/pi.6023
  55. J.-H. Kim, J.-W. Park, Intrinsically stretchable organic light-emitting diodes. Sci. Adv. 7, eabd9715 (2021). https://doi.org/10.1126/sciadv.abd9715
  56. J. Liang, L. Li, K. Tong, Z. Ren, W. Hu et al., Silver nanowire percolation network soldered with graphene oxide at room temperature and its application for fully stretchable polymer light-emitting diodes. ACS Nano 8, 1590–1600 (2014). https://doi.org/10.1021/nn405887k
  57. M.S. White, M. Kaltenbrunner, E.D. Głowacki, K. Gutnichenko, G. Kettlgruber et al., Ultrathin, highly flexible and stretchable pleds. Nat. Photon. 7, 811–816 (2013). https://doi.org/10.1038/nphoton.2013.188
  58. T. Sekitani, H. Nakajima, H. Maeda, T. Fukushima, T. Aida et al., Stretchable active-matrix organic light-emitting diode display using printable elastic conductors. Nat. Mater. 8, 494–499 (2009). https://doi.org/10.1038/nmat2459
  59. P. Lee, J. Lee, H. Lee, J. Yeo, S. Hong et al., Highly stretchable and highly conductive metal electrode by very long metal nanowire percolation network. Adv. Mater. 24, 3326–3332 (2012). https://doi.org/10.1002/adma.201200359
  60. Y. Lin, W. Yuan, C. Ding, S. Chen, W. Su et al., Facile and efficient patterning method for silver nanowires and its application to stretchable electroluminescent displays. ACS Appl. Mater. Interfaces 12, 24074–24085 (2020). https://doi.org/10.1021/acsami.9b21755
  61. B.S. Kim, H. Kwon, H.J. Kwon, J.B. Pyo, J. Oh et al., Buckling instability control of 1d nanowire networks for a large-area stretchable and transparent electrode. Adv. Funct. Mater. 30, 1910214 (2020). https://doi.org/10.1002/adfm.201910214
  62. E. Ducrot, Y. Chen, M. Bulters, R.P. Sijbesma, C. Creton, Toughening elastomers with sacrificial bonds and watching them break. Science 344, 186–189 (2014). https://doi.org/10.1126/science.1248494
  63. J.-Y. Sun, X. Zhao, W.R.K. Illeperuma, O. Chaudhuri, K.H. Oh et al., Highly stretchable and tough hydrogels. Nature 489, 133–136 (2012). https://doi.org/10.1038/nature11409
  64. G.D. Genesky, C. Cohen, Toughness and fracture energy of pdms bimodal and trimodal networks with widely separated precursor molar masses. Polymer 51, 4152–4159 (2010). https://doi.org/10.1016/j.polymer.2010.06.054
  65. Y. Jia, Z. Zhou, H. Jiang, Z. Liu, Characterization of fracture toughness and damage zone of double network hydrogels. J. Mech. Phys. Solids 169, 105090 (2022). https://doi.org/10.1016/j.jmps.2022.105090
  66. E. Zhang, R. Bai, X.P. Morelle, Z. Suo, Fatigue fracture of nearly elastic hydrogels. Soft Matter 14, 3563–3571 (2018). https://doi.org/10.1039/C8SM00460A
  67. R. Bai, Q. Yang, J. Tang, X.P. Morelle, J. Vlassak et al., Fatigue fracture of tough hydrogels. Extreme Mech. Lett. 15, 91–96 (2017). https://doi.org/10.1016/j.eml.2017.07.002
  68. J. Wang, P.S. Lee, Progress and prospects in stretchable electroluminescent devices. Nanophotonics 6, 435–451 (2017). https://doi.org/10.1515/nanoph-2016-0002
  69. C. Keplinger, J.-Y. Sun, C.C. Foo, P. Rothemund, G.M. Whitesides et al., Stretchable, transparent, ionic conductors. Science 341, 984–987 (2013). https://doi.org/10.1126/science.1240228
  70. D. Vij, Handbook of Electroluminescent Materials (CRC Press, London, 2004)
  71. K. He, X. Zhang, S. Ren, J. Sun, presented at Computer Vision–ECCV 2016: 14th European Conference, Identity mappings in deep residual networks, Amsterdam, The Netherlands, October, 2016.
  72. A. Srivastava, S. Jain, R. Miranda, S. Patil, S. Pandya et al., Deep learning based respiratory sound analysis for detection of chronic obstructive pulmonary disease. Peer. J. Comput. Sci. 7, e369 (2021). https://doi.org/10.7717/peerj-cs.369
  73. M. Vatankhah-Varnosfaderani, A.N. Keith, Y. Cong, H. Liang, M. Rosenthal et al., Chameleon-like elastomers with molecularly encoded strain-adaptive stiffening and coloration. Science 359, 1509–1513 (2018). https://doi.org/10.1126/science.aar5308
  74. A. Miserez, J.C. Weaver, O. Chaudhuri, Biological materials and molecular biomimetics–filling up the empty soft materials space for tissue engineering applications. J. Mater. Chem. B 3, 13–24 (2015). https://doi.org/10.1039/C4TB01267D
  75. Z. Wang, C. Xiang, X. Yao, P. Le Floch, J. Mendez et al., Stretchable materials of high toughness and low hysteresis. Proc. Natl. Acad. Sci. USA 116, 5967–5972 (2019). https://doi.org/10.1073/pnas.1821420116
Read More
Author biographies are not available.
Statistic
Read Counter : 4664 Download : 3380

Most read articles by the same author(s)

1 2 3 4 > >>