Classification of Spatial Temporal Patterns Based on Neuromorphic Networks
Keywords:
spatiotemporal patterns, classification, neural networks, support vectors, Riemannian geometry, neuromorphic networks, neurointerface, electroencephalography, imaginary movements, non-contact controlAbstract
This work is devoted to the problems of developing neuromorphic classifiers of spatiotemporal patterns, as well as their application in neurointerfaces. Classifiers of spatiotemporal patterns based on neural networks, support vector machines, deep neural networks, and Riemannian geometry are considered. A comparative study of these classifiers is carried out in the plane of the accuracy of multiclass recognition of electroencephalographic signals showing time-dependent bioelectrical activity in different areas of the brain during the imagination of different movements. It is shown that such classifiers can provide an accuracy of 60-80% when recognizing from two to four classes of imaginary movements. A new type of classifier based on a neuromorphic network, based on the biosimilar neurons built on the Izhikevich model, is proposed. The network processes input spike sequences and generates pulse streams of different frequencies at the outputs. The network is trained using the Supervised STDP algorithm based on labeled information containing examples of the correct recognition of the required pattern classes. The recognized pattern class is determined by the maximum frequency of the output sequence. The neuromorphic classifier showed an average classification accuracy of 90% for 4 classes of imaginary commands and a maximum of 95%. By modeling the robot control task in the virtual environment it is shown that such accuracy is sufficient for the effective use of the classifier as part of a non-invasive brain-computer interface for non-contact control of robotic devices.
References
1. Lisovsky A.L. Application of neural network technologies for management development of systems. Strategic decisions and risk management. 2020. vol. 11. no. 4. pp. 378–389. DOI: 10.17747/2618-947X-923. (In Russ.).
2. Blagoveshhenskij V.G., Blagoveshhenskij I.G., Blagoveshhenskaja M.M., Adnodvorcev A.M., Golovin V.V. [Control of technological product processes for confection wares using neural network regulator] Upravlenie tehnologicheskimi processami proizvodstva konditerskih izdelij s ispol'zovaniem nejrosetevogo reguljatora [Proceedings of the All-Russian Scientific and Technical Committee «Informatization and automation in the food industry»]. Kursk: Published: Books of Universities, 2022. pp. 78–83.
3. Uliev A.D., Rozaliev V.L., Zaboleeva-Zotova A.V., Orlova Y.A. [An Intelligent Video Surveillance System for Human Behavior]. Iskusstvennyj intellekt i prinjatie reshenij – Artificial intelligence and decision making. 2020. no. 4. pp. 21–32. DOI: 10.14357/20718594200403. (In Russ.).
4. Bohush R.P., Zakharava I.Y. Person tracking algorithm based on convolutional neural network for indoor video surveillance. Computer Optics. 2020. vol. 44. no. 1. pp. 109–116. DOI: 10.18287/2412-6179-CO-565. (In Russ.).
5. Brunner C., Birbaumer N., Blankertz B., Guger C., Kubler A., Mattia D., del R. Millan J., Miralles F., Nijholt A., Opisso E., Ramsey N., Salomon P., Muller-Putz G.R. BNCI Horizon 2020: towards a roadmap for the BCI community. Brain-Computer Interfaces. 2015. vol. 2. no. 1. pp. 1–10. DOI: 10.1080/2326263X.2015.1008956.
6. Sharmila A. Hybrid control approaches for hands-free high level human–computer interface-a review. Journal of Medical Engineering & Technology. 2021. vol. 45. no. 1. pp. 6–13.
7. Diez P. Smart Wheelchairs and BCI. Mobile Assistive Technologies. Academic Press, 2018. 492 p.
8. Kagirov I.A., Karpov A.A., Kipyatkova I.S., Klyuzhev K.S., Kudryavtsev A.I., Kudryavtsev I.A., Ryumin D.A. [Intellectual Interface to Control a Robotic Medical Exoskeleton of the Lower Limbs «Remotion»]. Aviakosmicheskaja i jekologicheskaja medicina – Aviacosmos and ecological medicine. 2019. vol. 53. no. 5. pp. 92–98. (In Russ.).
9. Li Z., Li B., Luo W., Cao J. Design and Implementation of P300 Brain-Controlled Wheelchair with a Developed Wireless DA Converter. International journal of computers & technology. 2023. vol. 23. pp. 93–104. DOI: 10.24297/ijct.v23i.9485.
10. Yakovlev L., Kaplan A., Sirov N. Gortz, N. BCI-Controlled Motor Imagery Training Can Improve Performance in e-Sports. HCI International 2020-Posters: 22nd International Conference. 2020. pp. 581–586. DOI: 10.1007/978-3-030-50726-8_76.
11. Zhu H.Y., Hieu N.Q., Hoang D.T., Nguyen D.N., Lin C.-T. A Human-Centric Metaverse Enabled by Brain-Computer Interface: A Survey. arXiv preprint arXiv:2309.01848. 2023.
12. Stankevich L., Sonkin K., Nagornova Z., Khomenko J., Shemyakina N. Classification of Electroencephalographic Patterns of Imaginary One-hand Finger Movements for Brain-Computer Interface Development. SPIIRAS Proceedings. 2015. vol. 3(40). pp. 163–182. DOI: 10.15622/sp.40.11. (In Russ.).
13. Stankevich L.A., Sonkin K.M., Shemyakina N.V., Nagornova Zh.V., Khomenko Ju.G., Perets D.S., Koval A.V. EEG Pattern Decoding of Rhythmic Individual Finger Imaginary Movements of one Hand. Human Physiology. 2016. vol. 42. no. 1. pp. 32–42.
14. Schirrmeister R.T., Springenberg J.T., Fiederer L.D.J., Glasstetter M., Eggensperger K., Tangermann M., Hutter F., Burgard W., Ball T. Deep learning with convolutional neural networks for brain mapping and decoding of movement-related information from the human EEG. arXiv:1703.05051v5. 2018.
15. Congedo M., Barachant A., Bhatia R. Riemannian geometry for EEG-based brain-computer interfaces: a primer and a review // Brain-Computer Interfaces. 2017. vol. 4. no. 3. pp. 155–174. DOI: 10.1080/2326263X.2017.1297192.
16. Weerasinghe M.M., Espinosa-Ramos J.I., Wang G.Y., Parry D. Incorporating Structural Plasticity Approaches in Spiking Neural Networks for EEG Modelling. IEEE Access. 2021. vol. 10. pp. 117338–117348. DOI: 10.1109/ACCESS.2021.3099492.
17. Kapralov N., Nagornova Z., Shemyakina N. Classification Methods for EEG Patterns of Imaginary Movements. Informatics and Automation. 2021. vol. 20. no. 1. pp. 94–132. DOI: 10.15622/ia.2021.20.1.4. (In Russ.).
18. Gerstner W., Kistler W.M., Naud R., Paninski L. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge: Cambridge University Press, 2014. 578 p.
19. Izhikevich E.M. Simple model of spiking neurons. IEEE Trans. Neural Networks. 2003. vol. 14. no. 6. pp. 1569–1572. DOI: 10.1109/TNN.2003.820440.
20. Cui Y., Ahmad S., Hawkins J. The HTM spatial pooler-a neocortical algorithm for online sparse distributed coding. Frontiers in computational neuroscience. 2017. vol. 11. DOI: 10.3389/fncom.2017.00111.
21. Auge D., Hille J., Mueller E., Knoll A. A Survey of Encoding Techniques for Signal Processing in Spiking Neural Networks. Neural Processing Letters. 2021. vol. 53. no. 6. pp. 4693–4710. DOI: 10.1007/s11063-021-10562-2.
22. Liu F., Zhao W., Chen Y., Wang Z., Yang T., Jiang L. SSTDP: Supervised Spike Timing Dependent Plasticity for Efficient Spiking Neural Network Training. Frontiers in Neuroscience. 2021. vol. 15. DOI: 10.3389/fnins.2021.756876.
23. Stankevich L.A. Gundelakh F.V. Robot control with use brain-computer interface. Robotics and technical cybernetics. 2017. no. 2(15). pp. 52–56. (In Russ.).
24. Gundelakh F., Stankevich L., Sonkin K., Nagornova G., Shemyakina N. Application of Brain-computer Interfaces in Assistive Technologies. SPIIRAS Proceedings. 2020. vol. 19. no. 2. pp. 277–301. (In Russ.).
25. Gundelakh F., Stankevich L., Kapralov N.V, Ekimovski J.V. Cyber-Physical System Control Based on Brain-Computer Interfaces. Springer International Publishing, 2020. pp. 458–469.
26. Tutorial: ROS integration overview. Available at: https://classic.gazebosim.org/tutorials?tut=ros_overview (accessed: 09.12.2023).
27. Ackerman E. Latest Version of Gazebo Simulator Makes It Easier Than Ever to Not Build a Robot. IEEE Spectrum. IEEE. 2016.
2. Blagoveshhenskij V.G., Blagoveshhenskij I.G., Blagoveshhenskaja M.M., Adnodvorcev A.M., Golovin V.V. [Control of technological product processes for confection wares using neural network regulator] Upravlenie tehnologicheskimi processami proizvodstva konditerskih izdelij s ispol'zovaniem nejrosetevogo reguljatora [Proceedings of the All-Russian Scientific and Technical Committee «Informatization and automation in the food industry»]. Kursk: Published: Books of Universities, 2022. pp. 78–83.
3. Uliev A.D., Rozaliev V.L., Zaboleeva-Zotova A.V., Orlova Y.A. [An Intelligent Video Surveillance System for Human Behavior]. Iskusstvennyj intellekt i prinjatie reshenij – Artificial intelligence and decision making. 2020. no. 4. pp. 21–32. DOI: 10.14357/20718594200403. (In Russ.).
4. Bohush R.P., Zakharava I.Y. Person tracking algorithm based on convolutional neural network for indoor video surveillance. Computer Optics. 2020. vol. 44. no. 1. pp. 109–116. DOI: 10.18287/2412-6179-CO-565. (In Russ.).
5. Brunner C., Birbaumer N., Blankertz B., Guger C., Kubler A., Mattia D., del R. Millan J., Miralles F., Nijholt A., Opisso E., Ramsey N., Salomon P., Muller-Putz G.R. BNCI Horizon 2020: towards a roadmap for the BCI community. Brain-Computer Interfaces. 2015. vol. 2. no. 1. pp. 1–10. DOI: 10.1080/2326263X.2015.1008956.
6. Sharmila A. Hybrid control approaches for hands-free high level human–computer interface-a review. Journal of Medical Engineering & Technology. 2021. vol. 45. no. 1. pp. 6–13.
7. Diez P. Smart Wheelchairs and BCI. Mobile Assistive Technologies. Academic Press, 2018. 492 p.
8. Kagirov I.A., Karpov A.A., Kipyatkova I.S., Klyuzhev K.S., Kudryavtsev A.I., Kudryavtsev I.A., Ryumin D.A. [Intellectual Interface to Control a Robotic Medical Exoskeleton of the Lower Limbs «Remotion»]. Aviakosmicheskaja i jekologicheskaja medicina – Aviacosmos and ecological medicine. 2019. vol. 53. no. 5. pp. 92–98. (In Russ.).
9. Li Z., Li B., Luo W., Cao J. Design and Implementation of P300 Brain-Controlled Wheelchair with a Developed Wireless DA Converter. International journal of computers & technology. 2023. vol. 23. pp. 93–104. DOI: 10.24297/ijct.v23i.9485.
10. Yakovlev L., Kaplan A., Sirov N. Gortz, N. BCI-Controlled Motor Imagery Training Can Improve Performance in e-Sports. HCI International 2020-Posters: 22nd International Conference. 2020. pp. 581–586. DOI: 10.1007/978-3-030-50726-8_76.
11. Zhu H.Y., Hieu N.Q., Hoang D.T., Nguyen D.N., Lin C.-T. A Human-Centric Metaverse Enabled by Brain-Computer Interface: A Survey. arXiv preprint arXiv:2309.01848. 2023.
12. Stankevich L., Sonkin K., Nagornova Z., Khomenko J., Shemyakina N. Classification of Electroencephalographic Patterns of Imaginary One-hand Finger Movements for Brain-Computer Interface Development. SPIIRAS Proceedings. 2015. vol. 3(40). pp. 163–182. DOI: 10.15622/sp.40.11. (In Russ.).
13. Stankevich L.A., Sonkin K.M., Shemyakina N.V., Nagornova Zh.V., Khomenko Ju.G., Perets D.S., Koval A.V. EEG Pattern Decoding of Rhythmic Individual Finger Imaginary Movements of one Hand. Human Physiology. 2016. vol. 42. no. 1. pp. 32–42.
14. Schirrmeister R.T., Springenberg J.T., Fiederer L.D.J., Glasstetter M., Eggensperger K., Tangermann M., Hutter F., Burgard W., Ball T. Deep learning with convolutional neural networks for brain mapping and decoding of movement-related information from the human EEG. arXiv:1703.05051v5. 2018.
15. Congedo M., Barachant A., Bhatia R. Riemannian geometry for EEG-based brain-computer interfaces: a primer and a review // Brain-Computer Interfaces. 2017. vol. 4. no. 3. pp. 155–174. DOI: 10.1080/2326263X.2017.1297192.
16. Weerasinghe M.M., Espinosa-Ramos J.I., Wang G.Y., Parry D. Incorporating Structural Plasticity Approaches in Spiking Neural Networks for EEG Modelling. IEEE Access. 2021. vol. 10. pp. 117338–117348. DOI: 10.1109/ACCESS.2021.3099492.
17. Kapralov N., Nagornova Z., Shemyakina N. Classification Methods for EEG Patterns of Imaginary Movements. Informatics and Automation. 2021. vol. 20. no. 1. pp. 94–132. DOI: 10.15622/ia.2021.20.1.4. (In Russ.).
18. Gerstner W., Kistler W.M., Naud R., Paninski L. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge: Cambridge University Press, 2014. 578 p.
19. Izhikevich E.M. Simple model of spiking neurons. IEEE Trans. Neural Networks. 2003. vol. 14. no. 6. pp. 1569–1572. DOI: 10.1109/TNN.2003.820440.
20. Cui Y., Ahmad S., Hawkins J. The HTM spatial pooler-a neocortical algorithm for online sparse distributed coding. Frontiers in computational neuroscience. 2017. vol. 11. DOI: 10.3389/fncom.2017.00111.
21. Auge D., Hille J., Mueller E., Knoll A. A Survey of Encoding Techniques for Signal Processing in Spiking Neural Networks. Neural Processing Letters. 2021. vol. 53. no. 6. pp. 4693–4710. DOI: 10.1007/s11063-021-10562-2.
22. Liu F., Zhao W., Chen Y., Wang Z., Yang T., Jiang L. SSTDP: Supervised Spike Timing Dependent Plasticity for Efficient Spiking Neural Network Training. Frontiers in Neuroscience. 2021. vol. 15. DOI: 10.3389/fnins.2021.756876.
23. Stankevich L.A. Gundelakh F.V. Robot control with use brain-computer interface. Robotics and technical cybernetics. 2017. no. 2(15). pp. 52–56. (In Russ.).
24. Gundelakh F., Stankevich L., Sonkin K., Nagornova G., Shemyakina N. Application of Brain-computer Interfaces in Assistive Technologies. SPIIRAS Proceedings. 2020. vol. 19. no. 2. pp. 277–301. (In Russ.).
25. Gundelakh F., Stankevich L., Kapralov N.V, Ekimovski J.V. Cyber-Physical System Control Based on Brain-Computer Interfaces. Springer International Publishing, 2020. pp. 458–469.
26. Tutorial: ROS integration overview. Available at: https://classic.gazebosim.org/tutorials?tut=ros_overview (accessed: 09.12.2023).
27. Ackerman E. Latest Version of Gazebo Simulator Makes It Easier Than Ever to Not Build a Robot. IEEE Spectrum. IEEE. 2016.
Published
2024-05-28
How to Cite
Gundelakh, F., & Stankevich, L. (2024). Classification of Spatial Temporal Patterns Based on Neuromorphic Networks. Informatics and Automation, 23(3), 886-908. https://doi.org/10.15622/ia.23.3.9
Section
Robotics, Automation and Control Systems
Copyright (c) Филипп Викторович Гунделах, Лев Александрович Станкевич
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms: Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).