Modern medicine is powerless against many eye diseases. Some of them lead to complete blindness and no technology can again give a person vision.
Except one.
In the 60s of the 20th century, American neurophysiologist Paul Bach-u-Rita invented sensory replacement technology. Based on the property of brain neuroplasticity, which allows certain areas of the brain to change under the influence of experience, as well as restore lost connections after damage or develop new connections, he suggested that one can learn to replace one way of perceiving another.
Paul Bach-u-Rita created the BrainPort device, which with the help of electrodes transforms the signal from the video camera into electrical pulses according to a certain algorithm. The electrodes are attached to the language and a person who has learned the algorithm can navigate in the surrounding area without using ordinary vision. Due to brain neuroplasticity, after a few days in the blind using BrainPort, visual information begins to reach the visual cortex. They are literally starting to see.
But touch is not the most perfect way of perception, accessible to man, and even more so - not the most convenient and accurate. If we take as a basis another one that uses a larger number of brain neurons, then the substitution will be more accurate and high-quality. A blind person will see better.
In the 1990s, Dutch engineer Peter Meyer suggested using hearing rather than touch for touch substitution. He developed the Sound Vision algorithm - a model for converting an image into sounds of different volume and pitch, capable of conveying its meaning. The video image was divided into frames once a second, the frame was converted to monochrome, and then each pixel in the image was voiced by a certain tone and volume - and so once a second.
His plan was a success. The algorithm worked. And people using Sound Vision were able to see again. But to realize the technical embodiment of the idea, using modern technologies and convenient design solutions, turned out to be a difficult task.
Russian neurophysiologist and researcher Igor Trapeznikov became interested in sensory replacement technology many years ago. His curiosity was not an idle hobby. Due to astigmatism, Igor began to lose his sight and the problems of visually impaired and blind people loomed on the horizon with his own misfortune.
“Once my elder brother brought a microscope from school and opened for me an amazing little world, invisible to the naked eye. I thought then - since I could see all this with my poor eyesight with a microscope, then you can surely create some kind of technical tool that will also help me to see the ordinary world surrounding me well, ”says Igor Trapeznikov. “Everyone around believed that a person who has lost his vision will never be able to restore it. Blindness cannot be cured - one of the stereotypes of our society that I have been trying to break for many years. ”
In 2008, Igor learned from colleagues in the RTD social movement about Peter Meyer's Sound Vision algorithm and became interested in the opportunity to implement a project in Russia. Igor contacted the author and so friendship was born, which subsequently gives hope to blind people. Having studied the Dutchman’s algorithm, Igor realized that it’s possible to create a device that can be worn on the head that translates the image into sound according to the principle proposed by Peter Meyer. So began the story of vOICe vision.
The figure shows the principles of conversion for a simple example (image 8 Ă— 8 pixels, three gray halftones).
The mechanism of Igor Trapeznikov’s device is entirely tuned to the Peter Meyer algorithm. The data of each pixel is converted into a matrix. Vertical position - in frequency, brightness in amplitude. A pixel in a higher position corresponds to a higher frequency sound. The greater the brightness of a pixel represented by a gray tone, the greater the amplitude (“volume”) of the corresponding generator. The sums of signals for one column are superimposed on each other. Then each subsequent column is converted to sound. This procedure continues until the rightmost column is converted to sound. For the next frame, the conversion procedure is repeated.
“The idea arose 10 years ago. During the first experiments, we fastened the cameras to ordinary caps, and used laptops in backpacks for data processing. Active work began in 2015, when our team, with the support of RTD, managed to raise the right amount of funds thanks to crowdfunding. With this money, the first prototypes of vOICe vision were made - glasses with an integrated video camera and bone conduction headphones, ”recalls Igor.
It was then that in 2015 Vadim Artsev, the chief tester of the device, joined the team. He became the first blind person to learn how to use Sound Vision effectively. Today Vadim applies the development of Igor Trapeznikov every day and depends on it just like other people depend on their eyes.
“In the beginning it was unusual and hard. The sounds generated by vOICe vision seemed fantastic and didn’t look like anything, ”Vadim admits four years later. “I had to study in short classes, which took no more than 10 minutes, and with prolonged use of the device, my head began to feel dizzy. But this was only due to the fact that I did not understand the meaning of all these sounds. However, over time, I studied Peter Meyer's algorithm and was able to sort everything I hear. Now I can easily distinguish your silhouette and can communicate at least all day, without being distracted by the sounds of vOICe vision and at the same time guided by them in space. This is a matter of habit and experience. It is hard to listen to Chinese for a long time without knowing it. But if you are Chinese, then you will easily speak it yourself every day. ”
Igor Trapeznikov compares Peter Meyer's algorithm, encoded in the vOICe vision soundtrack, with a foreign language. Each day of practice only improves their skills. And precisely for this reason, Sound Vision outperforms other means of rehabilitation for the blind, even the most expensive and modern.
“Bionic eye implants, which are often presented in the media field, as the most radical and effective solution for the blind, again allowing you to see, are inferior to Sound Vision in all respects. Firstly, image quality is much lower. Secondly, the installation of bionic vision requires complex surgical intervention. And thirdly, the nerve tissue, to which the artificial eyes are connected, eventually heals and overgrows. As a result, bionic contact worsens and vision is lost again over time. When using vOICe vision, the skill only gets better every day and the surrounding world is drawn to a greater extent. This is not to mention the price. VOICe vision glasses along with training will cost ten times cheaper, ”explains Igor Trapeznikov.
His words confirm the results of competitions in which rehabilitation facilities compete and people with disabilities using them. Last year, Neurotlon - the name of this competition - was held in Samara. Pilots with bionic vision, an ultrasonic cane and Sonic Vision converged on one obstacle course. vOICe vision was represented by Vadim Artsev. With the device of Igor Trapeznikov, he was twice as fast as the other participants went through the track without making a single mistake.
The main problem of Sound Vision lies in its main advantage. vOICe vision offers to see the world in another language, using the algorithm created by Peter Meyer. And every language must first be learned. Therefore, for an unprepared person, Sound Vision is nothing more than an unfamiliar set of sounds. To teach blind people to understand it, you must first build a teaching methodology.
In the vOICe vision team, this task fell on the shoulders of Svetlana Lebedeva, a clinical psychologist, research associate at the Institute of Biomedical Medicine RAS. Svetlana was able to study the Peter Meyer algorithm and write a training program, develop tasks for the online training platform and organize the work of the vOICe vision Training Center. Today she successfully teaches Sound Vision for blind people.
“Learning a new way of perceiving the world is always a challenge. This takes time and effort - a person, like a child, goes through the stages of diffuse perception, syncreta, preconception and, finally, concepts - i.e. goes through all stages of the formation of ideas, which will be based on a new way of perception. Using the feedback mechanism, he first begins to isolate individual objects from the surrounding world from the sound, then he notes the differences between the sound patterns and shapes of different objects, and only then begins to “guess” whether the sound of an unfamiliar object belongs to a certain category of things. This is a very interesting process for specialists - teaching Sound Vision resembles the classical experiments of Vygotsky and Piaget and asks fundamental questions in the field of studying human perception in adulthood. But the students, of course, feel the development of a new perception is more prosaic - like learning any other complex skill during the usual, working rhythm of life. These are always questions of internal resources and personal narrative, ”says Svetlana Lebedeva.
Thanks to the developed training program, students are now provided with all the necessary external resources. Particular attention is paid to issues of motivation.
“Our training begins with a careful immersion in the basics of Sound Vision. Then, using the development mechanisms of any of our main ways of perceiving, a person learns to navigate using the world of new sounds. In this he is helped by vOICe vision trainers - experienced psychologists who know how to support a person and direct the course of his training. My colleague Daria Shibankova helps me a lot in this. We have everything for comprehensive training - consultations with trainers and psychologists, homework, our virtual textbook platform, support for each other users themselves. Last summer, we held two meetings of people with Sound Vision skills. I hope this will be a good tradition - all discoverers want to be in the company of like-minded people with similar experiences. Sound Vision works, it’s small business to popularize it, ”says Svetlana Lebedeva.
Nevertheless, the most important question remains - what exactly gives vOICe vision to a blind person? What does he get? What does he spend his money on? As a man with a bionic eye sees, there is no secret to anyone. The "picture" of artificial vision was demonstrated many times in news releases. You can “take a look” at the world with a bionic eye using an ophthalmological VR-simulator developed by the Sensor-Tech Laboratory and published in open access.
But how to evaluate the effectiveness of Sound Vision without mastering the Peter Meyer algorithm language?
“This question can be answered in two ways - from the perspective of research and from the perspective of science,” says Igor Trapeznikov. “The first is simpler. Interview those who have already mastered Sound Vision. The second involves the study of the brain using an encephalogram and magnetic resonance imaging. Such studies have already been conducted. A little, but they were. And the scientific approach indicates that with prolonged use of Sound Vision in the brain, those zones that are responsible precisely for visual sensations are activated. A person feels the picture. ”
To understand the sight of how this works, Igor suggests conducting such an experiment. During a conversation with a person, carefully look at his face, and then close his eyes sharply. The image of the interlocutor is preserved in memory and the imagination instantly draws it. According to the creator of vOICe vision, subjectively Sound Vision creates the feeling of a picture in a similar way. This applies to the depth of space, the silhouettes of objects and objects, as well as dark and light colors.
Everyone can buy vOICe vision glasses today, now small-scale production has been launched. The developer is trying to work with key organizations that support the blind. Indeed, in Russian realities, not many people with disabilities have the opportunity to independently purchase a rehabilitation tool.
“We want to make vOICe vision accessible to all Russian visually impaired people. But in Russia this will have a long way to go. In order to register our glasses as a medical device that helps to rehabilitate the blind, it is necessary to conduct a series of clinical trials proving, for example, that vOICe vision is not harmful to health - from the point of view of ophthalmologists and other doctors, ”explains Igor Trapeznikov.
And this is despite the fact that in Russia Sound Vision has already proved itself. The experience of using and testing vOICe vision is positive and demonstrates high efficiency. Last year, the project became a finalist of the Open Inquiries contest, through the Moscow Innovation Agency, and this year it became its winner. The device was successfully tested in a boarding house for the blind in the village of Malino, Moscow Region. Based on this practice, expert reviews were collected recommending the blind to learn Sound Vision skills.
“The mission of vOICe vision is to make Sound Vision available to all people who have lost their normal vision, will lose it in the future, or were born blind. In the 19th century, Louis Braille spread the tactile alphabet throughout the world. And today is the standard for all blind people. We want to repeat this way, do the same with vOICe vision, because this is the only device in the world that allows you to literally see with sound. Functionally, those who have mastered Sound Vision are transformed from blind to visually impaired. As soon as this opportunity appears for all blind people and vOICe vision becomes available to each of them, I will consider our mission accomplished, ”Igor Trapeznikov sums up.
But the success of the project depends not only on Igor and his team. In Russia, as in other countries, the rehabilitation industry needs state support. And yet everyone can help the project, at least in the media space. After all, information is the main weapon of the 21st century. A short story to friends, colleagues or a banal repost on social networks can change the future of vOICe vision ... And the future of those who devoted their lives to it, who created an alternative to eternal darkness and gave people hope.
Peter Meyer created the algorithm of hope, Igor Trapeznikov embodied it in a technical device, and Svetlana Lebedeva, teaching, instills this hope in the hearts of those who do not want to live in the dark.
Ivan Chimbulatov