The mathematical model reveals the secrets of vision

Mathematicians and neuroscientists created the first anatomically accurate model explaining how vision works

The great mystery of human vision is this: we perceive a rich image of the world around us, despite the fact that the visual system of our brain receives extremely little information about it. Most of what we “see” is actually what we imagine in our heads.



“A lot of what you think you see is actually coming up with,” said Lai-Sang Young , a mathematician at New York University. “You really don't see them.”



However, the brain, apparently, copes well with the task of inventing the visual world, since we usually do not encounter doors. Unfortunately, studying anatomy alone does not show us exactly how the brain creates these images - nothing more than a closer look at the car engine will allow you to reveal the laws of thermodynamics.



A new study suggests that the key to understanding lies in mathematics. Over the past few years, Young has been working in unexpected partnerships with university colleagues Robert Shapley , a neuroscientist, and Logan Chariker, a mathematician. They created a unified mathematical model that combines the results of many years of biological experiments, and explains how the brain produces complex visual reproductions of the world based on scanty visual information.



“The task of the theorist, as I see it, is that we take various facts and bring them into a consistent picture,” Young said. “The experimenters will not tell you how something works.”



Young and his colleagues built a model, including one basic element of vision at a time. They explained how the neurons of the visual cortex interact, recognizing objects and contrast changes, and now they are working to explain how the brain perceives the direction in which the objects are moving.



Their work is one of a kind. Previous attempts to model human vision were wishful thinking, describing the architecture of the visual cortex. The work of Young, Shapley and Chariker recognizes the complex, non-intuitive biology of the visual cortex, and tries to explain how the phenomenon of vision nevertheless arises.



“I think their model improves those results that are really based on true brain anatomy. They need a biologically correct or acceptable model, ”said Alessandra Angelucci , a neuroscientist at the University of Utah.

Layer upon layer

We are certain of some vision-related issues.



The eye works like a lens. It receives light from the outside world and projects a small-scale copy of the observed field of view onto the retina located on the back of the eye. The retina connects to the visual cortex, a part of the brain located at the back of the head.



However, the connection between the retina and the visual cortex is very weak. About 10 nerve cells connecting the retina and the visual cortex fall on each part of the field of view with a size of about a quarter of the full moon in the sky. They make up the lateral cranked body, LKT, the only way in which visual information passes from the outside world to the brain.



LKT cells are not just small - they are almost not capable of anything. LCT cells send an impulse to the visual cortex, detecting a change from darkness to light, or vice versa, in its tiny part of the visual field. And that’s it. The backlit world is bombarding the retina with data, but the brain has only some miserable signals for working from a tiny collection of LKT cells. Trying to see the world on the basis of such meager information is similar to trying to recreate Moby Dick on the basis of doodles on a napkin.



“You can imagine that the brain is taking a picture of what you are observing in sight,” Young said. “However, the brain does not take photographs, the retina does them, and the information transmitted from the retina to the visual cortex is scarce.”



And then the visual cortex begins to work. Although relatively few neurons connect the cortex and retina, the cortex itself is a dense cluster of nerve cells. For every 10 LKT neurons coming from the retina, there are 4,000 neurons only in the first, “input layer” of the visual cortex - and even more in the following. This discrepancy suggests that the brain is actively processing the small amount of visual data that it receives.



“The visual cortex has its own mind,” said Shapley.



For researchers such as Young, Shapley, and Chariker, the challenge is to decipher what is happening in this mind.

Eye loops

Nervous anatomy of vision is provocative. She looks like a little man lifting a huge weight, and requires an explanation - how does she manage to do so much by using so little?



Young, Shapley, and Chariker are not the first scientists to try to find the answer to this question using a mathematical model. But all the previous ones assumed that more information is transmitted between the retina and the cortex - such an assumption would facilitate an attempt to explain the reaction of the visual cortex to stimulation.



“People did not take seriously what came from biology as part of a computational model,” said Shapley.



Mathematicians have a long history of success in modeling variable phenomena, from moving billiard balls to the evolution of space-time. These are examples of “dynamic systems” - evolving over time according to fixed rules. The interactions of neurons that are activated in the brain are also an example of a dynamic system - albeit quite thin, one that is not easy to hang a certain set of rules on.



LKT cells send to the cortex a sequence of electrical pulses with a voltage of 1/10 volt and a duration of 1 ms, which triggers a cascade of neural interactions. Young said the rules governing these interactions are “infinitely more complex” than the rules governing the more familiar physical systems.





Lai Sang Young and Robert Shapley



Individual neurons receive signals simultaneously from hundreds of other neurons. Some of these signals encourage neuronal activation. Others are overwhelming. When receiving electrical impulses from these exciting and suppressing neurons, voltage fluctuation is observed on the membrane of the neuron in question. And it is activated only when this voltage (“membrane potential”) exceeds a certain threshold. And it’s almost impossible to predict when this will happen.



“If you look at the membrane potential of a single neuron, it will jump up and down,” Young said. “It is completely impossible to predict exactly when it is activated.”



Moreover, the real situation is even more complicated. Remember these hundreds of neurons connected to one of ours? Each of them receives signals from hundreds of other neurons. The visual cortex is a mishmash of interacting feedbacks coupled with feedbacks.



“The problem with all this is that we have too many moving parts. This complicates matters, ”said Shapley.



In early models of the visual cortex, this feature was ignored. It was assumed that the information goes in one direction - from the front of the eye to the retina, then to the cortex, until finally - voila! - on the other end will not appear an image, like a gadget that appears on a conveyor belt. These “direct propagation” models were easier to create, but they ignored the effects of the anatomy of the cortex - which suggested that feedback loops play a big role in what is happening.



“It’s very difficult to work with feedback loops because the information comes back and changes state all the time, comes back and influences you,” Young said. “Almost no model deals with this, but this happens all over the brain.”



In their original 2016 work , Young, Shapley, and Chariker decided to try to take these feedback loops seriously. The feedback loops of their model led to the appearance of something like a butterfly effect: small changes in the LCT signal were amplified when the signal passed through one loop after another, during the so-called “Recurrent excitation”, which led to large changes in the visual representation, which the model eventually formed.



Young, Shapley, and Chariker showed that their model, rich in feedback, was able to reproduce the orientation of the faces of the objects - horizontal, vertical, and all the others - based on small changes in weak incoming signals from the LCT.



“They showed that you can create all orientations in the visual world using only a small number of neurons connected to other neurons,” said Angelucci.



But vision is much more than just face detection, and the work of 2016 was only the beginning. The next difficulty was to include additional elements of vision in the model without losing the only one with which they had already dealt.



“If a model does something right, it should be able to do several different things,” Young said. “Your brain continues to remain unchanged, but it is capable of different things in different conditions.”

Swarm of visions

In laboratory experiments, the researchers presented primates with simple visual stimuli — black and white patterns in which the contrast or direction in which they appeared in the field of vision changed. Using electrodes connected to the visual cortex of primates, researchers tracked nerve impulses originating in response to stimuli. A good model should reproduce similar impulses in response to similar stimuli.



“We know that if we show the primate this picture, he will react in such a way,” Young said. “Based on this information, we are trying to analyze what is happening inside him.”



In 2018, three researchers published a second work in which they showed that the same model that is able to recognize faces can also reproduce the general picture of the activity of cortical impulses, known as the gamma rhythm (it looks like a swarm of fireflies sequentially lighting its flashlights).



Now specialists are studying their third work, which explains how the visual cortex perceives changes in contrast. The explanation mentions the mechanism by which exciting neurons enhance each other's activity, something like the growth of crowd excitement in dancing. Such processes are necessary so that the visual cortex can create full-fledged images based on scarce input data.



So far, Young, Shapley and Chariker are working on adding directional sensitivity to the model - which will explain how the visual cortex recreates the direction of movement of objects along the field of view. After that, they will begin to explain how the visual cortex recognizes time sequences in visual stimuli. For example, they want to understand why we perceive flashes of a flashing traffic light, but at the same time we do not see individual frames when watching a movie.



After that, they will have in their hands a simple model of activity that occurs in only one of the six layers of the visual cortex - in a layer in which the brain roughly outlines the basic contours of the visual impression. Their work does not apply to the other five layers, where more complex visual processing takes place. It also does not say anything about how the visual cortex recognizes colors, which occurs along a completely different, more complex nervous path.



“I think they still have a lot to do, but I do not deny that they did their best,” said Angelucci. “This is a difficult job and it takes time.”



Although their models are still far from revealing all the secrets of vision, this is a step in the right direction - this is the first model trying to decipher vision in a biologically plausible way.



“People have been portraying activities in this area for a very long time,” said Jonathan Victor , a neuroscientist at Cornell University. “The fact that these scientists were able to demonstrate this with the example of their model corresponding to biology is a real triumph.”