IBM has adapted a convolutional neural network to work on a neuromorphic chip.

According to IBM, TrueNorth accuracy corresponds to the best modern systems of image and voice recognition, but at the same time, the system consumes less power and works faster. The company's research team is confident that combining convolutional networks with neuromorphic microcircuits will allow us to create more advanced smart cars and smartphones that correctly recognize a person’s voice command, even if they speak with their mouths full. Let's try to figure out what the advantages and disadvantages of TrueNorth, and where it has found application.



The human brain contains about 86 billion neurons - cells that connect to thousands of other neurons using processes - synapses. The neuron receives signals from many others, and when the stimulation reaches a certain threshold, it is “activated” by sending its own signal to the surrounding neurons. The brain learns, in particular, by adjusting strong connections. When a sequence of actions is repeated, for example, through practice, the accompanying synapses become stronger, and the lesson or skill “fits” into the network.



In the 1940s, scientists began to model neurons mathematically, and in the 50s - to create networks of neurons and computers. Artificial neurons and synapses are much simpler than in the brain, but act on the same principles. A set of simple units - neurons - are connected to others through “synapses” with their numerical values ​​depending on the values ​​of the units.



Convolutional neural networks (convolutional neural network, CNN) are a special type of network that has gained popularity in the past few years. CNN extracts important functions from incentives — usually photos. Take, for example, a photo of a dog. It can be represented as a layer of neurons, where the activation of a single neuron represents one pixel in the image. On the next layer, each neuron will take input from the first layer set and activate if it detects a particular pattern in that set, acting as a filter.



In subsequent layers, neurons will look for patterns in patterns, and so on. Within a single layer, filters may be sensitive to certain structures. First, to the borders of the figures, then to the paws, then to the dogs, until the net can determine the difference between the dog and the toaster.



Now such calculations are expensive. Considering that there are billions of neurons and trillions of synapses in the human brain, imitation of each is still impossible. Even simulating a small part of the brain will require millions of calculations for each input element, which requires enormous computational power. In the largest modern CNN there can be millions of neurons and billions of synapses, but no more.



The classical computational architecture of central processing units, designed to process one instruction at a time, is not suitable for the tasks that CNN imposes. Therefore, scientists turned to parallel computing, which can process several at once. Modern neural networks use graphics processors, which usually calculate graphics of video games and CAD. Due to the nature of the architecture and the similarity of mathematical calculations, video cards are suitable for deep learning.



But still, the hardware copes with deep learning not as effectively as the brain, which can drive a car and at the same time talk about the future of autonomous vehicles, while consuming less energy than a light bulb.



In the 1980s, engineer Carver Mead coined the term “neuromorphic processors” to describe computer chips that work in a sense like the brain. His work laid the foundation for this area. And although the term “neuromorphic” is now applied to a wide range of solutions, they all try to repeat the mechanism of neural networks at the hardware level, avoiding the bottlenecks encountered by traditional processors.



Having seen the need for fast and efficient machine learning, the Department of Defense United States Promising Projects (better known by the abbreviation DARPA) has been actively financing the IBM HRL Laboratories corporate laboratory since 2008 to develop neuromorphic machines that can be easily scaled.

Truenorth

In 2014, IBM introduced its TrueNorth chip on the cover of Science magazine. Since then, the company has been developing systems based on TrueNorth with financial support from the US Department of Energy, the Air Force and the Army. One such chip contains a million "neurons", each of which is represented by a group of digital transistors, and 256 million "synapses" —wired connections between the chips.



Neuromorphic architectures become more efficient compared to conventional chips due to two functions. First, such a chip, like the brain, communicates through “flashes” - one-dimensional packets of information sent from one neuron to descending neurons. The signals are simple (there is a flash or not) and are transmitted only occasionally when a neuron transmits a packet. Secondly, as well as in the brain, processing and memory are located in neurons and synapses. On a traditional computer, the data processing unit continuously retrieves information from individual memory areas, performs operations, and then returns new information to memory. This leads to many slow and energy-consuming activities.



The TrueNorth system is quite flexible, since it can be programmed to implement networks of various sizes and shapes and be scaled by “breaking down” several chips. In their research, the IBM team used a neuromorphic chip to identify people, bicycles, and cars in a video shot on the street. A comparative experiment showed that TrueNorth software running on a traditional microprocessor used 176 thousand times more energy for this task.



A key part of the IBM project was the creation of not only a chip, but also software. The company has created a simulator, a new programming language and a library of algorithms and applications. The company then provided these tools to more than 160 researchers in academic, national, and corporate laboratories. TrueNorth design was completed in 2011, and the convolutional neural network revolution occurred in 2012 as part of the ImageNet Challenge competition . Therefore, some people began to wonder if TrueNorth chips could handle these networks.



CNN uses the backpropagation method. Each time a network fails, the difference between its guess and the correct answer is calculated. The backpropagation error algorithm examines each neuron in the final layer and calculates how much a change in the output of this neuron will reduce the overall error. It then returns to the previous neurons and calculates how much the change in the strength of each incoming synapse will reduce the overall error.



We need to figure out whether to increase or decrease the synaptic power, so the algorithm slightly adjusts each weight in the right direction. Subsequently, the algorithm calculates a new error using the new weights and repeats the whole process. After many such steps, the error decreases in a process called gradient descent.



TrueNorth was initially considered incompatible with an error back-propagation algorithm, because gradient descent requires tiny adjustments in the weights and a vision of tiny improvements. TrueNorth maximizes its effectiveness using only three different weights: -1, 0 and 1, and the output from the neuron is 0 or 1. There are no gradients, only discrete steps.



One of the key achievements of the team was a series of methods for executing an error back-propagation algorithm with impulse neural networks. The researchers solved this problem by training the software model of the chip, programmed to use hardware approximation that is compatible with gradient descent.



Another key development was the mapping of CNN with multiple connections to neurons on a chip, which involves only 256 connections per neuron. This was achieved by assigning certain pairs of neurons that were triggered simultaneously, which were combined into one neuron through inputs and outputs.



Despite the fairly high performance of TrueNorth, it was created without taking into account the features of deep neural networks and CNN, so it has drawbacks compared to other systems. For example, to make a network of 30 thousand neurons work, you need 8 chips (8 million neurons). In addition, TrueNorth is a fully digital chip, while some have analog components. Their work is more unpredictable, but still more effective. And although each TrueNorth chip is divided into 4096 "cores" that work in parallel, the 256 neurons in each core are updated only sequentially and one at a time.



Sequential processing of neurons in the TrueNorth core may create a bottleneck, but it also provides regularity. This means that the behavior of the chip can be modeled with high accuracy on desktop computers. At the same time, the chip is universal - it can support many different types of networks, and the current goal of the chip maker team under the leadership of IBM's chief brain scientist for brain-like calculations, Dharmendra Modha, is to deploy several different networks working together to achieve composition.

Future plans

In addition to achieving composition, the Modha team is committed to exploring various learning methods. Also, scientists note that the methods described in their work can be applied to neuromorphic chips other than TrueNorth. Moreover, in addition to new teaching methods, the team thinks about more radical achievements. According to the 2015 US Department of Energy report on neuromorphic computing, about 5–15% of global energy is currently consumed in some form of data processing and transmission. At the same time, the Department wants to improve the speed, efficiency and resiliency of networks. This report pushed the IBM team to develop materials with new physical properties.



The global goal is to replace giant data centers with chips in smartphones, homes and cars that can “think” for themselves: make conversations, make scientific and medical discoveries, control cars, robots or prosthetic limbs. Ideally, these chips can achieve even greater success, for example, to solve the problem of world hunger.



Several research laboratories are already actively using TrueNorth. In August 2016, Samsung demonstrated a system that uses a video stream to create a 3D map of the environment in three dimensions at a speed of 2000 frames per second, consuming a third of a watt. The company used this system to control television with hand gestures.



The Lawrence Livermore National Laboratory has a board of 16 TrueNorth chips, which is used to enhance cybersecurity and ensure the safety of US nuclear weapons. The Air Force Research Laboratory, which uses TrueNorth to provide autonomous navigation for unmanned aerial vehicles, recently announced plans to test an array of 64 chips.