Intel proposed mathematical security rules for unmanned vehicles instead of learning neural network

Fig. 1. An unmanned vehicle in the center is not able to do anything to ensure guaranteed security. In the case of inadequate actions of a human driver from a nearby car, an accident is inevitable, and then a months-long investigation into the accident with the close attention of the press to the accident, where the "autopilot killed the man." No prior training of the neural network will help secure the machine in such a situation. The only solution to the problem is to foresee the rules of the road for robokars in order to rule out the occurrence of such a situation.



In recent years, automakers and IT companies are participating in the race, who will be the first to develop a fully unmanned vehicle capable of operating without human intervention in all situations (SAE Level 5). The mass distribution of such cars promises immense economic and social benefits for human society, among them - a drastic reduction in road mortality, a reduction in the number of disabled people, a more flexible and efficient transport system that is accessible to all.



It seems that the current automakers and IT companies that participate in the technological race, perceive an unmanned vehicle as a product - and think primarily about the development of the market, and not about cooperation with other automakers. Mobileye, recently bought by Intel, takes a different look at things. It reminds all participants of the "capitalist competition" that the robokar is not just a product, but an industry where joint rules should be developed. To this end, the company has developed the world's first mathematical framework for the guaranteed safety of unmanned vehicles (pdf) . The framework was presented at the World Knowledge Forum in Seoul (South Korea) by Professor Amnon Shashua, who is the CEO of Mobileye and senior vice president of Intel.



Naturally, Mobileye / Intel has its own interest. The development of world standards for cars can bring more profits than the release of these cars themselves.



For the unmanned vehicle industry to develop successfully, joint efforts are needed from automakers, technology companies and government regulators. Together, they must develop a common unified model of interaction for all. Mobileye has created a math framework for such a general model. It is described in detail in scientific work . The framework is based on two fundamental principles:

1. Proven security guarantees.
2. Economic scalability.

Mobileye believes that without a clear model describing these two critical parameters, all efforts to create autonomous vehicles will become meaningless, that is, robocar will turn into just an expensive scientific experiment.



Standard provable safety assurances are the minimum set of requirements that each unmanned vehicle must meet, and a description of how to comply with them.



Economic scalability is a requirement that is designed to ensure that the engineering solutions developed are really capable of being scaled to millions of cars, rather than slipping back into an academic niche, of interest only to scientific research. This has already happened in the history of the development of artificial intelligence and machine vision. Among Western researchers, there is the term "winter of AI" (winter of AI). Under it, they usually mean decades of almost complete inactivity that came after the failure of Artificial Intelligence research in the early 80s. The failure was inevitable, because the researchers initially set unattainable goals and inflated HYIP.



There is an opinion that the development of unmanned vehicles can follow the same scenario: the developers of this technology promise too bright prospects, which can turn into a frustration of society and the termination of financing the industry if developers can not produce a result that meets these expectations.



To prevent this from happening, you need to think through all the issues in advance when developing the framework, including economic scalability.



Existing designs do not meet precisely these two important criteria — standard provable security guarantees and requirements for economic scalability. Fortunately, the industry now understands possible problems - and we have a chance to foresee options for solving possible problems.



Mobileye experts are critical of how unmanned vehicle development engineers are now trying to solve security problems with statistical methods. Allegedly, it is enough to roll a certain (large) mileage for training a neural network - and this guarantees the safety of the car in almost any situation. But right away it should be noted that it is theoretically impossible to achieve absolute safety, because many accidents occur through the fault of other road users, and the subject himself has no opportunity to influence the situation (see Fig. 1 above for illustration). That is, in this way it is impossible to achieve the zero level of incidents, but only to reduce it to a socially acceptable level.



Mobileye gives this example. It is reasonable to assume that in order to reach a public consensus on the replacement of driven cars with drones, the death rate on the roads should be reduced by three orders of magnitude. Suppose that the probability of dying in an accident while driving a person is 10 ⁻⁶ per hour (this is a figure close to real). Then, for “guaranteed security”, we need to reduce this probability in unmanned vehicles to 10 ⁻⁹ per hour (this is also a real figure taken from the aviation industry: it corresponds to the probability of an involuntary separation of the aircraft wing from the fuselage during the flight).



So, in order to guarantee such a low probability of mortality of 10 ⁻⁹ per hour with statistical methods, it is obviously necessary to collect 10 ⁹ hours of experimental data, which corresponds to an impact of approximately 45 billion kilometers.



Moreover, in the case of a system of many agents that reacts with the outside world and living people (drivers and pedestrians), we cannot use virtual simulators, because an algorithm of actions has not yet been created (and is unlikely to be created in the future) and, accordingly, , simulator behavior unpredictable driver or pedestrian. That is, for any change in the control software, additional experimental data collection will be required - another 45 billion hours to match the statistical model.



In the end, security training on a neural network using the collected data will inevitably suffer from a lack of interpretability and explanability. If the autopilot knocks down a fatal pedestrian, you need to find the reason and explain why the incident happened, what needs to be fixed in the system so that this does not happen again. Unfortunately, the “black box” of the neural network does not provide a clear and understandable explanation in such situations.



Unmanned vehicles will have to share the road with people over the coming decades, so each model must take into account the unpredictable nature of people. In addition, the autopilot will have in the first decades to adapt to the driving style that people consider “normal”.



When developing a model of guaranteed safety, it should be borne in mind that any car at any time may experience mechanical damage or be exposed to external forces. Although it is impossible to fully envisage all the scenarios of a possible accident, it is possible to minimize the probability of such scenarios and the predicted death rate per hour of driving.



Mobileye proposes to implement a framework with a set of rules based on mathematical models. They will protect automakers themselves from public censure in case of fatal accidents involving unmanned vehicles - and such incidents are inevitable.



Mobileye developed a system called Responsibility-Sensitive Safety (RSS) . It ensures that, from a decision-making point of view, the autopilot system will never issue a command that could lead to a robokar causing an accident. To do this, the model introduced the concept of "safe state" (Safe State) and "cautious commands" (Cautious Commands), which ensure that the car does not go beyond the safe state.



The RSS system is different in that when it was developed, they abandoned the traditional approach with analysis of numerous conditions and intensive analysis of a large amount of data on the current situation - this is simply not feasible both in real road conditions and in the simulation. Instead, each autopilot action is tested in real time for compliance with simple mathematical formulas. The computing power of modern computers is quite enough to perform such calculations in real time. For example, in fig. Figure 2 shows the formula for calculating the safe distance to a vehicle in front of you with a live driver, if there is no wireless communication with the vehicle and coordination via the V2V inter-transport protocol.





Fig. 2. Formula for calculating the safe distance between cars



According to Mobileye, the developed mathematical rules will reduce the death rate on roads by going to unmanned vehicles by three orders of magnitude. If now the death rate is 1 person per million kilometers of travel, then in unmanned vehicles it will be 1 person per billion kilometers. For the American transport system, this means a reduction in mortality from about 40,000 people per year (statistics for 2016) to about 40 people per year.



It should be borne in mind that accidents and deaths of people due to the fault of the UAVs are still possible in the event of mechanical damage, failure of sensors on the unmanned vehicle and other faults. Nevertheless, the reduction in mortality by three orders of magnitude must convincingly show the benefits of unmanned vehicles to society.





Fig. 3. The calculated safety corridors around the car will determine the culprit during maneuvering and cropping



The unmanned vehicle (blue car in Fig. 3) clearly knows which corridors should be followed up to the car in front and behind the car. If a man-driver invades this corridor, for example, from the front (that is, cuts the drone), and because of a sudden ottormazhivaniya the drone will collide with the car from behind, then a red car with a man-driver will be recognized as the cause of such an accident.





Fig. 4. The RSS system even takes into account situations where objects (cars, pedestrians, etc.) are closed by other objects.



Mobileye proposes using Sensor Fusion, a sensor system based on three independently developed systems, where each relies on three different technologies: a camera, a high-resolution map, and a radar with lidar.



When calculating the safe condition and the corresponding cautious actions, the mathematical model of RSS takes into account situations where objects (cars, pedestrians, etc.) are closed by other objects. For example, in fig. 4 the car leaves the parking lot carefully, taking into account the maximum possible speed ( $$ $ inline $ V_ {limit} $ inline $ ) car on the road.



If such rules are firmly sewn up with the autopilot program, then the investigation of unmanned vehicle incidents will become simple, short and fact-based, and responsibility for accidents can be established precisely and definitively. Such rules, according to the company, will increase public confidence in unmanned vehicles. Everyone will know that there are immutable rules that obey any machine with autopilot. As the three laws of robotics Isaac Asimov, only for traffic rules.