Forest doesn't surrender to search technology, but engineers strike back
Photo: Lisa Alert .
If a person has disappeared in the forest, the best way to find him is to send trained search engines for combing. No search technology can replace people yet. In early summer, we wrote about several teams that, as part of the Odyssey contest, developed solutions for wildlife. Many engineers relied on drones with cameras and computer vision, but faced not obvious problems - poor communication, slow processing and transmission of huge data, dense crowns of trees, and much more.
As a result, not a single solution with computer vision reached the final of the competition. But technical experts said - if the teams joined forces, put together a dataset and trained algorithms on it, computer vision could have a chance.
Last week, on August 9, the search squad Lisa Alert together with Beeline announced the launch of a tool that will search for people in photos from drones. We went to a press conference dedicated to the launch and found out how it works.
Tens of thousands of photos
The presentation took place the day after the scandalous announcement by the Ministry of Emergency Situations about the hurricane that hung on the air of the main television channels for ten minutes. The hurricane did not take place, but the weather was still terrible. For the Lisa Alert detachment, this is sometimes at hand — the number of applications for people missing in the forest drops sharply, because in bad weather they simply don’t go there. This frees up forces for more complex searches. On the other hand, during a cold snap, missing people are in even greater danger.
For example, in July Lisa Alert received 1,083 applications for the forest. About 700 of them are in the suburbs. Many lost people still have a telephone, and they are in the network coverage area. From the forest they are led out by the “Forest in touch” group, which explains how to get out by the sun or natural landmarks. Sometimes they send a group that sets up a siren or takes a person out.
But the most difficult searches are those where there is no connection with the missing. For example, at the time of the press conference for several days there was a search for a 73-year-old woman in the Shchelkovo area. Search engine groups walked a total of 2,300 kilometers and took 8,400 photos from the drone over the forest.
“Now our technology works like this,” says Grigory Sergeyev, “we take photos, upload them to a service written by our volunteers, and special groups view photos. A large number of people outside the squad can also help with this. In these frames, we are looking for something that is different from the forest. Hiking groups immediately go out to see what it is. Now in searches in Shchelkovo we have more than a hundred marks that need to be checked. To get around them all, you have to walk more than three hundred kilometers. ”
When we spoke with the participants in the Odyssey contest, they were skeptical about taking pictures of the forest from the air. But this year, “Lisa Alert” finds at least one person a week with the help of copters. According to Gregory, two years ago they found in this way only two people for the entire season. Last year - six.
Drones fly over territories where it’s very difficult for a person to go - windbreaks, swamps. Alexander Lomonosov, the head of the unmanned aerial vehicle Lisa Alert, showed recordings from trackers of search groups. Combing with units left a lot of white spots. Drones completely covered this territory.
“If the group went stormy, they would spend at least three hours. The drone worked this area in 15 minutes. It was there that a man was found - from the air. He could get into the windbreak, and there was no strength left to get out of there. Several trained people evacuated him for more than four hours. And during this time only 370 meters passed. Wind drills, open fields and swamps, the drone will work many times faster than an ordinary walking group. If the field is overgrown with grass by 15 cm, a walking group of trained search engines will comb it for 6–8 hours, and the drone for an hour. He will bring photographs of the entire area, and dead zones will be completely excluded. ”
On average, one search brings 250 photos. For six months, they accumulate 33 thousand. But now there are few copters and pilots in the squad. If the number grows, and drones are used in all 20 regions where Lisa Alert operates, 33 thousand images will be collected in one day. But the detachment spent 240 hours just looking at photos from a search in Shchelkovo.
Datasets of search engines and telecom power
Two months ago, Lisa Alert, along with Beeline, began testing the Beeline AI - People Search algorithm, which analyzes photos from drones.
The project began with the idea of learning how to automatically filter out empty photographs so that only the photographs suspected of a human being came to the group that was watching.
“At one time I tried to find special datasets, but they do not exist in the world, although some tried to collect them,” says Alexander Lomonosov, “therefore, we ourselves passed to the developers photographs where people were: ordinary passers-by who got into the frame, mushroom pickers , the search engines themselves at work. All this laid the foundation for the dataset on which machine vision works. ”
To assemble such a dataset is also a task with many subtle subtleties. People can get into the frame in any position: sitting, lying, standing. They can be so covered with branches that only a small part is visible. The background around is very different depending on the landscape, region and time of year.
“In many regions, we specifically asked people to go to typical poses for the victims and photograph them. Then the pictures were artificially propagated. For example, they took a person photographed in Rostov and put them on substrates from other regions, ”says Grigory Sergeyev.
In the photographs received, Beeline data scientists began to learn algorithms.
“We, as an operator, are used to working with a huge amount of data. We have five huge data centers. Only the Yaroslavl data center, which we use to work with Lisa Alert, processes 15 petabytes of data. Thanks to this, we learned how to build very modern models, ”says George Held, vice president of digital business development at Beeline.
“Photos come to our system, and we need to determine if there is a person on them or not. See a hat, boot, hand, ear, whatever. Therefore, we cut each image into 30-35 layers. Gigantic work was done to find the architecture suitable for this and to create an algorithm, because any mistake will lead to the fact that we will miss people in photos. Huge computing resources are needed to quickly process each layer. We use the same models that we use for targeting. We take their components and reuse on this project. ”
The model can not only search for the whole person, but notices all small and suspicious artifacts in the forest in the pictures. Now comes the first stage of testing, when the algorithm discards all photos where there is a 98% percentage probability that there is no person.
The remaining images immediately go to the volunteers who are looking for people there, and if necessary they are sent to search engines or drones for additional shooting. While the system is being tested, the discarded images are also checked by people - but in the second place.
The next step, the developers plan to teach the algorithm to more accurately determine the pictures of people and build predictive models of their movements in the forest.
Why 5G here
Now the system has two options. The first is an offline program to process pictures on laptops in conditions where there is no connection. Its power is limited, plus it must be updated every time you connect to the network, because the algorithm is constantly learning. The online service works at its full potential - in it the photos are uploaded to the Beeline server. Already 15 million images have passed through the data centers.
Lisa Alert uses DJI drones in search, because she uses their programs for autonomous flight. One shot from the Mavic 2 Pro model weighs about 20 MB, and after each flight you have to upload and transfer an average of 20 GB of photos to the service. This is a lot even for places with good network coverage.
George Held believes that the work of search algorithms is an argument in favor of building 5G networks .
“It is very important that information is transmitted over a large high-quality network, so we are now investing a lot of money in improving base stations, a 5G network is being built. Many operators say this with pride. But the biggest question is - what is it for? To just watch a video on YouTube? It will also work just fine on 4G. 5G needs real reasons. Working with Lisa Alert is a very good example, because a huge amount of information must be transmitted quickly. ”
No thermal imagers needed
In early summer, in a search technology competition, only one team that relied on computer vision qualified for the finals. But she used a thermal imager for filming. Grigory Sergeev refers to this with skepticism.
“For 9 years we have not found a single person with the help of a thermal imager. Everyone wants to believe that this is a cool thing. Yes, we can use it so as not to stop searching, for example, at night in open spaces. But for everything else, it is not suitable. The grass is not transparent, all crowns are opaque. A person is in the forest in clothes, and clothing has an ambient temperature. The thermal imager is a cool fantasy, but it is ineffective for searches. We tried all our defense devices, tried foreign models. "They are suitable for duplication, but given the cost and effectiveness, I would say no to thermal imagers."
When asked why no one in that competition was able to effectively use computer vision, Georgy replies simply: “they did not have the development power that we now have.”
All Articles