In recent years, there has been much talk that machine learning will radically change the gaming industry, because this technology has already become a breakthrough in many other digital applications. But do not forget that the games are much more complicated than a car driving simulator, a drone control program or face recognition algorithms in an image.
So far, in the gaming industry, it is still customary to use traditional AI methods, such as the finite state machine method, behavior trees, and - more recently - Utility-based AI (utility-based systems). Such AIs are also called design-based (artificial design intelligence) or expert systems. But it is becoming increasingly obvious - and to players in the first place - that these systems are less and less suitable for creating truly advanced opponents that could imitate the behavior of players. This is especially true for creative solutions. This can be explained by the fact that the developers of artificial intelligence are not able to take into account all possible tactics and strategies of behavior and successfully implement them in traditional AI systems. For players, this often turns out to be boring and predictable to play with an opponent whose behavior is easy to remember.
Many reasons lead to this result, but one of the main ones is the inability of AI to learn. Therefore, when creating the enemy's artificial intelligence, the decision on the transition to machine learning, which has proven itself in many other applications, comes to mind by itself. But there are a few nuances that are worth considering. So, a game AI must be able to adapt to any situation and use the advantages that it gives it, as well as adapt to the different game styles of rivals - live players and other AI.
How are things now
The British company DeepMind, engaged in the development of artificial intelligence, recently showed how AI can independently learn to play games, learn their rules and find ways to pass the game or win it - however, so far only on the example of simple games, such as the early Atari games - for example, chess and the Japanese logic game go. The results obtained for them show that artificial intelligence is able to form for itself an adequate assessment of what is happening on the field. If we talk about the adaptation of AI to various styles of the opponent’s game, then the results are not so impressive.
Nowadays, neural networks have already learned to recognize images and drive cars. But these functions can be realized with the help of relatively simple architectures, even if as a result they turn out to be quite deep and voluminous. So, AI for image recognition on Facebook has a depth of about 100 layers, which is why it resembles a biological brain - in the number and complexity of the relationship between neurons that form one large network.
Game AI
As for the application of machine learning in the gaming industry, there are a number of limitations, because of which it is not always possible to use this type of architecture. These include system requirements, in particular related to the CPU, which determine the ability of a computer to process complex game structures and its suitability for game storytelling and gameplay.
Thus, it turns out that in many games for the implementation of a complex artificial intelligence system it is not possible to organize the necessary hardware and even more so the server cluster, which exists, for example, for image recognition networks on Facebook. Sometimes several AIs must simultaneously work at the same time - and not only on computers, but also on mobile devices, and on other less productive platforms. All this imposes restrictions on the size and complexity of the machine learning architecture, because all calculations must also be performed with a frame duration of the order of 1 or 2 milliseconds. Of course, you can use various optimization technologies and load balancing between frames, but still you can’t get rid of these restrictions at all.
Serious problems for AI can cause the complexity of the game. Indeed, in such games as, for example, StarCraft II, the mechanics of the game are many times more complicated than in Atari games. Therefore, you should not expect that for a given frame frequency and with known system requirements, machine learning will surely cope with the study of the entire state of the game and will be able to interact with it. As a player is often guided by intuition in the early stages of the game, so the AI must learn how to process the game’s state in order to simplify its further passage. For example, in one of the latest APIs for Starcraft II on maps, only that information that the developers considered important was displayed: in one case, the AI used a reduced view of the entire territory of the map, in the second - like a player, he could move the camera, and then his perception was limited to information on the screen.
Visualization of the AlphaStar AI game against a player in StarSraft II: the screenshot shows “raw” initial observations, the activity of the neural network, some of its possible actions and coordinates, as well as the expected outcome of the match
This is a particularly relevant aspect in the case of games. Often, generally accepted methods for solving machine learning problems are not applicable to gaming artificial intelligence. For example, usually he is not obliged to win or do everything in his power to win, as was the case with Atari games. More often, the role of AI is to make the passing game more exciting. It may be necessary for him to play a role and behave as the character of the character for whom he is responsible suggests. Thus, gaming AIs are more tied to game design and storytelling and must have the necessary tools to control their behavior to achieve their goals. Pure machine learning is not always suitable for this - which means that you need to look for something else.
Practical Machine Learning Issues
These problems surfaced in the development of machine learning-based artificial intelligence for Unleash , where AIs should behave like regular players - that is, be as flexible and resourceful.
Like Starcraft II, Unleashed is much more complicated than chess and go for Atari. The gameplay in it is intuitive and easy to learn, but in order to truly succeed in it, you need certain meta management skills. The player must build mazes, set monsters on enemies and think over his strategy in economics, attack and defense of structures throughout the game. To do this, he needs to bluff and calculate other people's moves ahead, as well as manage a psychological meta - it is she who makes poker something more than just a statistical game.
Screenshot from Unleashed
In search of the architecture most suitable for these purposes, technologies such as neuroevolution and deep learning were first introduced into the game in an almost unchanged form, and we checked how they would show themselves raw as an enemy AI.
It was terrible.
It quickly became apparent that Unleashed needed to solve many global problems that were hard to adapt to machine learning.
One of them is the construction of an effective maze. As in many games whose goal is to protect the tower, here players need to build a maze around it, through which monsters will break through. They, in turn, must be eliminated with the help of weapons placed in a maze. Ideally, the maze should be as long as possible in order to be able to inflict enough damage on the monsters and prevent them from reaching the tower. For some types of weapons, monsters are more vulnerable than for others, so for greater efficiency they should be placed in the maze earlier than everyone else. The peculiarity of Unleash is that there is no perfect maze: there are so many types of monsters in the game that anyway one of them will freely pass through any part of the maze. Any maze must be adapted for new monsters launched by other players. Thus, it was necessary not only to teach artificial intelligence to build labyrinths - it was necessary to teach him how to create effective labyrinths for various scenarios, which can only be encountered in both the early and late versions of the game.
Also, the AI had to learn how to calculate which monsters would appear in the maze. This is a kind of inverse problem with respect to the construction of the maze. As in many other games, in Unleash it’s not enough just to build up an army and send it to the enemy’s camp: you also need to spy on the enemy’s defense and structure your army in such a way as to hit the enemy’s pain points as efficiently as possible. The army of monsters should interact with each other in such a way as to most successfully break through the maze. Sometimes it is also necessary to release monsters in a certain order depending on their functions and role. This also increases the number of different combinations.
Finally, since the player must both create mazes and assemble an army of monsters, AI also needs to learn how to find balance in attack and defense. It is also worth considering that the more the player builds up an army of monsters and the more he builds the maze, the more resources he needs for this. Therefore, the correct attack strategy is extremely important both for the economy during the game and for victory in it. And to ensure competitiveness, the AI must be able to dispose of resources in such a way as to create a powerful army of monsters, without jeopardizing the strength of the maze. Investing in monsters to the maximum can be economically profitable, but this increases the risk of enemy monsters taking the maze. If you bet on strengthening the protection of the labyrinth, it can undermine your economy. None of these scenarios will lead to victory. Thus, the optimization problem in Unleashed is larger than in the case of chess or Starcraft, and includes the need to sacrifice something and calculate your profit several steps forward.
As artificial intelligence learns, many previously unaccounted problems pop up. So, at first, the AI often reached a certain level of development, at which he began to understand some aspects of the game - for example, which weapons in the maze are effective against specific types of monsters or which monsters pass certain sections of the maze best of all. But learning was slow and led to the development of uniform strategies.
The need for parallel approaches
While machine learning-based AI learning developed slowly and not particularly successfully, for other stages of testing and development, better AIs and more robust rival AIs became necessary. To implement them, the Utility architecture was used, with which you can create special AIs for testing and checking the quality of the game, in-game tests and balancing weapons and monsters, and creating specific labyrinths and monsters. However, during the development of Unleash, the creators themselves honed their skills in passing it, and then decided to use the acquired knowledge to create a more complex AI Utility. So it became clear that many problems that arise in artificial learning systems based on machine learning can be easily solved with the help of Utility systems that use their knowledge, and vice versa.
For example, it is better to build more effective labyrinths using Utility AI, based on knowledge bases compiled from internal tests. You can easily describe and program the algorithm for constructing the maze and the location of weapons in it in such a way as it would be easier to defend the tower from specific monsters to a living player. But the creation of an army of monsters based on knowledge of the enemy’s base was a difficult task for such AIs, since the number of different conditions and combinations that must be taken into account was amazing. With this kind of AI architecture, finding suitable sets of monsters would take an infinitely long time. Then, given the given restrictions, deep learning would be an ideal solution for this task.
Hybrid AI Creation
So, it was decided to combine the two approaches and thus create a hybrid system of artificial intelligence based on machine learning and Utility. The idea was that where it was necessary to process a huge number of combinations and states of the game, or where it was required to train it with something, machine learning was used. For other tasks, where it is better to rely on the personal experience of developers, Utility systems were involved. The advantage of this approach also lies in the fact that, if necessary, it is possible to better control the behavior of AI in order to ensure its more exact adherence to a given goal. For example, you can use the AI Utility to ensure a balance between attack and defense and thus create different levels of aggression, or you can create different labyrinth configurations for different AIs to create individual game styles for them. You can also set certain value systems for neural networks to form various preferences when recruiting air or ground monsters, and thus also add personality to individual AIs. There are many other options for implementing design decisions, and all of them emphasize the strengths of a particular type of AI architecture.
The hybrid approach also answered another question that the development team faced in the process of developing artificial intelligence for Unleash: is it worth using a single global deep neural network based on machine learning to take into account all the input and output data, or is it better to design AIs with a hierarchical structure?
Two architectures used in Unleash: on the left is a large deep neural network with its own unified architecture, on the right is a hierarchical system in which each network has its own task
Nevertheless, I would like to create a common approach to the artificial intelligence system, in the architecture of which the developers would not lay their own experience. However, the more entrances to the game were made, the more the neural network grew. At the same time, it was impossible to separate the training of AI and teach them one thing only: either defense or attack. And there were fears that a more general approach would lead to a significant increase in the number of calculations.
From this came the idea of creating a hierarchical architecture in which each specific task would be performed by a specialized neural network. According to this idea, first artificial intelligence needs to make a decision on the allocation of resources to attack (increase the army of monsters) and defense (building a maze). As soon as he does this, he goes to the next layer in accordance with his choice and gets access to the necessary part of the state of the game, after which he makes detailed decisions about which monsters to choose and which weapons to install in the maze.
Conclusion and next steps
In the hybrid Utility approach, AI with machine learning-enabled networks resembles a hierarchical architecture. And she, in turn, is similar to a biological brain in which different nerve centers are each responsible for their task.
Now the enemy AI in Unleash is very difficult to defeat: they are able to adapt to any situation in the game - but at the same time, developers can change their settings at their discretion. According to the author of the article, over time, the hybrid approach should become more widespread and appear in many other games. Perhaps someday it will be possible to introduce artificial intelligence based on machine learning in its purest form in the gameplay. But, obviously, this will still take time. In the meantime, the goal is to find such an architecture, which itself will adapt to the tasks facing it and find the best ways to solve them.