Blitz Engine & Battle Prime: ECS and Network Code

Battle Prime is the first project of our studio. Despite the fact that many members of the team have decent experience in developing games, we naturally encountered various difficulties while working on it. They arose both in the process of working on the engine and in the process of developing the game itself.

In the gamedev industry, a huge number of developers who willingly share their stories, best practices, architectural decisions - in one form or another. This experience, laid out in the public space in the form of articles, presentations and reports, is an excellent source of ideas and inspiration. For example, reports from the Overwatch development team were very useful for us when working on the engine. Like the game itself, they are very talentedly made, and I advise everyone who is interested to see them. Available in GDC vault and on YouTube .

This is one of the reasons why we also want to contribute to the common cause - and this article is one of the first devoted to the technical details of developing the Blitz Engine and playing it - Battle Prime.

The article will be divided into two parts:

ECS: implementation of the Entity-Component-System pattern inside the Blitz Engine. This section is important for understanding the code examples in the article, and in itself is a separate interesting topic.
Netcode and gameplay: everything related to the high-level network part and its use inside the game - client-server architecture, client predictions, replication. One of the most important things in a shooter is shooting, so it will be given more time.

Under the cut a lot of megabytes of gifs!

Inside each section, in addition to the story about functionality and its use, I will try to describe the shortcomings that it carries in itself - be it its limitations, inconvenience in work, or just thoughts about its improvements in the future.

I will also try to give code examples and some statistics. Firstly, it’s just interesting, and secondly, it gives a little context on the scale of using this or that functionality and project.

ECS

Inside the engine, we use the term “world” to describe a scene containing a hierarchy of objects.

Worlds work according to the Entity-Component-System template ( description on Wikipedia ):

Entity - an object inside the scene. It is a repository for a set of components. Objects can be nested, forming a hierarchy within the world;
Component - is the data necessary for the operation of any mechanics, and which determines the behavior of the object. For example, `TransformComponent` contains the transformation of the object, and` DynamicBodyComponent` contains data for physical simulation. Some components may not have additional data, their simple presence in the object describes the state of this object. For example, in Battle Prime, `AliveComponent` and` DeadComponent` are used, which mark live and dead characters, respectively;
System - a periodically called set of functions that support the solution of its task. With each call, the system processes objects that satisfy some condition (usually having a certain set of components) and, if necessary, modifies them. All game logic and most of the engine are implemented at the system level. For example, inside the engine there is a `LodSystem` that calculates the LOD (level of detail) indices for an object based on its transformation in the world and other data. This index contained in the `LodComponent` is then used by other systems for its tasks.

This approach makes it easy to combine different mechanics within the same object. As soon as the entity receives enough data for the work of some mechanics, the systems responsible for this mechanics begin to process this object.

In practice, adding a new functional reduces to a new component (or set of components) and a new system (or set of systems) that implement this functional. In the vast majority of cases, working on this pattern is convenient.

Reflection

Before proceeding to the description of components and systems, I will dwell a little on the reflection mechanism, since it will often be used in code examples.

Reflection allows you to receive and use information about types while the application is running. In particular, the following features are available:

Get a list of types according to a specific criterion (for example, the heirs of a class or having a special tag),
Get a list of class fields,
Get a list of methods inside the class,
Get a list of enum values,
Call some method or change the value of a field,
Get the metadata of a field or method that can be used for a particular functional.

Many modules inside the engine use reflection for their own purposes. Some examples:

Integrations of scripting languages use reflection to work with types declared in C ++ code;
The editor uses reflection to obtain a list of components that can be added to the object, as well as to display and edit their fields;
The network module uses the field metadata inside the components for a number of functions: they indicate the parameters for replicating fields from the server to clients, data quantization during replication, and so on;
Various configs are deserialized into objects of the corresponding types using reflection.

We use our own implementation, the interface of which is not much different from other existing solutions (for example, github.com/rttrorg/rttr ). Using the example of CapturePointComponent (which describes the capture point for the game mode), adding reflection to the type looks like this:

//     class CapturePointComponent final : public Component { //            BZ_VIRTUAL_REFLECTION(Component); public: float points_to_own = 10.0f; String visible_name; // …   }; //   .cpp  BZ_VIRTUAL_REFLECTION_IMPL(CapturePointComponent) { //       ReflectionRegistrar::begin_class<CapturePointComponent>() [M<Serializable>(), M<Scriptable>(), M<DisplayName>("Capture point")] //      .field("points_to_own", &CapturePointComponent::points_to_own) [M<Serializable>(), M<DisplayName>("Points to own")] .field("visible_name", &CapturePointComponent::visible_name) [M<Serializable>(), M<DisplayName>("Name")] // …     }

I would like to pay special attention to the metadata of types, fields, and methods that are declared using the expression

 M<T>()

where `T` is the type of metadata (inside the command we just use the term“ meta ", in the future I will use it). They are used by different modules for their own purposes. For example, the editor uses `DisplayName` to display type names and fields inside the editor, and the network module receives a list of all components, and among them it searches for fields marked as` Replicable` - they will be sent from the server to the clients.

Description of components and their addition to the object

Each component is an inheritor of the `Component` base class and can describe with reflection the fields that it uses (if necessary).

This is how the `AvatarHitComponent` is declared and described inside the game:

 /** Component that indicates avatar hit event. */ class AvatarHitComponent final : public Component { BZ_VIRTUAL_REFLECTION(Component); public: PlayerId source_id = NetConstants::INVALID_PLAYER_ID; PlayerId target_id = NetConstants::INVALID_PLAYER_ID; HitboxType hitbox_type = HitboxType::UNKNOWN; }; BZ_VIRTUAL_REFLECTION_IMPL(AvatarHitComponent) { ReflectionRegistrar::begin_class<AvatarHitComponent>() .ctor_by_pointer() .copy_ctor_by_pointer() .field("source_id", &AvatarHitComponent::source_id)[M<Replicable>()] .field("target_id", &AvatarHitComponent::target_id)[M<Replicable>()] .field("hitbox_type", &AvatarHitComponent::hitbox_type)[M<Replicable>()]; }

This component marks an object that is created as a result of a player hitting another player. It contains information about this event, such as the identifiers of the attacking player and his goal, as well as the type of hit box on which the hit occurred.

Simply put, this object is created inside the server system in a similar way:

 Entity hit_entity = world->create_entity(); auto* const avatar_hit_component = hit_entity.add<AvatarHitComponent>(); avatar_hit_component->source_id = source_player_id; avatar_hit_component->target_id = target_player_id; avatar_hit_component->hitbox_type = hitbox_type; //      //      // ...

The object with the `AvatarHitComponent` is then used by different systems: to play the sounds of hitting players, collecting statistics, tracking player achievements and so on.

Description of systems and their work

A system is an object with a type inherited from `System`, which contains methods that implement a particular task. As a rule, one method is enough. Several methods are necessary if they must be performed at different points in time within the same frame.

Like the components that describe their fields, each system describes the methods that should be performed by the world.

For example, the ExplosiveSystem responsible for the explosions is declared and described as follows:

 // System responsible for handling explosive components: // - tracking when they need to be exploded: by timer, trigger zone etc. // - destroying them on explosion and creating separate explosion entity class ExplosiveSystem final : public System { BZ_VIRTUAL_REFLECTION(System); public: ExplosiveSystem(World* world); private: void update(float dt); //    ,     // ... }; BZ_VIRTUAL_REFLECTION_IMPL(ExplosiveSystem) { ReflectionRegistrar::begin_class<ExplosiveSystem>()[M<SystemTags>("battle")] .ctor_by_pointer<World*>() .method("ExplosiveSystem::update", &ExplosiveSystem::update)[M<SystemTask>( TaskGroups::GAMEPLAY_END, ReadAccess::set< TimeSingleComponent, WeaponDescriptorComponent, BallisticComponent, ProjectileComponent, GrenadeComponent>(), WriteAccess::set<ExplosiveComponent>(), InitAccess::set<ExplosiveStatsComponent, LocalExplosionComponent, ServerExplosionComponent, EntityWasteComponent, ReplicationComponent, AbilityIdComponent, WeaponBaseStatsComponent, HitDamageStatsComponent, ClusterGrenadeStatsComponent>(), UpdateType::FIXED, Vector<TaskOrder>{ TaskOrder::before(FastName{ "ballistic_update" }) })]; }

The following data is indicated inside the system description:

The tag to which the system belongs. Each world contains a set of tags, and on them are the systems that should work in this world. In this case, the `battle` tag means the world in which the battle between the players takes place. Other examples of tags are `server` and` client` (the system runs only on the server or client, respectively) and `render` (the system runs only in GUI mode);
The group within which this system is executed and the list of components that this system uses - for writing, reading and creating;
Update type - whether this system should work in normal update, fixed update or others;
Explicit permission dependencies between systems.

More information about groups of systems, dependencies and update types will be described below.

Declared methods are called by the world at the right time to maintain the functionality of this system. The content of the method depends on the system, but, as a rule, it is a passage through all objects that meet the criteria of this system, and their subsequent update. For example, an `ExplosiveSystem` update inside the game looks like this:

 void ExplosiveSystem::update(float dt) { const auto* time_single_component = world->get<TimeSingleComponent>(); // Init new explosives for (Component* component : new_explosives_group->components) { auto* explosive_component = static_cast<ExplosiveComponent*>(component); init_explosive(explosive_component, time_single_component); } new_explosives_group->components.clear(); // Update all explosives for (ExplosiveComponent* explosive_component : explosives_group) { update_explosive(explosive_component, time_single_component, dt); } }

The groups in the example above (`new_explosives_group` and` explosives_group`) are auxiliary containers that simplify system implementations. new_explosives_group is a container with new objects that are necessary for this system and which have never been processed, and explosives_group is a container with all objects that need to be processed every frame. The world is directly responsible for filling these containers. Their receipt by the system occurs in its constructor:

 ExplosiveSystem::ExplosiveSystem(World* world) : System(world) { // `explosives_group`        `ExplosiveComponent` explosives_group = world->acquire_component_group<ExplosiveComponent>(); // `new_explosives_group`        //  `ExplosiveComponent` -       new_explosives_group = explosive_group->acquire_component_group_on_add(); }

World update

The world, an object of type `World`, each frame calls the necessary methods in a number of systems. Which systems will be called depends on their type.

Part of the systems every frame is necessarily updated (the term “normal update” is used inside the engine) - this type includes all systems that affect the rendering of the frame and sounds: skeletal animations, particles, UI, and so on. The other part is executed at a fixed, predetermined frequency (we use the term “fixed update”, and for the number of fixed updates per second - FFPS) - they process most of the gameplay logic and everything that needs to be synchronized between the client and server - for example, part of the player’s input, character movement, shooting, part of the physical simulation.

The frequency of execution of a fixed update should be balanced - a too small value leads to unresponsive gameplay (for example, the player’s input is processed less often, which means with a longer delay), and too high - to great performance requirements from the device on which the application is running. It also means that the higher the frequency, the greater the cost of server capacity (fewer battles can work simultaneously on the same machine).

In the gif below, the world works at a frequency of 5 fixed updates per second. You can notice the delay between pressing the W button and the start of movement, as well as the delay between releasing the button and stopping the movement of the character:

In the following gif, the world works at a frequency of 30 fixed updates per second, which gives significantly more responsive control:

At the moment, in Battle Prime fixed update the world runs 31 times per second. This “ugly” value has been specially chosen - it may cause bugs that would not exist in other situations when the number of updates per second is, for example, a round number or a multiple of the screen refresh rate.

System execution order

One of the things that complicates the work with ECS is the task of executing systems. For context, at the time of writing, in the Battle Prime client during the battle between the players there is a 251 system and their number is only growing.

A system that is mistakenly executed at the wrong moment in time can lead to subtle bugs or to a delay in the operation of some mechanics for one frame (for example, if the damage system works at the beginning of the frame and the projectile flight system at the end, then damage is done with a delay of one frame).

The execution order of systems can be set in various ways, for example:

Explicit ordering
Indication of the numerical “priority” of the system and subsequent sorting by priority;
Automatically build a graph of dependencies between systems and install them in the right places in the order of execution.

At the moment, we are using the third option. Each system indicates which components it uses for reading, which for writing, and which components it creates. Then, the systems are automatically arranged among themselves in the necessary order:

The system reading component A comes after the system writing to component A;
A system that writes to or reads component B comes after the system that creates component B;
If both systems write to component C, the order can be any (but can be specified manually if necessary).

In theory, such a solution minimizes control over the execution order; all that is needed is to set component masks for the system. In practice, with the growth of the project, this leads to more and more cycles between systems. If system-1 writes to component A, and reads component B, and system-2 reads component A and writes to component B, this is a cycle, and it must be resolved manually. Often, there are more than two systems in a cycle. Their resolution requires time and explicit indications of the relationship between them.

Therefore, the Blitz Engine has “groups” of systems. Inside groups, systems are automatically lined up in the desired order (and cycles are still manually resolved), and the order of groups is set explicitly. This solution is a cross between a fully manual order and a fully automated one, and the size of the groups seriously affects its effectiveness. As soon as the group gets too big, programmers again often come across problems with loops within them.

There are currently 10 groups in Battle Prime. This is still not enough, and we plan to increase their number by building a strict logical sequence between them, and using the automatic construction of a graph inside each of them.

Indication of which components are used by systems for writing or reading will also allow in the future to automatically group systems into “blocks” that will be executed in parallel with each other.

Below is an auxiliary utility that displays a list of systems and the dependencies between them inside each of the groups (full graphs inside the groups look intimidating). Orange color shows explicitly defined dependencies between systems:

Communication between systems and their configuration

The tasks that systems perform within themselves can, to one degree or another, depend on the results of other systems. For example, a system processing collisions of two objects depends on a simulation of physics that registers these collisions. And the damage system depends on the results of the ballistic system, which is responsible for the movement of shells.

The easiest and most obvious way to communicate between systems is to use components. One system adds the results of its work into a component, and the second system reads these results from the component and solves its problem on their basis.

A component based approach may be inconvenient in some cases:

What if the result of the system is not tied directly to some object? For example, a system that collects battle statistics (the number of shots, hits, deaths, and so on) - collects it globally, based on the entire battle;
What if the system needs to be configured in some way? For example, a physical simulation system needs to know which types of objects should record collisions between themselves and which are not.

To solve these problems, we use the approach that we borrowed from the Overwatch development team - Single Components.

Single component is a component that exists in the world in a single copy and is obtained directly from the world. Systems can use it to add up the results of their work, which are then used by other systems, or to configure their work.

At the moment, the project (engine modules + game) has about 120 Single Components that are used for different purposes - from storing global data of the world to the configuration of individual systems.

“Clean” approach

In its purest form, such an approach to systems and components requires the availability of data only within the components and the presence of logic only within the systems. In my opinion, in practice this restriction rarely makes sense to strictly abide by (although debates on this subject are still periodically raised).

The following arguments in favor of a less “rigorous” approach can be highlighted:

Part of the code should be shared - and executed synchronously from different systems or when setting some properties of the components. Similar logic is described separately. As part of the engine, we use the term Utils. For example, inside the game `DamageUtils` contains the logic associated with the application of damage - which can be applied from different systems;
It makes no sense to keep the system’s private data in some place other than this system itself - no one will need it except for it, and moving it to another place is of little use. There is an exception to this rule, which is associated with the functionality of client predictions - it will be written about in the section below;
It is useful for components to have a small amount of logic - for the most part these are smart getters and setters that simplify working with the component.

Netcode

Battle Prime uses an architecture with an authoritarian server and client predictions. This allows the player to receive instant feedback from their actions even at high pings and packet losses, and the project as a whole - to minimize cheating by players, because the server dictates all the simulation results inside the battle.

All code inside the game project is divided into three parts:

Client - systems and components that work only on the client. These include things like UI, auto shooting and interpolation;
Server - systems and components that work only on the server. For example, everything related to damage and spawn characters;
General - this is all that works on both the server and the client. In particular, all systems that calculate the movement of the character, the state of the weapon (the number of rounds, cooldowns) and everything else that needs to be predicted on the client. Most of the systems responsible for visual effects are also common - the server can optionally be launched in GUI mode (for the most part only for debugging).

User input (input)

Before moving on to the details of replication and predictions on the client, you should dwell on working with input inside the engine - the details of this will be important in the sections below.

All input from the player is divided into two types: low-level and high-level:

Low-level input - these are events from input devices, such as keystrokes, touching the screen, and so on. Such an input is rarely processed by gameplay systems;
High-level input - is the user's actions committed by him in the context of the game: shot, change of weapon, character movement, and so on. For such high-level actions, we use the term `Action`. Also, additional data can be associated with the action - such as the direction of movement or the index of the selected weapon. The vast majority of systems work with Actions.

A high-level input is generated either on the basis of binders from a low-level input, or programmatically. For example, a firing action can be tied to a click of a mouse button, or it can be generated by the system responsible for auto-shooting - as soon as the player has aimed at the enemy, this system generates an action shot if the user has the corresponding setting enabled. Actions can also be sent by the UI system: for example, by pressing the corresponding button or when moving the on-screen joystick. A system that fires does not matter how this action was created.

Logically related actions are grouped together (objects of type `ActionSet`). Groups can be disconnected if they are not needed in the current context - for example, in Battle Prime there are several groups, among which:

Actions to control the movement of the character,
Actions for firing automatic weapons,
Actions for firing semi-automatic weapons.

Of the last two groups, only one is active at a time, depending on the type of weapon selected - they differ in how the FIRE action is generated: while the button is pressed (for automatic weapons) or only once when the button is pressed (for semi-automatic weapons )

Similarly, groups of actions are created and configured within the game within one of the systems:

 static const Map<FastName, ActionSet> action_sets = { { //     ControlModes::CHARACTER_MOVEMENT, ActionSet { { DigitalBinding{ ActionNames::JUMP, { { InputCode::KB_SPACE, DigitalState::just_pressed() } }, nullopt }, DigitalBinding{ ActionNames::MOVE, { { InputCode::KB_W, DigitalState::pressed() } }, ActionValue{ AnalogState{0.0f, 1.0f, 0.0f} } }, //    ... }, { AnalogBinding{ ActionNames::LOOK, InputCode::MOUSE_RELATIVE_POSITION, AnalogStateType::ABSOLUTE, AnalogStateBasis::LOGICAL, {} } //    ... } } }, { //       ControlModes::AUTOMATIC_FIRE, ActionSet { { // FIRE    ,      DigitalBinding{ ActionNames::FIRE, { { InputCode::MOUSE_LBUTTON, DigitalState::pressed() } }, nullopt }, //       ... } } }, { //       ControlModes::SEMI_AUTOMATIC_FIRE, ActionSet { { // FIRE          DigitalBinding{ ActionNames::FIRE, { { InputCode::MOUSE_LBUTTON, DigitalState::just_pressed() } }, nullopt }, //       ... } } } //   ... };

Battle Prime describes about 40 actions. Some of them are used only for debugging or recording clips.

Replication

Replication is the process of transferring data from a server to clients. All data is transmitted through objects in the world:

Their creation and deletion,
Creating and deleting components on objects,
Change component properties.

Replication is configured using the appropriate component. For example, in a similar way the game sets up replication of the player’s weapons:

 auto* replication_component = weapon_entity.add<ReplicationComponent>(); replication_component->enable_replication<WeaponDescriptorComponent>(Privacy::PUBLIC); replication_component->enable_replication<WeaponBaseStatsComponent>(Privacy::PUBLIC); replication_component->enable_replication<WeaponComponent>(Privacy::PRIVATE); replication_component->enable_replication<BallisticsStatsComponent>(Privacy::PRIVATE); // ...

For each component, the privacy that is used during replication is specified. Private components will be sent from the server only to the player who owns this weapon. Public components will be sent to everyone. In this example, the `WeaponDescriptorComponent` and` WeaponBaseStatsComponent` are public - they contain the data necessary for the correct display of other players. For example, the index of the slot in which the weapon lies and its type are needed for animations. The remaining components are sent privately to the player who owns this weapon - the parameters of the ballistics of the shells, information about the total number of rounds, the available shooting modes, and so on. There are more specialized privacy modes: for example, you can send a component only to allies or only to enemies.

Each component within its description must specify which fields should be replicated within this component. For example, all fields inside the `WeaponComponent` are marked as` Replicable`:

 BZ_VIRTUAL_REFLECTION_IMPL(WeaponComponent) { ReflectionRegistrar::begin_class<WeaponComponent>() .ctor_by_pointer() .copy_ctor_by_pointer() .field("owner", &WeaponComponent::owner)[M<Replicable>()] .field("fire_mode", &WeaponComponent::fire_mode)[M<Replicable>()] .field("loaded_ammo", &WeaponComponent::loaded_ammo)[M<Replicable>()] .field("ammo", &WeaponComponent::ammo)[M<Replicable>()] .field("shooting_cooldown_end_ms", &WeaponComponent::shooting_cooldown_end_ms)[M<Replicable>()]; }

This mechanism is very convenient to use. For example, inside the server system, which is responsible for “ejecting” tokens from killed opponents (in a special game mode), it is enough to add and configure `ReplicationComponent` on such a token. It looks like this:

 for (const Component* component : added_dead_avatars->components) { Entity kill_token_entity = world->create_entity(); //           // ... //   auto* replication_component = kill_token_entity.add<ReplicationComponent>(); replication_component->enable_replication<TransformComponent>(Privacy::PUBLIC); replication_component->enable_replication<KillTokenComponent>(Privacy::PUBLIC); }

In this example, the physical simulation of the token upon occurrence will occur on the server, and the final transformation of the token will be sent and applied on the client. An interpolation system will also work on the client, which will smooth the movement of this token, taking into account the frequency of updates, the quality of the connection to the server, and so on. Other systems associated with this game mode will add a visual part to objects with a `KillTokenComponent` and monitor their selection.

The only inconvenience of the current approach that you want to pay attention to and which you want to get rid of in the future is the inability to set privacy for each component field. This is not very critical, since a similar problem can be easily solved by splitting the component into several: for example, the game contains `ShooterPublicComponent` and` ShooterPrivateComponent` with the corresponding privacy. Despite the fact that they are tied to the same mechanics (shooting), you need to have two components to save traffic - some of the fields are simply not needed on clients who do not own these components. However, this adds work to the programmer.

In general, objects replicated to a client can have states for different frames. Therefore, the ability to group objects by creating replication groups was added. All components on objects within the same group always have a state for the same frame on the client - this is necessary for the predictions to work correctly (more about them below). For example, a weapon and a character who owns it are in the same group. If the objects are in different groups, then their state in the world can be for different frames.

The replication system tries to minimize the amount of traffic, in particular by compressing the transmitted data (each field inside the component can optionally be marked accordingly for compression) and by transmitting only the difference in values between the two frames.

Customer predictions

Client predictions (the term client-side prediction is used in English) allows the player to receive instant feedback on most of his actions in the game. At the same time, since the last word is always behind the server, in case of an error in the simulation (the term mispredict is used in English, in the future I will call them simply “mispredictions”) the client must fix it. More details about prediction errors and how they are corrected will be described below.

Client predictions work according to the following rules:

The client simulates himself forward by N frames;
All input generated by the client is sent to the server (in the form of actions performed by the player);
N depends on the quality of the connection to the server. The smaller this value, the more “up-to-date” the picture of the world is for the client (ie the time gap between the local player and other players is smaller).

As a result, both the server and the client perform the simulation based on client input. The server then sends the results of this simulation to the client. If the client determines that its results do not coincide with the server ones, then it tries to correct the error - rolls itself back to the last known server state, and again simulates N frames ahead. Then everything continues according to a similar scheme - the client continues to simulate itself in the future with respect to the server, and the server sends it the results of its simulation. It follows that all code that affects client predictions must be shared between the client and server.

Also, in order to save traffic, the entire input is pre-compressed based on a predetermined scheme. Then it is sent to the server and immediately unpacked on the client. Packaging and subsequent unpacking on the client are necessary to eliminate the difference in the values associated with the input between the client and server. When creating a scheme, the range of values for this action is indicated, and the number of bits into which it should be packed. Similarly, the announcement of the packaging scheme in Battle Prime looks like inside a common system between the client and server:

 auto* input_packing_sc = world->get_for_write<InputPackingSingleComponent>(); input_packing_sc->packing_schema = { { ActionNames::MOVE, AnalogStatePrecision{ 8, { -1.f, 1.f }, false } }, { ActionNames::LOOK, AnalogStatePrecision{ 16, { -PI, PI }, false } }, { ActionNames::JUMP, nullopt }, // ..    action' };

A critical condition for the performance of client predictions to work is the need for the input to have time to get to the server by the time the frame simulation to which this input relates. In the event that the input did not manage to reach the desired frame on the server (this can happen during, for example, a sharp ping jump), the server will try to use the input of this client from the previous frame. This is a backup mechanism that can help get rid of mispredictions on the client in some situations. For example, if the client simply runs in one direction and its input does not change for a relatively long time, using the input for the last frame will be successful - the server will “guess” it, and there will be no discrepancy between the client and server. A similar scheme is used in Overwatch (was mentioned in a lecture on GDC:www.youtube.com/watch?v=W3aieHjyNvw ).

Currently, the Battle Prime client predicts the status of the following objects:

Player avatar (position in the world and everything that can affect it, state of skills, etc.);
All player’s weapons (number of rounds in the store, cooldowns between shots, etc.).

Using client predictions comes down to adding and configuring the `PredictionComponent` on the client to the desired objects. For example, prediction of a player’s avatar in one of the systems is turned on in a similar way:

 // `new_local_avatars`       , //      for (Entity avatar : new_local_avatars) { auto* avatar_prediction_component = avatar.add<PredictionComponent>(); avatar_prediction_component->enable_prediction<TransformComponent>(); avatar_prediction_component->enable_prediction<CharacterControllerComponent>(); avatar_prediction_component->enable_prediction<ShooterPrivateComponent>(); avatar_prediction_component->enable_prediction<ShooterPublicComponent>(); // ...      }

This code means that the fields inside the above components will be constantly compared with the same fields of the server components - if a discrepancy in the values within a single frame is noticed, an adjustment will be made on the client.

The discrepancy criterion depends on the type of data. In most cases, this is just a call to `operator ==`, the exception is data based on float - for them the maximum allowable error is currently fixed and is equal to 0.005. In the future, there is a desire to add the ability to set the accuracy for each component field separately.

The replication and client prediction workflow is based on the fact that all the data needed for the simulation is contained in the components. Above, in the section on ECS, I wrote that systems are allowed to hold part of the data - this can be convenient in some cases. This does not apply to any data that affects the simulation - it must always be inside the components, since the client and server snapshot systems work only with the components.

In addition to predicting field values within components, it is possible to predict the creation and removal of components. For example, if, as a result of using the ability, a `SpeedModifierComponent` is superimposed on the character (which modifies the speed of movement, for example, accelerates the player), then it must be added to the character both on the server and on the client on the same frame, otherwise it will lead to an incorrect prediction of the character’s position on the client.

Prediction of creating and deleting objects is not currently supported. This may be convenient in some situations, but it will also complicate network modules. Perhaps we will return to this in the future.

Below is a gif in which character control takes place with RTT for about 1.5 seconds. As you can see, the character is controlled instantly, despite the high delay: moving, shooting, reloading, grenade throwing - everything happens without waiting for information from the server. You can also notice that the capture of a point (a zone limited by triangles) begins with a delay - this mechanics only works on the server and is not predicted by the client.

Mispredictions and resimulations

Mispredict - discrepancy between the results of server and client simulations. Resimulation is the process of correcting this discrepancy by the client.

The first reason for the appearance of mispredicts is the sharp ping jumps, for which the client did not have time to adjust. In such a situation, the input from the player may not have time to get to the server, and the server will use the backup mechanism described above with duplication of the last input for some time, and after a while it will stop using it.

The second reason is the interaction of the character with objects that are fully server-controlled and are not predicted locally by the client. For example, a collision with another player will cause a mispredict - since they, in fact, live in two different time periods (a local character is in the future relative to another player - whose position comes from the server and is interpolated).

The third and most unpleasant reason is bugs in the code. For example, a system can mistakenly use non-replicated data to control the simulation, or the systems work in the wrong order, or even in different orders on the server and client.

Finding these bugs sometimes takes a decent amount of time. In order to simplify their search, we made several auxiliary tools - while the application is running, you can see:

Replicated components
The number of mispredictions
On which frames did they happen,
What data was on the server and on the client in the diverged components,
What input was applied on the server and on the client for this frame.

Unfortunately, even with them, the search for the causes of resimulations still takes a decent amount of time. Toolkit and validation certainly need to be developed to reduce the likelihood of bugs and simplify their search.

In order to support the operation of resimulations, the system must inherit from a specific class `ResimulatableSystem`. In a situation when a mispredict occurs, the world “rolls back” all objects to the last known server state, and then makes the necessary number of simulations ahead to fix this error - only Resimulatable systems will participate in this.

In general, client resimulations should not be noticeable to players. When they occur, all component fields are smoothly interpolated into new values to visually smooth out possible “twitches”. However, it is critical to keep their number as low as possible.

Shooting

Damage to players is completely dictated by the server - customers cannot be trusted in such an important mechanics to reduce the likelihood of cheating. But, like movement, shooting on the client should be as responsive as possible and without delay - the player needs to receive instant feedback in the form of effects and sounds - muzzle flash, the trace of the projectile’s flight, as well as the effects of the projectile hitting the surroundings and other players.

Therefore, the entire state of the character associated with the shooting is predicted by the client - how many rounds are in the store, the dispersion during firing, the delay between shots, the time of the last shot, and so on. Also on the client are the same systems responsible for the movement of shells as on the server - this allows you to simulate shots on the client without waiting for the results of their simulation on the server.

The ballistics of the shells themselves are not predicted - since they fly at a very high speed, and, as a rule, finish their movement in a few frames, the shell will already have time to get to some point in the world and lose the effect before we get the simulation results this is a shell from the server (or the lack of results if, due to a mispredict, a client fired a shell by mistake).

The scheme of work of slowly flying shells is slightly different. If a player throws a grenade, but as a result of the mispredict it turns out that the grenade was not thrown, it will be destroyed on the client. Similarly, if a client incorrectly predicted the destruction of a grenade (it already exploded on the server, but not yet on the client), then the client grenade will also be destroyed. All information about the explosions that are displayed on the client comes from the server in order to avoid situations when, as a result of a client error, the server explosion occurred in one place and on the client in another.

Ideally, I would like to fully predict slow-flying shells in the future - not only the time of life, but also their position.

Lag compensation

Lag compensation is a technique that allows you to level out the effect of the delay between the server and the client on the accuracy of shooting. In this section, I will assume that shooting always comes from “hitscan” weapons — that is, a projectile fired by a weapon travels at infinite speed. But everything that is described here also matters with other types of weapons.

The following points make it necessary to compensate for the lag when shooting:

The character under the control of the player is in the future relative to the server (predicting its state for a certain number of frames ahead);
Consequently, the rest of the players are relatively in the past;
When fired, the corresponding action is sent by the client to the server and applied on the same frame on which it was applied on the client (if possible).

If we assume that the player is aiming at an enemy running towards the head and presses the shot button, the following picture is obtained:

On the client: the shooter on frame N1 fires a shot at the head of an enemy located on frame N0 (N0 <N1);
On the server: the shooter on frame N1 fires a shot at the head of the enemy, also located on frame N1 (on the server, everyone is at the same time).

The result of this, with a high probability, is a miss during a shot. Since the client targets on the basis of his picture of the world, which does not coincide with the picture of the server’s world, in order to get into the enemy, he needs to aim ahead of him even when using hitscan weapons, and the distance in front of which he must shoot depends on the quality of the connection with server. This, to put it mildly, is not the best experience for a shooter.

To get rid of this problem, lag compensation is used. The scheme of her work is as follows:

The server holds a limited-sized history of snapshots of the world;
When fired, the enemies (or part of the enemies) “roll back” in such a way that the world on the server matches the world that the client saw in itself - the client is in the “present” (the moment of the shot), and the enemies are in the past;
Hit detection mechanics work, hits are recorded;
The world is returning to its original state.

Since the picture of the world on the client also depends on the operation of the interpolation system, in order to “roll back” the world to the most accurate client state on the server, the client gives him additional data - the difference between the current frame of the client and the frame for which he sees all the other players (at the moment these are two bytes per frame), as well as the time of generation of the shot input relative to the beginning of the frame.

Lag compensation exists at the level of a separate module inside the engine and is not tied to a specific project. From the point of view of the developer of gameplay mechanics, its use is as follows:

`LagCompensationComponent` is added to the player, and the list of hitboxes to be stored in the history is filled;
When shooting (or other mechanics that require compensation - for example, with melee attacks), `LagCompensation :: invoke` is called, where the functor is passed, which will be executed in the“ compensated ”, from the point of view of a particular player, world. It must have all the necessary hit detection.

Code with an example of using lag compensation from Batle Prime when moving ballistic projectiles:

 // `targets_data`    , //   “”    , //    const auto compensated_action = [this](const Vector<LagCompensation::LagCompensationData>& targets_data) { process_projectile(projectile, elapsed_time); }; LagCompensation::invoke( observer, // ,       projectile_component->input_time_ms, // ,      compensated_entities, // ,    compensated_action // ,       );

I would also like to note that lag compensation is a scheme that puts the shooter's experience above the experience of the target he is shooting at. From the point of view of the target, the enemy can get into him at a time when he is already behind an obstacle (a frequent complaint in gaming forums). To do this, lag compensation has a limited number of frames for which goals can be “pumped out”. At the moment, in Battle Prime, a shooter with an RTT of about 400 milliseconds can comfortably hit enemies. If the RTT is higher, you have to shoot ahead.

Example of shooting without compensation - you need to shoot ahead to steadily hit the enemy:

And with compensation - you can comfortably aim directly at the enemy:

Our build agents also periodically run autotests that check the work of different mechanics. Among them, there is also an autotest for firing accuracy with lag compensation enabled. The test below is shown in the gif below - the character simply shoots the head of an enemy running past and counts the number of hits on him. For debugging, the enemy’s hitboxes that were on the server at the time of the shot (in white) and hitboxes that were used for hit detection inside the compensated world (in blue) are additionally displayed:

An additional factor that affects the accuracy of shooting is the position of the hitboxes on the character. Hitboxes depend on skeletal animations, and their phases are currently not synchronized in any way, so a situation is possible where hitboxes differ between the client and the server. The consequences of this depend on the animations themselves - the larger the range of motion inside the animation, the greater the potential difference in the position of the hitboxes between the server and the client. In practice, such a difference is weakly noticeable to the player and affects the lower body more, which is less critical compared to the upper (head, trunk, arms). Nevertheless, in the future I would like to address in more detail the issue of synchronizing animations between the server and the client.

Conclusion

In this article I tried to describe the foundation on which Battle Prime is built - this is the implementation of the ECS pattern inside the Blitz Engine, as well as the network module that is responsible for replication, client predictions and related mechanics. Despite some shortcomings (which we continue to work on fixing), using this functionality is now simple and convenient.

To show the overall picture of Battle Prime, I had to touch on a large number of topics. Many of them may be devoted to separate articles in the future, in which they will be described in more detail!

The game is already being tested in Turkey and the Philippines.

Our previous articles can be found at the following links:

All Articles