How we made the engine and the game on it for a year and a half. Part two. Infrastructure
First, a few comments on the traces of the previous article. We really used to work at Wargaming , where we developed an engine known as dava.framework or dava.engine . Therefore, many old colleagues, with whom we are still in good relations, actively participate in the discussion.
A number of people have doubts: is this the same technology or another? Answer: this is a new technology written from scratch.
How did we manage in just a year? Our team has vast experience. Many have been developing engines and games for over 15 years.
Why from scratch, if you could take our old engine, which also lies in open-source? He is about 10 years old, and most of the code is out of date. Even the best parts of the engine, which we are proud of, sometimes contained pieces of code and some rudiments of 5, 7 and sometimes even 10 years ago. Many architectural solutions were designed for devices of that time - starting with a 3G iPhone. Now we focus on at least the iPad Air 1 and similar in power Android devices. Accordingly, the approaches have somewhat changed.
And the most common question: why own engine? In the last article, there were several arguments of varying degrees of persuasiveness. I want to concentrate on the main thing: only our own technology can allow you to get the most out of iron, make the maximum number of optimizations specifically for your gameplay, visual style. We position ourselves, including as a technology company, not only a game developer. We believe that with our level of engineers and our experience we can seriously compete in the market of high-tech mobile products.
And now to the point: what tools and techniques have helped us accomplish this rather ambitious task in a short time?
We chose Atlassian Bitbucket Server + Jenkins. In the Bitbucket lies the main repository (master), to which Jenkins is connected. Each developer has his own fork. A new fork in the fork is created for each task, which is integrated back through the pull request. In general, the scheme is pretty standard. Each pullrequest undergoes a mandatory review and automatic tests. And, if successful, it automatically merges into the master.
Jenkins has several shortcomings: he is an ancient, not very fast, gluttonous, web-muzzle looks like an Internet portal from the 90s. However, its flexibility, a huge number of modules and free-of-charge make it a good choice even in 2019. Having played with the modules and customization, you can achieve a digestible appearance, declarative description of pipes (lying in the repository). By the way, there are about 40 pipelines now: tests, editors, a game for all platforms; work with server infrastructure and metagame. Collect it all 20 buildagentov.
In the future, of course, I want to try modern hipster solutions, for example GitLab or self-hosted TravisCI. We do not consider completely cloud solutions (Nevercode, Bitrise, CircleCI, etc.) due to the large size of our repository, assets, and, accordingly, the assembly time and the size of artifacts.
The main system requirement was as follows: project generation for iOS, MacOS, Android, Windows, Linux in one script. We managed to try Premake, SCons, Bazel and CMake. For various reasons, we stopped at the time-tested CMake.
In recent years, CMake has become almost the standard for C ++ libraries. Almost everything from abseil to SDL can be connected to your CMake project in just a few lines. There are of course exceptions, like OpenSSL or V8, with which I had to sweat a bit. On top of the naked Zmeik we developed a small framework (about 3,000 lines in total). Key features:
Modularity. The individual parts of the engine are designed as modules. For example, sound, UI, physics, network, etc. Each module can have its own assets (for example, shaders) and may have dependencies on other modules.
The final application on the engine (game, editor, utilities) connects only those modules that it needs. A little apart is the core module, which is a dependency for most other modules. Core implements the entry point, the main application cycle, interaction with the operating system and other basic entities.
Thirdparty modules. Our framework allows you to download a git repository or archive in several lines, unpack, compile, copy libraries and / or sources. Today we have 66 such thirdparty modules: analytics, third-party file formats, middleware like physics, a sound library, etc.
Based on previous experience, we decided to add both the engine and the game into one repository. This allows you to painlessly make changes to the engine API and synchronously adapt the game for it. The result was the so-called monorepository with its advantages and disadvantages. But, since we immediately planned to maintain a very high pace of development, the possibility of synchronous refactoring of the engine and the game outweighed all the other disadvantages of this solution.
On average, we add more than 20 pull requests per day. This means that the master can potentially be broken 20 times a day. Fortunately, back in 1991, they came up with the Continuous Integration technique. What have we come to?
As mentioned above, a brunch is created for each task in the fork of the developer. Next, a pull request is created from this brunch to the main repository. This pull request passes a series of automated tests at Jenkins:
Most projects for tests are collected with the “treat warnings as errors” flag turned on, and on the MacOS platform with the clang AddressSanitizer additionally turned on, which allows you to catch even more errors at the stage of preparing the pull request.
In addition to tests, each pull request is reviewed by at least two other developers and, if necessary, is sent for revision. When all the tests have been completed and the reviewers have no comments, the pullrequest automatically freezes.
Since some tests can take considerable time (for example, a full GUI editor test lasts more than an hour), a shortened script is used in pull requests. The full set of tests is run in the wizard every 4 hours.
To date, 6,600 pullrequests have already been created and merged.
We use the concept of automatic daily (or rather daily) releases. How exactly does this happen:
Of course, not every nightly release ends successfully. Some tests may fail, or a critical error will occur at application startup. In this case, the problems found are repaired by the developer on duty during the day and the release process is restarted.
There are several on duty every day:
Also on the day of duty, you can engage in technical debt: add the missing tests for the old functionality, supplement the documentation, or do refactoring, which does not take the time with the usual release planning.
In the next article, we will take a closer look at the software architecture of the engine itself, as well as the main modules and subsystems.
To be continued…
The first part: habr.com/en/post/461623
Game site
A number of people have doubts: is this the same technology or another? Answer: this is a new technology written from scratch.
How did we manage in just a year? Our team has vast experience. Many have been developing engines and games for over 15 years.
Why from scratch, if you could take our old engine, which also lies in open-source? He is about 10 years old, and most of the code is out of date. Even the best parts of the engine, which we are proud of, sometimes contained pieces of code and some rudiments of 5, 7 and sometimes even 10 years ago. Many architectural solutions were designed for devices of that time - starting with a 3G iPhone. Now we focus on at least the iPad Air 1 and similar in power Android devices. Accordingly, the approaches have somewhat changed.
And the most common question: why own engine? In the last article, there were several arguments of varying degrees of persuasiveness. I want to concentrate on the main thing: only our own technology can allow you to get the most out of iron, make the maximum number of optimizations specifically for your gameplay, visual style. We position ourselves, including as a technology company, not only a game developer. We believe that with our level of engineers and our experience we can seriously compete in the market of high-tech mobile products.
And now to the point: what tools and techniques have helped us accomplish this rather ambitious task in a short time?
Infrastructure
We chose Atlassian Bitbucket Server + Jenkins. In the Bitbucket lies the main repository (master), to which Jenkins is connected. Each developer has his own fork. A new fork in the fork is created for each task, which is integrated back through the pull request. In general, the scheme is pretty standard. Each pullrequest undergoes a mandatory review and automatic tests. And, if successful, it automatically merges into the master.
Jenkins
Jenkins has several shortcomings: he is an ancient, not very fast, gluttonous, web-muzzle looks like an Internet portal from the 90s. However, its flexibility, a huge number of modules and free-of-charge make it a good choice even in 2019. Having played with the modules and customization, you can achieve a digestible appearance, declarative description of pipes (lying in the repository). By the way, there are about 40 pipelines now: tests, editors, a game for all platforms; work with server infrastructure and metagame. Collect it all 20 buildagentov.
In the future, of course, I want to try modern hipster solutions, for example GitLab or self-hosted TravisCI. We do not consider completely cloud solutions (Nevercode, Bitrise, CircleCI, etc.) due to the large size of our repository, assets, and, accordingly, the assembly time and the size of artifacts.
Build system
The main system requirement was as follows: project generation for iOS, MacOS, Android, Windows, Linux in one script. We managed to try Premake, SCons, Bazel and CMake. For various reasons, we stopped at the time-tested CMake.
In recent years, CMake has become almost the standard for C ++ libraries. Almost everything from abseil to SDL can be connected to your CMake project in just a few lines. There are of course exceptions, like OpenSSL or V8, with which I had to sweat a bit. On top of the naked Zmeik we developed a small framework (about 3,000 lines in total). Key features:
Modularity. The individual parts of the engine are designed as modules. For example, sound, UI, physics, network, etc. Each module can have its own assets (for example, shaders) and may have dependencies on other modules.
The final application on the engine (game, editor, utilities) connects only those modules that it needs. A little apart is the core module, which is a dependency for most other modules. Core implements the entry point, the main application cycle, interaction with the operating system and other basic entities.
Thirdparty modules. Our framework allows you to download a git repository or archive in several lines, unpack, compile, copy libraries and / or sources. Today we have 66 such thirdparty modules: analytics, third-party file formats, middleware like physics, a sound library, etc.
Development process
Based on previous experience, we decided to add both the engine and the game into one repository. This allows you to painlessly make changes to the engine API and synchronously adapt the game for it. The result was the so-called monorepository with its advantages and disadvantages. But, since we immediately planned to maintain a very high pace of development, the possibility of synchronous refactoring of the engine and the game outweighed all the other disadvantages of this solution.
On average, we add more than 20 pull requests per day. This means that the master can potentially be broken 20 times a day. Fortunately, back in 1991, they came up with the Continuous Integration technique. What have we come to?
Continuous integration
As mentioned above, a brunch is created for each task in the fork of the developer. Next, a pull request is created from this brunch to the main repository. This pull request passes a series of automated tests at Jenkins:
- Unit tests for all platforms (windows, linux, macos, ios, android). Googletest is used as the basis, and OpenCppCoverage, the report of which is checked by an additional python script, is used to check the percentage of coverage. If the percentage of coverage for a particular file is less than 75%, the test is considered failed. Thus, we have covered most of the low-level engine classes with tests.
- Codeformatter For C ++ code we use clang-format. The formatting of the changed code first automatically occurs when committing on the developer's machine, and then it is checked in the test. For javascript, which is used as a scripting language, npm linter is used.
- Asset Tests. A fairly large group of tests: from validating file formats to checking dependencies (for example, checking that the texture used in the game level really exists).
- Unit and functional tests of the editor. An integral part of the engine is the editor, where game levels and other assets are created and edited. In addition to unit tests, froglogic Squish for Qt, a utility for automatic GUI testing, is used to test the editor. All this allows us to do without manual testing of the editor. Moreover, according to the reviews of artists and level designers, the level of its quality and stability is higher than in the previous company, when we had a team of five testers. At the same time, releases occur daily, and with manual testing, releases occurred every 2 weeks.
- Functional tests of the game. It is clear that I want to use automatic functional tests for the game. Therefore, we began developing the following system:
- test application (specifically - a python script) launches a game server and client with certain parameters
- running server and client open the network port,
- The test application connects to them and sends commands: download the map, select a character and weapons, move to a point, aim, shoot, etc.
- the test syntax itself is python pytest. This system is currently under active development.
Most projects for tests are collected with the “treat warnings as errors” flag turned on, and on the MacOS platform with the clang AddressSanitizer additionally turned on, which allows you to catch even more errors at the stage of preparing the pull request.
In addition to tests, each pull request is reviewed by at least two other developers and, if necessary, is sent for revision. When all the tests have been completed and the reviewers have no comments, the pullrequest automatically freezes.
Since some tests can take considerable time (for example, a full GUI editor test lasts more than an hour), a shortened script is used in pull requests. The full set of tests is run in the wizard every 4 hours.
To date, 6,600 pullrequests have already been created and merged.
Continuous delivery
We use the concept of automatic daily (or rather daily) releases. How exactly does this happen:
- git tag is created,
- it runs full versions of all tests,
- if successful, artifacts are collected:
- editors for MacOS and Windows. Thus, every morning everyone has a fresh version of the tools. And, thanks to automatic tests, we are confident in their certain quality and stability.
- game client and server for all platforms. The client for iOS is uploaded to TestFlight, for Android - to Google Play, other platforms - to JFrog Artifactory, game servers and other services - to the cloud. That is, every morning we have a fresh version of the game, ready for testing and playtests.
Of course, not every nightly release ends successfully. Some tests may fail, or a critical error will occur at application startup. In this case, the problems found are repaired by the developer on duty during the day and the release process is restarted.
There are several on duty every day:
- 1st level attendant. Monitors the stability of tests in the main repository.
- 2nd level attendant on the game. Repairing game bugs.
- 2nd level attendant on editors. Fixes editorial bugs, advises users (artists, level designers, game designers).
Also on the day of duty, you can engage in technical debt: add the missing tests for the old functionality, supplement the documentation, or do refactoring, which does not take the time with the usual release planning.
In the next article, we will take a closer look at the software architecture of the engine itself, as well as the main modules and subsystems.
To be continued…
The first part: habr.com/en/post/461623
Game site
All Articles