Features of testing mobile MMO
Recently I had a chance to chat with Alexei Nelyubov, QA Director of Datcroft Games. Now the guys are working on the mobile MMO Action Pixel Wars, the project is in the soft-launch stage. The testing department accompanied the game at every stage of its development, and I decided that a good article on the hub would come out of Alexey’s story.
Next is direct speech.
Pixel Wars has a long and difficult story. Guided by business goals, we began assembling the first prototypes of the game in 2016. Subsequently, the concept was completely redesigned taking into account the changing realities of the market, and the Pixels came out in the soft-game at the end of 2018. In the near future, a commercial release of the project is planned.
The leader of the department distributes tasks, is present at all meetings, creates test plans, sets priorities for tasks, and also acts as a test analyst.
Next, there are three experienced employees who have stood guard over the quality of the game for more than a year. One of them has recently sat down tightly on autotests - it creates scripts on cases on a self-written tool.
Recently, a young and promising specialist joined our team (at first he came to us for training courses, then he got an internship - and now he is on staff), which already solves many interesting problems, from checking maps to writing simple automated scripts. And soon we are waiting for another one.
We have a large fleet of test devices, from old life-beaten phones to ultra-modern devices with the latest OS and functionality. Sometimes it’s interesting to catch a bug on only one tablet, because it has a special hard drive, and then try to recreate the appropriate conditions in the development studio.
As you know, in iterative models, testing begins with the completion of the formation of technical specifications and high-level design. So it works with us. We read the technical specifications before they begin to implement it, look at the layouts of new interfaces, check the gray boxes of cards, look at the math calculations in the documents of game designers.
At the time of implementation, we do not wait until all edits are uploaded to master, but we are actively testing individual parts of the functionality in separate branches. They may lack beautiful graphics, a pink cone may run instead of a character, and gray barrels will be on the map instead of trees and bushes. But we can already check the functionality.
There are frequent cases of pair testing with a developer on his computer - it greatly saves assembly time. If a programmer usually looks only for the functionality that he made or corrected, and only positive cases, then the tester can immediately find bugs in negative cases, plus from experience know what this programmer in this module could break. Analysis of influence also helps: for example, whenever a guild code rules, something happens to friends' functionality. Why not check this moment right away?
When all the individual pieces of code, graphics, effects, sounds and localization are merged into the master, the time comes for integration and system testing. We look at how new features work with each other, and conduct mass tests with the whole team. Often, bugs like “I'm uncomfortable” or “I don’t feel interest” get out here, which are also recorded and processed. We look not only at the functional characteristics, but also at how the game behaves under heavy load, “bad” Internet and in various non-ideal environments, such as a bus ride through the city or the basement of a business center.
And now, all obvious bugs are fixed, all new things are ready to see the light. Can I upload to the prod? Of course not. Need the opinion of representatives of all departments and some real players. At the same time, regression begins (the time of large autotests) and acceptance testing. If everything is clear with the first one (we check all the functionality of the game, add outdated cases, add new ones, check them too), then few people talk about acceptance. But the thing is useful. We send the build to the project’s general channel and urge everyone to play and evaluate the game “from their bell tower” for a couple of days. It may turn out that game designers didn’t represent the feature of the pit with stakes at all, although everything was done according to the description and layouts, and marketing was needed another skin of the warrior character for the action on the social network.When all the wishes are taken into account, we go to the producer, the main person on the project, he gives the go-ahead, and then the process of uploading the version to the prod starts. At the moment, we are only on Google Play. the action takes half an hour to about an hour.
When everything is ready, we do not immediately remove the plate of professional work. First you need to make sure yourself that the ratings are drawn, payments go through, games are launched, and new features have got to the players.
Next, the process begins anew.
Often, in order to test a game, it does not need to be launched. There is existing mechanics in which some parameters were changed. Great, download the fresh repository, look at the corresponding xml-file and close the task. Sometimes bugs do not want to be repeated. For example, one of the partners played in our project in the subway, at this time his phone rang, and the Internet from 3g switched to wifi without access to the global network. At the same time, the device ran out of space and the battery overheated. But we have no irreproducible bugs, there are those to which hands have not reached yet.
Of course, it happens that bugs get out on the prod. And if users found one, then we most likely found it too. It just has a low priority or reproducibility, so the fix was postponed for two months.
It’s rare, but it happens to prove to a programmer or PM that a bug is a bug and it needs to be fixed right now. For example, on the cards you can go into the safe zone of the enemy. Yes, you can’t attack there, but this is a great way to restore health and ambush the enemy. He proved - repair. No - look for more compelling arguments. Arguments in the style of “cool” or “I do not like” are not accepted, concrete examples with concrete consequences are needed.
Sometimes we had to test disconnects, running out onto the balcony in the office, until there was a “bad” Internet setting on the router.
To simulate a full network and test the application in such conditions, the tester specifically walks around a crowded shopping center with a telephone.
Many bugs are caught when you make a disconnect and quickly switch something. Here the speed of tapes on the screen is really important. For example, they found a completely non-obvious mistake: when you quickly switch classes and character skins, in the end your archer holds a shield or warrior with a magic wand.
Thanks for attention! I hope you were interested in learning more about the game testing process.
Next is direct speech.
Pixel Wars has a long and difficult story. Guided by business goals, we began assembling the first prototypes of the game in 2016. Subsequently, the concept was completely redesigned taking into account the changing realities of the market, and the Pixels came out in the soft-game at the end of 2018. In the near future, a commercial release of the project is planned.
Who is in the team?
The leader of the department distributes tasks, is present at all meetings, creates test plans, sets priorities for tasks, and also acts as a test analyst.
Next, there are three experienced employees who have stood guard over the quality of the game for more than a year. One of them has recently sat down tightly on autotests - it creates scripts on cases on a self-written tool.
Recently, a young and promising specialist joined our team (at first he came to us for training courses, then he got an internship - and now he is on staff), which already solves many interesting problems, from checking maps to writing simple automated scripts. And soon we are waiting for another one.
We have a large fleet of test devices, from old life-beaten phones to ultra-modern devices with the latest OS and functionality. Sometimes it’s interesting to catch a bug on only one tablet, because it has a special hard drive, and then try to recreate the appropriate conditions in the development studio.
What tasks do QA-specialists solve?
As you know, in iterative models, testing begins with the completion of the formation of technical specifications and high-level design. So it works with us. We read the technical specifications before they begin to implement it, look at the layouts of new interfaces, check the gray boxes of cards, look at the math calculations in the documents of game designers.
At the time of implementation, we do not wait until all edits are uploaded to master, but we are actively testing individual parts of the functionality in separate branches. They may lack beautiful graphics, a pink cone may run instead of a character, and gray barrels will be on the map instead of trees and bushes. But we can already check the functionality.
There are frequent cases of pair testing with a developer on his computer - it greatly saves assembly time. If a programmer usually looks only for the functionality that he made or corrected, and only positive cases, then the tester can immediately find bugs in negative cases, plus from experience know what this programmer in this module could break. Analysis of influence also helps: for example, whenever a guild code rules, something happens to friends' functionality. Why not check this moment right away?
When all the individual pieces of code, graphics, effects, sounds and localization are merged into the master, the time comes for integration and system testing. We look at how new features work with each other, and conduct mass tests with the whole team. Often, bugs like “I'm uncomfortable” or “I don’t feel interest” get out here, which are also recorded and processed. We look not only at the functional characteristics, but also at how the game behaves under heavy load, “bad” Internet and in various non-ideal environments, such as a bus ride through the city or the basement of a business center.
How is the version accepted?
And now, all obvious bugs are fixed, all new things are ready to see the light. Can I upload to the prod? Of course not. Need the opinion of representatives of all departments and some real players. At the same time, regression begins (the time of large autotests) and acceptance testing. If everything is clear with the first one (we check all the functionality of the game, add outdated cases, add new ones, check them too), then few people talk about acceptance. But the thing is useful. We send the build to the project’s general channel and urge everyone to play and evaluate the game “from their bell tower” for a couple of days. It may turn out that game designers didn’t represent the feature of the pit with stakes at all, although everything was done according to the description and layouts, and marketing was needed another skin of the warrior character for the action on the social network.When all the wishes are taken into account, we go to the producer, the main person on the project, he gives the go-ahead, and then the process of uploading the version to the prod starts. At the moment, we are only on Google Play. the action takes half an hour to about an hour.
When everything is ready, we do not immediately remove the plate of professional work. First you need to make sure yourself that the ratings are drawn, payments go through, games are launched, and new features have got to the players.
Next, the process begins anew.
Some more facts
Often, in order to test a game, it does not need to be launched. There is existing mechanics in which some parameters were changed. Great, download the fresh repository, look at the corresponding xml-file and close the task. Sometimes bugs do not want to be repeated. For example, one of the partners played in our project in the subway, at this time his phone rang, and the Internet from 3g switched to wifi without access to the global network. At the same time, the device ran out of space and the battery overheated. But we have no irreproducible bugs, there are those to which hands have not reached yet.
Of course, it happens that bugs get out on the prod. And if users found one, then we most likely found it too. It just has a low priority or reproducibility, so the fix was postponed for two months.
It’s rare, but it happens to prove to a programmer or PM that a bug is a bug and it needs to be fixed right now. For example, on the cards you can go into the safe zone of the enemy. Yes, you can’t attack there, but this is a great way to restore health and ambush the enemy. He proved - repair. No - look for more compelling arguments. Arguments in the style of “cool” or “I do not like” are not accepted, concrete examples with concrete consequences are needed.
Sometimes we had to test disconnects, running out onto the balcony in the office, until there was a “bad” Internet setting on the router.
To simulate a full network and test the application in such conditions, the tester specifically walks around a crowded shopping center with a telephone.
Many bugs are caught when you make a disconnect and quickly switch something. Here the speed of tapes on the screen is really important. For example, they found a completely non-obvious mistake: when you quickly switch classes and character skins, in the end your archer holds a shield or warrior with a magic wand.
Thanks for attention! I hope you were interested in learning more about the game testing process.
All Articles