Some companies mistakenly believe that automation is a waste of time and money, others - that this is a cool trend and a “pill” for all diseases. Consider where the middle ground is and what is the point of automation.
Why test
The slightest mistake in the software threatens huge costs. The better the development processes are built, the lower the risks. However, if we turn to the facts, we see that even giants such as Google, Microsoft or leading banks make mistakes.
For example, one of the famous “accidents” of the financial world occurred at the First National Bank of Chicago in the late 90s: overnight, customer accounts unreasonably grew by $ 900 million. Then the senior vice president of the bank, James Lancaster, admitted that it was a “software error in the computer” - which, fortunately, no one had time to use.
Costly mistakes occur in other industries that seemingly are not directly related to IT. A well-known example is the accident at the launch of space technology, in particular, in 2017, Roscosmos lost a satellite worth 2.6 billion rubles.
When working with large-scale IT solutions, for example, remote banking systems (RBS), it is important to constantly test not only the operation of individual functionalities, but also their interaction. In the conditions of tight deadlines, when leading banks update their applications every month, it is impossible to check everything manually - at least there will not be enough time for this.
When creating IT products for business, two approaches are usually combined:
- carry out checks manually with the help of QA (Quality Assurance) specialists;
- combine manual testing and automation of key test cases, with the help of experts from SDET (Software Development Engineer in Test) .
The work of the SDET specialist is on the border of three areas - development, QA and DevOps, covering both the direct writing of automatic tests and other tasks. For example, SDET can configure CI to automatically deliver and deploy applications, maintain documentation, and organize processes. However, in this article we consider one aspect - test automation.
What gives automation
This is one of the fairly new directions in development, surrounded by a large number of myths. Most often, a business turns to automation, believing that it:
- will solve all the problems of releasing quality software;
- will allow to refuse manual testing;
- necessary simply because it is a “cool trend”;
- accelerate the release of releases;
- will increase the coverage of platforms and versions of operating systems during testing.
Obviously, the first three statements are excessively categorical, and therefore, as is often the case, erroneous: to ensure the quality of an IT product, it is important to consider a complex of many parameters, and not just test automation.
In turn, the last two statements are really true: competent test automation helps both speed up releases and increase case coverage. At the same time, manual quality assurance and automation are, as a rule, equally important and are used in combination. In addition, each project is individual, and sometimes automation is not necessary.
When automation is required
- Large-scale application with many business functions
- Significant life of the application (from 1 year or more)
- Implementation of CI / CD, regular releases + a small number of QA specialists
Automation Tasks
As a rule, we engage SDET experts to solve the following problems:
- Automation of routine and frequent checks, reducing the burden on QA specialists.
- Monitoring the main functions of the application and tracking changes in the product.
- The ability to test with a large number of devices, browser versions and operating systems.
- Testing application performance in conditions of simultaneous work with a large amount of data and users.
results
Automation helps build balance:
- manually check what requires human attention (usually up to 25% of cases);
- Automate other cases.
At the same time, automation in the long run reduces both testing costs and the risks associated with the human factor.
In addition, if necessary, you can speed up releases. For example, if you need to check about 400 cases in a development sprint, then manual verification will take up to two weeks, and autotests can be performed at night and analyzed in 4 hours.
Thanks to automation, a business gets the opportunity at any time to make sure that the key functions of the system are working correctly and to check for errors (and if there are any, what are they).
Example
Suppose that at the moment in a mobile bank you need to go through up to 700 cases, each from 70 to 100 times a year. Less than 25% of cases require manual verification, the remaining 75% can be automated.
Time spent on manual verification:
- 30 hours
Automation time:
A night test run takes 8 hours, but without human intervention, and therefore is not taken into account.
Other time costs:
- 8 hours for manual verification of cases that cannot be covered by self-tests (25%);
- 6 hours to analyze the results, as well as, if necessary, to check for failures (up to 10% of tests).
Total: test automation reduces the time spent from 30 to 14 hours .
Of course, each case has its own characteristics, so the time spent can be different. We constantly analyze how much time it takes to write automatic tests, support, analysis of the results.
On average, automation saves from 30 to 50% of the time at least , allowing you to allocate more time, for example, to develop and improve the product.
How does test automation work?
Each IT company has its own characteristics in testing automation, as in other work processes. At SimbirSoft, we adhere to the following methods that help you quickly build work with our clients' projects.
The testing process begins with the development of a strategy - a test plan, the basis for compiling individual test cases. To automate SDET, engineers choose key, frequently used scenarios of the user working with the product - such scenarios give about 80% of business value.
After that, we create the basis for further auto tests, set up stands and workflow for working with them, CI to regularly run tests on various branches. We choose which approaches to the preparation of test data we will use (API, access to databases, generation of synthetic data, use of data from sales). SDET engineers write tests that cover key scenarios of working with the product, analyze the results and the need for further automation.
We develop automated tests using all the most popular programming languages - Java, Python, Kotlin, etc. Our main tools and technologies are Appium, TestNG | JUnit, RobotFramework | Pytest, Selenium | Senenide, Allure, TeamCity, Jenkins, JMeter.
What tests need to be automated first and foremost depends on the characteristics of a particular product. Most companies automate smoke tests, regression tests to check ready-made functionalities, and cases to check various parameters (for example, valid and invalid data during registration).
In our practice, we build testing automation processes already at the start of development, in parallel with manual testing. This work is not one-time, it is carried out continuously, especially on large projects, including banking.
To summarize
The balance of manual and automated testing allows you to constantly monitor the quality of the IT product. Some projects are checked manually; other tasks can be successfully solved with the help of automation. Manual testing is used in cases where people are irreplaceable, for example, if you need localization, description of errors, manual usability testing. In small projects, developers often write tests.
Manual testing in conjunction with automation is carried out when creating large IT-products, where several teams work (for example, in banking applications), where there are complex algorithms and business logic. This method helps to build product testing processes and reduce the risk of costly errors, which is especially important when working in a tight release schedule.
Thanks for attention! We hope the article was useful to you!