I have been doing test automation for quite some time. And I know firsthand what pain sometimes deliver new versions of anything. We updated Xcode, released a new Selenium, came up with a new browser (special thanks to Microsoft for Edge and its driver), for some reason, here's another programming language for you ... All this makes the automation tool completely joyful of realizing its own significance. After all, only he is now able to run tests on all of this.
Less experienced guys who only comprehend all the basics and mysteries of working with tests, for some reason, do not experience joy. In our automation course, students are often asked how to work with new versions of various components of the stack. I decided to talk about this today. Who cares - welcome to cat.
The essence of the problem
The main problem is that it is impossible to predict the behavior of the system after changing the version of any component of the stack. The funny thing is, no one will clearly tell you that such and such a version of one software will not work with such and such a version of another. All this innermost knowledge is cherished like the apple of an eye.
Good example. If you write mobile autotests in Java using Appium, then you know that there are two components: the java_client.jar library generates HTTP requests, and the Appium Server accepts these requests. If you use java_client version 4, and Appium Server is higher than version ten, then the search for elements by ID will fall off. There will be no error, Appium will not send any warings or other signals that something is going wrong. Just the elements will cease to be. And until you decide to upgrade the java_client version, the tests will not start back. But it was fun to understand all this in due time! :)
A common situation: the new version works with bugs or simply does not make friends with current versions of other programs. It’s bad if in this case we start randomly updating the remaining components of the stack. This will surely entail even more updates and instability.
In inexperienced hands, all this can lead to anything until the complete breakdown of autotests.
Let's think about how to work with updates correctly.
Introductory
Let's say we have a stable working technology stack. In our case, these are Java, JUnit, Java client (Appium client), Appium Server, Selenium library, emulators and simulators (XCode) + the application itself.
A number of tests have already been written on this stack that check the regression of the application before its release. It is important to understand that this is their main task. The release of a broken application can affect its reputation and even lead to financial losses.
New Version Released
Next, one of the components of the stack is updated. This will happen regularly, as our stack includes many programs and libraries.
Do not immediately connect the new version to a combat project. This can break (and most likely break) our tests. And this is already regression testing and probably the time of the release of the next release. You have to repair everything in emergency mode, and nobody needs it.
Instead, you first need to decide on the feasibility of an upgrade. To get started, look at the change list. Then read reviews on the Internet: if something is wrong with the release, posts, bug reports or questions in the style of “Did I break down, but did someone else break down?” Appear very quickly.
In short, we answer the question - do we need this update? And if so, how urgent?
Update
When we understand that an upgrade is needed, we create an update ticket, add it to our planning. This is quite the same task as others: writing a new test, making changes to the infrastructure, and so on. At the moment, the update task may be less priority than others in the backlog - this is also normal.
When we get to the ticket, somewhere “on the sidelines” we are collecting a test project, changes in which will not affect the combat stack. There we install a new version and try to run tests with it. If necessary, make changes to the code or update dependent components and run the tests again. And so until it works. Moreover, it will not work stably!
Important: in the end, you must definitely run the tests again after adding all new versions and making changes to the code.
All this time, the combat stand has been working, fulfilling its main task (regression testing and help in releasing the application) on the old stack. This is normal. The process can take more than one week or even a month.
When we are confident in the new stack, it can be transferred to the combat stand. It’s better to do this as far as possible from the release in time so that there is still time to roll back.
Total
You need to think carefully about updates. There are mandatory steps:
=> Examine the list of updates
=> Planning
=> Run separately on the test assembly
=> Only after stabilization of the assembly - transfer to the combat stand
The answer to a common question. Is it possible to know in advance which versions will work together and which do not? Alas, it is impossible. Only empirically build a stack and try to start on it. Even if the developers wrote that they should work, this is no reason not to test in advance. :)
This is how this article came about. I hope it was useful for someone and will save a couple of hours (or even days) on fixing the falling tests. And if you still do not write autotests, but really want to learn, come to our automation courses. All information about them is in my profile. :)
Thanks for attention!