Doing the right thing
Modern development cannot exist without automated tests. Tests can be written very quickly even by lay people. Thus, we have tests, everything is as if good. Can we trust such tests? What is the quality of our quality control tools? Let's look at Specflow tests that can be written in human language.
Feature: SpecFlowFeature In order to avoid silly mistakes As a math idiot I want to be told the sum of two numbers Scenario: Add two numbers Given I have entered 50 into the calculator And I have entered 70 into the calculator When I press add Then the result should be 120 on the screen
This test consists of three standard parts called setup, action, and verification. Everything looks good. But what happens when someone adds an extra step to this test?
Feature: SpecFlowFeature In order to avoid silly mistakes As a math idiot I want to be told the sum of two numbers Scenario: Add two numbers Given I have entered 50 into the calculator And I have entered 70 into the calculator When I press add And I press memorize Then the result should be 120 on the screen
Memorization is a common operation in the calculator to write the current number into memory. If we run this test, it will still be green. This is not a good test now, because no one checks how a new step in the test affects the entire system.
This type of tests may appear as a result of a lot of refactoring or significant changes in the terms of reference. This means that we will have all the tests green, but they do not check the performance of the system very well. The test mentioned above can be corrected in two ways: delete this added step “And I press memorize” or add one more step at the end of the test to check the memory status “Then value stored in memory is 120”.
SpecFlowMaster plugin
This plugin for Specflow allows you to find such tests and suspicious test lines. How it works? For each Specflow feature line, it generates a special test that runs the same test, but without this line. If the test execution leads to an error, then this is the expected behavior and everything is fine here. If such a test is successful, then something is wrong with the test line in this test and the system determines this line as suspicious. Those. the new generated test will be red.
More complex cases for background steps or scenarios with multiple data sets are also checked. For precondition steps, the plugin performs all the tests from the file, but without the test line. For tests with data sets, the plugin generates test execution for all data sets.
Technical notes
- C #, VB
- SpecFlow 3.0 or higher
- .NET Framework 4.7.1 or later, .NET Core 2.0 or later
What is planned to be done in the future?
- Support for Before and After attributes
- Similar plugins for Java and Node.js
References
- The plugin can be found on github
- Send any feedback to arskiev@gmail.com