Choosing a test management system in 2019

Despite the enormous capabilities of modern table editors, for many development teams, sooner or later the question arises of introducing specialized tools (systems) to manage the testing process in their projects. Which one to choose?







Such a question in my practice has been asked more than once. This prompted me to create this review article. In the future, to save a little time for ourselves and all those who also face this issue. And even if there is no answer to it at the end of the article (sorry for the spoiler) , but your subconscious, most likely, will be determined right now and at the right time on a specific project will prompt an answer.



In my searches, I came across a lot of similar articles (in one of which I borrowed the structure of the comparison table), but they all sparingly list functions from the sites of these systems and 1-2 screenshots of the interface and often contain hidden or open advertising of one of such tools. I wanted to test each system “on myself” and openly share what I saw, tell about my feelings.



Disclaimer
The article is only my subjective opinion, some points could change (given the time elapsed from the idea to publication, - 9 months) . I have no relation to the listed companies and products, all screenshots are presented for informational purposes. There are a lot of anglicisms in the text. The position in the table or text is not the position in the ranking, you determine the “best” and “worst” for yourself.

A very similar article came out just a month ago from another author - you don’t have deja vu, but this does not exclude the fact that in the parallel world (if you chose the blue pill) you already read my article.


Below is the final comparative table (clickable), after which the reader, not in a hurry to meet, will find a few words about each system.







Paid tools:



Testral

QA Touch

Test collab

qTest Manager

PractiTest

Zephyr Standalone

Test lodge

Easyqa



Open source tools:



Testlink

SquashTM

Kiwi TCMS




Important:

Screenshots of interfaces are clickable and hidden under a spoiler. You can find even more screenshots on my website . Nobody likes self-promotion, so if there are examples of articles or a way to quickly and conveniently add similar image galleries to an article on Habré, tell me in the comments or in a personal message. I will try to redo and then remove this remark.




Testral



Website: www.gurock.com/testrail

Developer: Gurock Software GmbH

Price: TestRail Cloud $ 30 per user / month (1-20 users)



Pleases: convenient and intuitive interface; quick insertion of images; Markdown text markup; Adding custom fields setting the display order of the case fields; the ability to assign each case (or testran) to an individual user; quick copying and moving cases; grouping several testranes into a test plan; fast integration with bugtrackers (by tracker URL or using plugins); export of both cases and reports in XML and CSV formats; Extensive API capabilities for integration with your internal systems (synchronization of information in cases, saving the results of autotests, etc.); visual reports; neat printing forms; When creating a project, you can choose three types of structures for storing test cases (one of which allows you to divide cases into branches that correspond to development).



Not happy: no built-in requirements; there is no bugtracker; the price is quite high (especially for small companies).



Screenshots of the interface:








QA Touch



Website: www.qatouch.com

Developer: DCKAP

Price: Professional $ 69 for 50 users / month



Pleases: wide available and planned opportunities at an adequate price; nice UI; built-in bugtracker and requirements; integration with external bug trackers JIRA, GitHub, GitLab; release planning of testranes; there is the possibility of creating a re-run (re-run) with cases of a certain status; voice dialing and text completion of the case title and bug; reusable case steps.

Unfortunately, I could not try and take a screenshot of some components of the product, because they are available only in paid plans.



Not happy: case components (main fields, steps, attachments) are spread out into tabs when editing; no text formatting; the picture, although it can be added to a single step, but only to one and as a downloadable file; no case versioning; text areas of steps and expected results, when changing their size manually, get lost during editing (not on the same line); the form of selection of cases for testran is slightly different when creating and editing it; the list of cases and bugs includes only their name and ID (for a quick overview of the list, additional custom fields would not be superfluous); the status mark during the run is also placed on a separate tab from the general information of the case; there are small shoals in the displayed data (the total number of cases on the dashboard and the strange field “Assigned to” - I attribute this to the fact that I registered back in that year, since then the vendor has substantially supplemented it ).



Screenshots of the interface:








Test collab



Website: testcollab.com

Developer: Test Collab

Price: Startups from $ 25 per user / month (over 50 users - $ 10)



Pleases: a wide configuration of mail notifications (with the ability to edit letter templates); combining testranes into test plans; nice interface with hints (which can be turned off); reusable steps function the ability to comment on the case while editing; tags selection of text formatting type (Markdown, HTML / WYSIWYG, Plain Text); Adding custom fields customizable templates for selecting cases in testran; built-in requirements; price (for small projects there is a free plan with 200 cases and 400 completed cases).



Not happy: you cannot edit system fields (in particular Priority); there is no way to save the displayed fields in lists (when updating pages are reset by default); there are small flaws in the display of UI elements; there are no printed forms either for cases or for reports (probably now they are rarely used, but other tools offer such a function) ; data on the general dashboard is updated with a delay; export of cases for only one test set (in this case, nested sets only process export to XML); inconvenient insertion of images into the steps of the case (via the link); there is no bugtracker.



Screenshots of the interface:








qTest Manager



Website: tricentis.com

Developer: Tricentis (acquired QASymphony in 2018)

Price: from $ 99 per user / month (the price is approximate and depends on the number of users)



I am glad: setting up system fields and adding custom fields to various objects (with the ability to preview this object and even change colors for the status of the run); informative case change history; built-in HTML-editor in the text fields of the case; commenting on a single case; You can subscribe to a separate case and receive notifications by mail about changes; An interesting implementation of case auto-versioning (with major and minor components); function of quick run of the case (without setting the status for each step); built-in requirements with case binding and an indication of the target assembly; there is also a built-in bug tracking; reusable cases; customizable testran configurations; ample opportunities for integration with both bugtrackers and CI / CD services; API for even more flexible interaction; Stored search queries by case, requirements, etc .; various reports; flexible settings on the administrator side (users, notifications, rights, groups, etc.); embedded project with all the data to familiarize yourself with the capabilities of the product; test plans.



Not happy: no printed forms of cases; export cases only in Excel format (but it has a readable look); slightly inconvenient viewing of pictures in the steps of the case during the run; the site does not have information about the price of the product (you need to make a separate request) and it is impressive.



Screenshots of the interface:








PractiTest



Website: www.practitest.com

Developer: PractiTest

Price: ENTERPRISE $ 49 per user / month (minimum 3 users)



Pleased: the ability to comment on the case (test); tags (keywords) for the test; quick jump to the test number; generation of permutations of testranes (for different testers, versions); the use of parameters in the steps of the case (with the ability to change the value for each run); inclusion of steps (call to test) of one case in another (sometimes it is really necessary and you do not have to jump to links to other cases) ; two-way integration with Jira and YouTrack (by creating an API token); Adding custom fields kanban status board for cases; their bug tracker and requirements (but did not take screenshots with them).



Not happy: complex UI / UX as a whole (in particular, the division of the case structure into tabs: description and steps on different tabs); the meager, unpresentable print version of both the cases themselves and the reports; although there is support for markdown markup, but pictures are added to steps only as attachments; you cannot change the status of the entire case if some step has not passed (this is correct, but sometimes you need to "green" the whole test) ; the user list cannot be given the priority of sorting (sorts alphabetically); price (overpriced, even considering the unique features); support chat widget on each page.

The company's website at the time of publication was not available from the Russian IP, while the web platform was opening.



Screenshots of the interface:








Zephyr Standalone



Website: www.getzephyr.com

Developer: SmartBear Software

Price: from $ 50 per user / month (price taken from the past, now not listed on the site)



I am glad: attractive planning of testing stages (similar to a Gantt chart) with the addition and customization of various testing works; built-in requirements and bugtracking, which, with JIRA, seamlessly integrate with it (with the ability to search and view tickets directly from Zephyr); it is possible to add custom fields to objects (project, cases, requirements), change the name and color to the status of the run (the whole case and a single step); You can add widgets for the project with various characteristics (diagrams) that will be updated after a specified time; export cases in HTML and PDF format with the necessary fields.



Not happy: unusual interface, you need to get used to UX / UI for a while; there is no direct link to a separate case (only through a search query you can generate a URL for it); the dashboard with the help widget below is not removed; You can not change the system fields; Pictures for the steps of the case can only be added as attachments to the entire case; sharpening only under JIRA; The price must be found out upon request (and only by phone, this information is not provided in the electronic message) .



Screenshots of the interface:








Test lodge



Website: www.testlodge.com

Developer: TestLodge Limited

Price: Basic $ 49 for 600 cases / month (subscription by amount of data)



Pleases: concise interface; creation and selection of a configuration when creating a testran; quick integration with popular bug trackers; creating a test plan (with a default template); coverage matrix (if requirements are kept inside the product); subscription model by the amount of data (for some projects this is a plus); set export to HTML (it is also a printed form).



Not happy: there is no division of the case into separate steps; user fields cannot be moved above system fields; a picture can only be inserted as an attachment or a link to it; export of cases only each set separately; a test suite can be specified for only one test plan; there is no bugtracker.



Screenshots of the interface:








Easyqa



Website: geteasyqa.com

Developer: ThinkMobiles

Price: $ 10 per user / month



Pleasing: the ability to monitor and test mobile assemblies (APK and IPA files), integrating with GitHub or GitLab repositories; own bug tracker with the ability to add a plugin for integration with other systems (GitHub, GitLab, YouTrack, JIRA, RedMine, Pivotal Tracker, Trello); ease of use (sometimes it’s a plus) with test objects (plans, testranes, test cases, reports); visual comparison of fields when exporting cases from a CSV file (but requires attention, you can go into an endless export cycle).



Not happy: there is no way to at least somehow customize objects; there is no expected result for each step of the case; cases cannot format text and attach pictures; there are questions to the UI in some places (especially when displaying tables); did not receive an invitation letter for the second participant to the mail from a personal domain; you can not change the rights for roles (+ could not change the project role for an unacknowledged user).

Judging by the activity on social networks and the blog on the site, they suspended the development.



Screenshots of the interface:









Free systems attract many by the lack of a monthly fee. But they will certainly require certain costs. Firstly, hosting rental (computing resources). Secondly, the labor costs of the system administrator to install, configure and maintain the system’s health. But you can already have a server (or a small virtual machine), and the availability of ready-made Docker images simplifies deployment, so starting a free system from scratch can be easier than registering in a paid system (I exaggerate, of course, but there is some joke in every joke ) Add to this the trend of “total control” over data, and open source products are becoming more attractive for implementation. I reviewed three such systems: TestLink, SquashTM, and Kiwi TCMS.




Testlink



Website: testlink.org

Development Language: PHP



Pleases: built-in requirements and tracking them through cases; various text reports; the ability to add custom fields; flexible customization of user roles; integration with bugtrackers (JIRA, YouTrack, GitLab, Bugzilla, etc.); testlid can additionally set urgency for each case in testran, which will affect the order of cases; management of the list of tested platforms; inventory of hosts (a trifle, but nice) ; HTML editor with the ability to insert pictures, links, tables, lists; in my opinion, the most popular Open Source tool with many instructions and articles on setting up and using; the presence of a development resource in the PHP language allows you to change the product to your needs (if standard functions seem insufficient or they turn out to be not so convenient).



Not happy: a little addictive interface; you will have to tinker with configuring some components for full operation (fine-tuning of configuration files, sending mail, integration with a bug tracker); in the text windows of the HTML editor, the standard spelling check of the browser does not work; you can not edit system fields and the order in which they are displayed on the case form; manual change of the order of steps (you have to enter numbers, instead of dragging and dropping); assignment of cases for testran (build) after paid tools will seem uncomfortable.



Screenshots of the interface:








SquashTM



Website: www.squashtest.org

Development Language: Java



Pleased: requirements management with reference to test cases; HTML editor use of parameters in case steps and preconditions; the ability to create a test case with a Gherkin script ( BDD ); UI function of folding the contents of the steps of the case; there is a mode for running cases with the display of the tested website in the frame (in most cases it is no longer relevant, but there is such a possibility); use of an existing case (Call a test case); Adding custom fields to objects color differentiation of product components; a color status indicator in the case library also seemed convenient; plugins for integration with bug trackers JIRA and Mantis; there is detailed documentation (though some sections are in the wrong order) .



Not happy: to attach a picture in an attachment, you must first add extensions for such files to the white list of system settings (by default png and jpg are missing); You cannot edit system fields (category and priority); you cannot edit the available seven roles of access rights (however, 7 levels of this are often enough); I couldn’t generate some reports (I learned from the logs that there was a problem with the initialization of the class); unusual terminology for test plans and testranes (here they are called Campaign and Iteration); Tab UI.



Screenshots of the interface:








Kiwi TCMS



Website: kiwitcms.org

Development Language: Python



Pleases: flexible configuration of user group rights; integration with bug trackers JIRA, Bugzilla, GitHub and GitLab; quick system operation; Markdown text markup with the ability to quickly preview; the ability to connect plugins (some, however, only in development plans) to collect the results of autotests; External API detailed product documentation; Active development of new features and plans for the future.



Not happy: in the test there is no division into separate steps; some counters in the UI are not updated after the action is completed (you need to refresh the page); own spelling highlighting in a text editor; one report on the created cases (but plans to add a few more); no dividing into projects within the system.



Screenshots of the interface:








Conclusion



This is only a small part of the systems available on the market. However, the review is already getting rather long. Plugins for JIRA ( TM4J , Zephyr , Xray , TestFLO , etc.) were not included in it, because these are birds of different flight. JIRA, although it is now used on many projects, but still it is not a panacea. Therefore, this is a topic for a separate, no less article. Also, Test IT didn’t fall under the gun, but it came out recently when the main part of this article was already written. Perhaps, during this time, more systems were "born" or there are already existing ones worthy of a review - write in the comments. It seems people love TV shows - write a sequel!



In the meantime, I propose to participate in the discussion “What is the testing management system on your project?”.



(I understand that this is not always the “desired” system, then share your pain: with which you work, and with which you would like to work.)










All Articles