Allure TestOps - Avis

Using Allure TestOps 3.193.0

Commentaires : Overall I am very happy with Allure TestOps. I am using it to improve the manual testing experience for testers and integrate reporting with automated tests. My experience has been with the 3.193.0 version. Most of the issues I have experienced are fixed/improved by the recent 4.x release. I'm looking forward to upgrading.

Avantages :

- Handling both manual and automated tests with a consistent UI is fantastic - Manual test development and execution is a breeze - Automation integration with Jenkins is easy - It is easy to learn and use - Automated tests executed with the allure-framework (https://github.com/allure-framework) provides rich reporting in TestOps

Inconvénients :

- There is no way to modify test result messages for manual tests - There is no way to provide pre-defined test environments for test plans. They must be entered manually each time the test plan is launched. - I've experienced some API bugs that I am able to workaround, but they were unexpected. Support worked with me to resolve the issues. - Custom field and environment definitions are globally scoped (applies to all projects). This is confusing when I want to manage projects differently, but the options for one project are shown in another.

Unique solution on the market

Commentaires : I think this is a good solution overall that eliminates a need in other software for Quality Assurance department.

Avantages :

Single point for all QA-related activities (test case management, automated executions, reporting, analytics, etc.) It helps in collaboration between different departments. Brings great visibility on overall QA department activities to management. Customizable dashboards!

Inconvénients :

Configuring dashboards quite challenging. Dependency on the Allure framework that is not part of the TestOps customer support, so If you got strike with a problem you have to solve it on your own, or ask in Telegram chat where I did not get any response in most of occasions.

Pros and cons of Allure TestOps

Commentaires : We use a tool for manual testing and automation.
We store test cases, test plans in it, use tags and other types of sorting.
We also store test cases for automation and reports on the passage of tests in it. Recently we added a video to the reports, which is very convenient!
A good tool to help you test and improve processes.
I would like it to become more stable, to better integrate with iOS.

Avantages :

- Conveniently build a system for working with test cases and test plans - Integrates well with Android - Conveniently sort test cases - It is possible to add dashboards, watch metrics - Autotest reports can include both screenshots and videos - Quite easy to use tool - There is an opportunity to interact with the developers of the Allure TestOps

Inconvénients :

- The release note on the site is not updated in a timely manner - Poor support for integration with ios, not given due attention to this platform, few examples in the documentation - It takes a long time to complete feature requests that are written to the support service for fairly simple features. It would be nice to see a public road map - It would be cool to see tutorials on YouTube as a Allure, because there is not much in the documentation (restarting autotests for example)

Most useful test reporting tool

Avantages :

The product have complete integration with most programmings languages and testing frameworks; good support by email and Telegram chat; wide community which give helpful responses for questions or troubles; easy integrations with CI/CD.

Inconvénients :

The new versions could break old functionality (regression).

The Best TMS system

Commentaires : I used it for automation mostly but I really like metrics and test-case generation

Avantages :

When you generate test cases from the code. Test case management - many filters and etc. Easy to connect with your code. Works with any languages. Design of the system!

Inconvénients :

Detailed documentation in English. That's it!)

Outstanding visibility into everything test related once you reach a large volume of tests

Commentaires : We upload a ton of tests (~2M per day) and we want to maintain the history and/ ownership of test execution from our CI (Jenkins). When you have that many tests, the existing unit test reports built-in into Jenkins no longer cut it. Short of deploying something of our own, there is not much out there that allows engineers to flexibly decorate tests with metadata and integrate with CI effectively.

Avantages :

The software is written for the engineers. It is very common to offer a solution for the managers. Allure TestOps approaches it from the bottom up. By leveraging existing built-in plugins used by Allure echo system, you can provide all metadata you want to your tests dynamically. Test results are generated per test case, so the daemon picks up and displays test results in real-time. Lots of thought went into structuring test views and the organization of test cases. You can also import manual test cases and slowly automate them, and this workflow is supported out of the gate. We upload ~2M tests per day, and the system is capable of digesting it with the right scalability deployment. It also looks like the team chose a very modern stack: you have a variety of options to deploy and scale the product for your needs.

Inconvénients :

We have been running Allure TestOps from the very early days of availability and have seen significant improvement in maturity: both in the software and documentation. Nevertheless, it is not for the faint of heart: sometimes you feel like you are flying an airplane that is still undergoing a transformation. The development team has been extremely responsive to our support requests.

Good product with helpful features

Commentaires : We collected all our cases (automated and manual) in one place, restructured our tests and received always up-to-date documentation. We got the opportunity to analyze information on launches and runs results. Easy way to work with flaky tests. And we continue to change our processes further.

Avantages :

Allure TestOps helps us to rework our testing approaches and fix processes. Most impactful features are: updating test cases from launches -> test cases are always up to date, it is easy to work with flaky tests and issues (defects). Need time to understand how it works, for me after some days and changing configuration it is easy to use it. It was not easy to integrate with our business processes, but it didn’t depends on Allure TestOps. We change our approach and see that it is better to use some features by another way, and the tools helps how you can do it)

Inconvénients :

Features that would be nice to have: - layer of automation in Jira from Allure TestOps (to see coverage via Jira for PO, PM. A lot of managers want to see more information in one place) - total launch time duration in the launches list (often we are working on TTM improvements and need to see this information in launches) - make it possible to remove or switch off default values for test layers, test statuses, workflow settings (default automated/default Manual) because a user need to choose from the list when sets/changes this values for test cases - make it possible to drag multiple cases from folder to folder via drag and drop - save filters for launches or make it possible to configure fields which user can for filters

Great service to manage your tests

Avantages :

- Allure TestOps is really adding value to the way we build & test software - I think one of the most advantages of TestOps is that it automatically transforms manual tests to automated ones as soon as you push your code - It is built for the engineers, almost everything can be done from your IDE without interacting with UI (besides manual testing ofc) - It has a broad variety of API endpoints that can be used on CI and locally - It is much easier to debug failed tests and keep track of any flakiness or instability with Allure TestOps - It also provides quite good dashboards with plenty of valuable details. There is one dashboard in particular that I like the most, it allows you to import any third-party data (e.g. Sonar Cloud) in markdown format

Inconvénients :

- There are a few tiny bugs on the UI side of TestOps Cloud. We started using TestOps when it was still in beta, so there's nothing to complain about. Although we do notice how with each new release it has become better and more stable. - At the moment, there is no way to showcase your projects publicly / make them publicly accessible (even though there is a checkbox to mark your project as public). I'd say it's the most disadvantage that I could come up with as we are an open-source project

Approach and implementation

Avantages :

This software implements modern approach to test documentation and test automation. Here in Wrike value it a lot. To name those core principles - minimization of manual test documentation, rely heavily on test automation, self descriptive code and test results. In general it supports our approach of quality at speed. I believe that engineering architecture is made based on modern approaches and allows Allure team to provide scalability and performance to match our loads of tens and hundreds thousands tests per day.

Inconvénients :

With our amounts of tests we still experience some issues with performance when test results arrives later than finish of the build. As on mid 2021 application experiencing boom of new clients and have to improve its support chain

Allure for magnificent test reports

Commentaires : As a QA this tool is the best I've ever used for reporting automated tests, I've never seen devs as interested by test reports as with Allure, the reporting of this tool is just magnificent.

Avantages :

Compared to many other reporting solutions on the market, to me Allure is definitely the best. - the test report generated by Allure is very detailed and gives anyone, including non technical people, a great insight of test execution (passed, failed and skipped tests) also gives exactly where the test failed and that is missing in many other reporting tools, making it harder for other people than QAs to understand the test report.- no more test environment confusion- the customer support is very good and responsive- it is used by a lot of users which makes troubleshooting issues a lot easier because of the large community using the tool

Inconvénients :

the initial configuration was a bit challenging since we use in the company CircleCI as our CI/CD server and many people use Jenkins and finding documentation other than the one proposed by CircleCI was a bit challenging

Best choice for QA teams

Commentaires : Triple-A. Luxury TMS ever used.

Avantages :

very convenient when working with automated tests

Inconvénients :

parametrized tests (at least in python). When {param} is used in @allure.title decorator, one of parametrized tests will be displayed in Test Cases section -- not all parametrized ones. It will be more convenient if allure ignores {param} in title or use a new decorator for test cases. Don't know how to implement this.

Tinkoff <3 Allure

Avantages :

- Flexible configuration of the test case tree - Launches. Tree. Restarting failed test cases

Inconvénients :

- Performance - Allure TestOps is difficult for a beginner to dive into

Great test report tool

Commentaires : I have had a good overall experience with Allure TestOps.

Avantages :

Allure TestOps allows users to easily view and analyze the results of automated tests, track test execution, and share test results with team members. It also provides features such as real-time reporting, test history, and integration with popular test frameworks, such as JUnit, TestNG, and NUnit.

Inconvénients :

It has limited support for manual testing, Allure TestOps primarily focuses on automated testing. The reports are not fully customizable.

Great tool for managing & simplifying all testing activities

Commentaires : I would definitely recommend Allure TestOps for QA teams who wants to manage all test related activities & processes in one place.

Avantages :

Allure Testops provides a lof of useful features for test management, tests results analysis, visibility of all testing activities. It works for both automated & manual tests and even has a default flow for manual-automated tests transition. There are all necessary integrations with CI servers, bug/task tracking systems. Also I would like to mention that technical support is ready to answer about all questions and take into account our feature requests.

Inconvénients :

Seems there is a fast-paced development right now & sometimes there could be better documentation. Although, this is easily solved by technical support who is answering very fast.

Advanced testing tool Allure TestOps

Avantages :

Allure TestOps is easy to install and integrate with other systems. One of the best features is selective test run and rerun. You don't need to rerun tests manually and remember what environment did you used. Also I liked Defects feature. It is very useful and powerful tool to manage test results. As well as user support is impressive.

Inconvénients :

What I did not like about TestOps is lack of integration features. At least with systems that we use in our company. Want to see TestRail in TMS integrations and "per user" Jira integration to create issues using personal account. However, support told me that Jira update is coming soon :)

Allure testOps review

Commentaires : Allure helped us manage our test reports for a very scalable system
The support team is very dedicated and responsible

Avantages :

Greate report History Self-documented by code Greate support Easy to use and implement

Inconvénients :

TMS integration not so smooth But overall greate product

Best TMS system

Avantages :

flexible system. Easy to customize for your tasks and processes. Fast work with large amounts of data. New useful features appear every month.

Inconvénients :

Manual test case parametrisation, test cases export and test runs export could be improved.Some modal views and menus is not easy to use, UX could be improved in this parts.

Test Ops visualized

Avantages :

We mainly use Allure to track our UI scenarios. Running release QA It allows us to clearly connect manual and automated ui tests and their results to release candidates. Which in turn allows us to increase our confidence when releasing.

Inconvénients :

Mo real issues with the product were found yet.