Quality Gates configured in allurerc.json do not appear to be evaluated when generating an Allure 3 report using the CLI. #3441
|
Summary Quality Gates configured in allurerc.json do not appear to be evaluated when generating an Allure 3 report using the CLI. Environment The report is generated using: allure generate --config allurerc.json allure-results -o allure-report --clean The report is then opened with: allure open allure-report Example qualityGate configuration tested: The same behavior is observed with other configurations, including: maxFailures: 0 Generate the Allure report using: allure generate allure-results; allure open The configured Quality Gate should be evaluated during report generation. If the configured rule is not satisfied, the CLI should fail (non-zero exit code) or otherwise indicate that the Quality Gate has failed. Actual Behavior The report is generated successfully regardless of the configured Quality Gate. The configured rules do not appear to affect the generation process, even when the execution results clearly violate the configured thresholds. Additional Notes |
Replies: 1 comment
|
Hello @jvazquez-ccc , thank you so much for posting this question Conceptually, allure generate command does not invoke evaluation of Quality Gates thresholds configured for Allure 3. If you would like to apply quality gates to existing allure-results generated earlier by the runner with allure Integration, you could use the explicit command (doc link for more details)
However, this will produce command line results and the report as such will not be modified If you would like to incorporate Quality Gates in the Report, you should run your tests with allure3 wrapper - something like
instead of pure runner command
Note that Allure3 generates report automatically, so allure generate command can be omitted, and after allure run you can proceed directly to allure open Hopefully this has been useful, do let me know if you have any further questions |
Hello @jvazquez-ccc , thank you so much for posting this question
Conceptually, allure generate command does not invoke evaluation of Quality Gates thresholds configured for Allure 3. If you would like to apply quality gates to existing allure-results generated earlier by the runner with allure Integration, you could use the explicit command (doc link for more details)
allure quality-gateHowever, this will produce command line results and the report as such will not be modified
If you would like to incorporate Quality Gates in the Report, you should run your tests with allure3 wrapper - something like
allure run --config ./allurerc.mjs -- gradle clean test --alluredir allure-resultsinst…