Skip to content

Allure Quality Gate doesn't work with the config file anymore #895

Description

@auranemartin

What happened?

I am using the allure quality-gate command to run my quality gate rules against my tests results with the following command:
npx allure quality-gate ./allure-results --config ../../libs/testlib/testlib/allure_report/allurerc.mjs
or
npx allure quality-gate ./allure-results -c ../../libs/testlib/testlib/allure_report/allurerc.mjs

i have a allure-results folder and no allure-report folder

my allurerc.mjs contains the config shown in the official documentation:

    rules: [
      {
        // use id field to make ruleset's rules easier to identify
        id: "fast-failing ruleset",
        maxFailures: 1,
        fastFail: true,
      },
      {
        id: "non-fast-failing ruleset",
        successRate: 0.95,
      },
    ],
  },

it generates a allure-report folder containing one file 'test-results.json'. This file only contains this:

    "byId": {
        "mytestid1": {
            "id": "mytestid1",
            "name": "my test name",
            "duration": 19562,
            "status": "failed"
        },
        "mytestid2": {
            "id": "mytestid2",
            "name": "my test name",
            "duration": 19321,
            "status": "failed"
        },
    }
}

i have 2 tests and both are KO, so both rules should fail but nothing is printed to the console.

However if i run the quality gate with the inline rule arguments, i get this:

npx allure quality-gate ./allure-results --max-failures 1
Quality Gate failed with following issues:

 ⨯ The number of failed tests 2 exceeds the allowed threshold value 1    maxFailures

1 quality gate rules have been failed.

if i run npx allure quality-gate ./allure-results i get an error message "Quality gate is not configured!
Add qualityGate to the config or consult help to know, how to use the command with command-line arguments"

and if i run npx allure generate ./allure-results --config ../../libs/testlib/testlib/allure_report/allurerc.mjs to generate a allure-report folder and the run npx allure quality-gate ./allure-results --config ../../libs/testlib/testlib/allure_report/allurerc.mjs, nothing is printed to the console and the same "test-results.json" file i mentioned is created inside my existing allure-report folder

So my guess is that the config file is recognized, but the rules defined in the config file aren't used

For reference, this was working properly in allure 3.14.0

How can we reproduce it?

Using the config from the official documentation in my allurerc.mjs config file:

import { defineConfig } from "allure";

export default defineConfig({
  qualityGate: {
    rules: [
      {
        // use id field to make ruleset's rules easier to identify
        id: "fast-failing ruleset",
        maxFailures: 5,
        fastFail: true,
      },
      {
        id: "non-fast-failing ruleset",
        successRate: 0.95,
      },
    ],
  },
});

Run 2 tests that are KO (i am using playwright typescript with the allure-playwright reporter) > this should generate the allure-results folder

Run npx allure quality-gate ./allure-results -c
The command runs and shows nothing in the terminal
a allure-report folder was created and contains 1 single file test-results.json

What did you expect?

The command runs and shows the results of the quality gate defined in the config file against the tests results from the allure-results folder

Environment

  • Allure 3 version: 3.16.0
  • Node.js version: v22.11.0
  • OS: Windows
  • @playwright/test": "1.59.1",

Additional context

I need the console print to work with a config file because i want to declare custom rules, and i cant run my tests directly using allure run because i have some data enrichment that needs to run between my test run and my allure report generation

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions