Skip to content

Azure Pipelines: macOS CI Tests and Ubuntu CI Tests (coverage) jobs are always skipped #109395

Closed
Closed
Azure Pipelines: macOS CI Tests and Ubuntu CI Tests (coverage) jobs are always skipped#109395
Labels
buildThe build process and cross-buildtestsTests in the Lib/test dir

Description

@sobolevn
sobolevn
opened on Sep 14, 2023

I cannot find any CI runs where these two are completed / failed. They are always skipped.
Looks like Azure only stores build result for ~1 month or so, so I cannot trace it back further.

Started: Today at 09:12
Duration: 2h 3m 40s
Evaluating: False
Result: False

Examples:

Why does it happen?

Linked PRs

Activity

  1. sobolevn
    added
    testsTests in the Lib/test dir
    buildThe build process and cross-build
    on Sep 14, 2023

  2. hugovk commented on Sep 14, 2023

    @hugovk
    hugovk
    on Sep 14, 2023
    Member

    macOS CI Tests

    The macOS tests were disabled 3.5 years ago, when GitHub Actions CI was newer: PR #18818.

    Ubuntu CI Tests (coverage)

    This is skipped because the condition:

        and(
        and(
            succeeded(),
            eq(variables['coverage'], 'true')
        ),
        eq(dependencies.Prebuild.outputs['tests.run'], 'true')
    )

    evaluates to false:

        and(
        and(
            True, 
            eq('false', 'true')
        ),
        eq(dependencies['Prebuild']['outputs']['tests.run'], 'true')
    )

    Because variables['coverage'] is 'false'.

    There is a coverage: true for the job's only step, but it looks like the above condition is evaluating to false before the step can even run. I've not checked if this is intentional.

    Are these Azure Pipelines runs doing anything that is not covered by the GitHub Actions CI?

    If not, a better course of action would be to remove them from Azure Pipelines. GitHub Actions is easier to manage, especially when it comes to restarting and maintenance. (Compare: PR #105823 that removed docs testing from AP.)

    (The same applies to the other Ubuntu and Windows jobs that are running, but there's at least some unique things run in patchcheck that hasn't been migrated over, so we can deal with these later.)

    In the issue that led to disabling the macOS jobs, @zooba outlined the required steps (#84018 (comment)):

    Deleting the files is not the right first step.

    First, it needs to be changed to a non-required check.

    Then, I can use the web UI to disable it starting.

    Then, we can remove some of the files in the directory. Others are used for the official release, and have to stay.

    @zooba, for macOS:

    For Ubuntu coverage:

    • First off, do we want to fix coverage?
    • If so, should we add something to GitHub Actions first?
    • In any case, can we remove this skipped job now?
  3. sobolevn
    added a commit that references this issue on Sep 14, 2023

    pythongh-109395: Remove macOS builds from Azure Pipelines

    497e2e6

  4. sobolevn commented on Sep 14, 2023

    @sobolevn
    sobolevn
    on Sep 14, 2023
    MemberAuthor

    @hugovk thanks a lot for the context. I've sent #109400 to see what will happen if macos-steps.yml is deleted.

    Looks like it should be fine, but we will see :)

  5. hugovk commented on Sep 14, 2023

    @hugovk
    hugovk
    on Sep 14, 2023
    Member

    Ubuntu CI Tests (coverage)

    A quick experiment that hardcodes coverage: true to check if the run completes shows it core dumped after 59 minutes:

    Compared to 15 minutes for the regular Ubuntu CI Tests build which passes.

    Here's a PR to remove the skipped coverage job: #109412

  6. zooba commented on Sep 14, 2023

    @zooba
    zooba
    on Sep 14, 2023
    Member

    Yeah, coverage just never worked, unfortunately. It was a nice idea. It's possible that the segfault at 59 minutes is just the 1 hour timeout sending a Ctrl+C at a bad time, but the tests are so slow I don't think we'd want it enabled anyway. Maybe as a scheduled (daily) job.

    Removing the files is fine. They don't do any harm by staying there, and might be helpful to someone trying to run tests on their fork, but if people are going to keep suggesting we remove them then I guess we may as well make them happy.

  7. hugovk commented on Sep 14, 2023

    @hugovk
    hugovk
    on Sep 14, 2023
    Member

    At work, the coverage is also slow on the CI. But it's faster locally, so we sometimes use it for local development. And you might just run a subset of tests for the bit of code you're working on, and check you're properly covering your new code.

    So something for the future may be to set tooling up for local development, if people would find that useful.

  8. hugovk
    added a commit that references this issue on Sep 15, 2023

    gh-109395: Remove skipped coverage job from Azure Pipelines (#109412)

    fa49390
  9. miss-islington
    added 2 commits that reference this issue on Sep 15, 2023

    pythongh-109395: Remove skipped coverage job from Azure Pipelines (py…

    2ece24a

    pythongh-109395: Remove skipped coverage job from Azure Pipelines (py…

    1be3974
  10. hugovk
    added a commit that references this issue on Sep 15, 2023

    gh-109395: Remove skipped macOS builds from Azure Pipelines (#109400)

    1ece084
  11. miss-islington
    added 2 commits that reference this issue on Sep 15, 2023

    pythongh-109395: Remove skipped macOS builds from Azure Pipelines (py…

    242dbda

    pythongh-109395: Remove skipped macOS builds from Azure Pipelines (py…

    b7838c4
  12. hugovk
    added 2 commits that reference this issue on Sep 15, 2023

    [3.11] gh-109395: Remove skipped coverage job from Azure Pipelines (G…

    72ffd80

    [3.11] gh-109395: Remove skipped macOS builds from Azure Pipelines (G…

    e6e6767
  13. Yhg1s
    added 2 commits that reference this issue on Sep 15, 2023

    [3.12] gh-109395: Remove skipped coverage job from Azure Pipelines (G…

    dba95c5

    [3.12] gh-109395: Remove skipped macOS builds from Azure Pipelines (G…

    316f0ee

  14. hugovk commented on Sep 15, 2023

    @hugovk
    hugovk
    on Sep 15, 2023
    Member

    All merged and backported, thanks!

    Let's continue AP -> GHA migration in #109408.

  15. hugovk
    closed this as completedon Sep 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    buildThe build process and cross-buildtestsTests in the Lib/test dir

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions