Skip to content

Return the correct exit code if all tests have been skipped in unittest #106584

Closed
Closed
Return the correct exit code if all tests have been skipped in unittest#106584
Labels
3.12only security fixes3.13only security fixesstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@EliseevEgor
EliseevEgor
opened on Jul 10, 2023

Bug report

In Python 3.12, a new exit code (5, instead of 0) was added if all tests were skipped in unittest. But in some cases it still returns 0. See code examples.

Code example 1:

import unittest


class TestSimple(unittest.TestCase):

    @classmethod
    def setUpClass(cls):
        raise unittest.SkipTest("Skip whole Case")

    def test_true(self):
        self.assertTrue(True)

    def test_false(self):
        self.assertTrue(False, msg="Is not True")

Output:

Ran 0 tests in 0.000s

NO TESTS RAN (skipped=1)

Ok, we get exit code 5 here.

Code example 2:

import unittest


class TestCase(unittest.TestCase):
    @unittest.skip("something")
    def test(self):
        self.assertTrue(False)

Output:

Ran 1 test in 0.000s

OK (skipped=1)

Here we can see output 0. Ran one test and skipped. But I think it is incorrect, because all tests were skipped.

Your environment

  • CPython versions tested on: 3.12
  • Operating system and architecture: I think it does not matter.

Linked PRs

Activity

  1. EliseevEgor
    added
    type-bugAn unexpected behavior, bug, or error
    on Jul 10, 2023
  2. EliseevEgor
    added a commit that references this issue on Jul 10, 2023

    pythonGH-106584: Fix exit code for unittest in Python 3.12

    6e1be71
  3. AlexWaygood
    added
    stdlibStandard Library Python modules in the Lib/ directory
    3.12only security fixes
    3.13only security fixes
    on Jul 10, 2023
  4. terryjreedy
    added this to Unittest & doctest issueson Jul 10, 2023
  5. github-project-automation
    moved this to Todo in Unittest & doctest issueson Jul 10, 2023
  6. EliseevEgor
    added 4 commits that reference this issue on Jul 11, 2023

    pythonGH-106584: Apply comment suggestion

    43248a9

    pythonGH-106584: Remove the version info in the NEWS

    901cf01

    pythonGH-106584: Remove empty line in the NEWS

    1ab5cfa

    pythonGH-106584: Check buffers before seek

    6313c98
  7. ambv
    added a commit that references this issue on Sep 22, 2023

    gh-106584: Fix exit code for unittest in Python 3.12 (#106588)

    8fc0713
  8. miss-islington
    added a commit that references this issue on Sep 22, 2023

    pythongh-106584: Fix exit code for unittest in Python 3.12 (pythonGH-…

    95fcc0d
  9. csm10495
    added a commit that references this issue on Sep 28, 2023

    pythongh-106584: Fix exit code for unittest in Python 3.12 (python#10…

    53fda70
  10. Yhg1s
    added a commit that references this issue on Oct 2, 2023

    [3.12] gh-106584: Fix exit code for unittest in Python 3.12 (GH-106588)…

    551aa6a

  11. 1 remaining item

  12. hugovk
    closed this as completedon Nov 10, 2023
  13. github-project-automation
    moved this from Todo to Done in Unittest & doctest issueson Nov 10, 2023
  14. serhiy-storchaka
    added a commit that references this issue on Jan 23, 2024

    Revert "pythongh-106584: Fix exit code for unittest in Python 3.12 (p…

    15d1968

  15. serhiy-storchaka commented on Jan 23, 2024

    @serhiy-storchaka
    serhiy-storchaka
    on Jan 23, 2024
    Member

    I think that it was a wrong change, and not only because it makes startTest() and stopTest() calls unbalanced (#113267), but because it does not solve the original issue. The third example should be equivalent to the second example:

    import unittest

class TestCase(unittest.TestCase):
    def test(self):
        self.skipTest("something")

    But it gets error code 0 while the second example gets error code 5.

    It also gives a wrong answer to the problem. #113661 fixes it in the opposite way.

  16. serhiy-storchaka
    reopened this on Jan 23, 2024
  17. github-project-automation
    moved this from Done to In Progress in Unittest & doctest issueson Jan 23, 2024
  18. serhiy-storchaka
    added a commit that references this issue on Feb 4, 2024

    gh-113267: Revert "gh-106584: Fix exit code for unittest in Python 3.…

    ecabff9
  19. miss-islington
    added a commit that references this issue on Feb 4, 2024

    pythongh-113267: Revert "pythongh-106584: Fix exit code for unittest …

    b79c37f
  20. serhiy-storchaka
    added a commit that references this issue on Feb 4, 2024

    [3.12] gh-113267: Revert "gh-106584: Fix exit code for unittest in Py…

    0a73763
  21. aisk
    added a commit that references this issue on Feb 11, 2024

    pythongh-113267: Revert "pythongh-106584: Fix exit code for unittest …

    9d44784
  22. fsc-eriker
    added a commit that references this issue on Feb 14, 2024

    pythongh-113267: Revert "pythongh-106584: Fix exit code for unittest …

    39378cc

  23. carljm commented on Feb 29, 2024

    @carljm
    carljm
    on Feb 29, 2024
    Member

    Looks like this change has been reverted, and #113661 PRs are merged also, and that issue is closed. Seems there is nothing left to do here?

  24. carljm
    closed this as completedon Feb 29, 2024
  25. github-project-automation
    moved this from In Progress to Done in Unittest & doctest issueson Feb 29, 2024
  26. Glyphack
    added a commit that references this issue on Sep 2, 2024

    pythongh-106584: Fix exit code for unittest in Python 3.12 (python#10…

    4eaba4a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.12only security fixes3.13only security fixesstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions