Skip to content

Non-ASCII characters in os.environ cause silent failures in test_httpservers #88813

Closed
Closed
Non-ASCII characters in os.environ cause silent failures in test_httpservers#88813
Labels
3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifetype-bugAn unexpected behavior, bug, or error

Description

@ambv
ambv
opened on Jul 15, 2021
BPO 44647
Nosy @ambv, @miss-islington
PRs
  • bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows #27161
  • [3.10] bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161) #27169
  • [3.9] bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161) #27170
  • bpo-44647: Add a permanent Unicode-valued env var to regrtest  #27187
  • [3.10] bpo-44647: Add a permanent Unicode-valued env var to regrtest (GH-27187) #27191
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-07-15.20:13:07.907>
created_at = <Date 2021-07-15.17:50:20.422>
labels = ['type-bug', '3.9', '3.10', '3.11']
title = 'Non-ASCII characters in os.environ cause silent failures in test_httpservers'
updated_at = <Date 2021-07-16.13:55:33.744>
user = 'https://github.com/ambv'

    bugs.python.org fields:

    activity = <Date 2021-07-16.13:55:33.744>
actor = 'lukasz.langa'
assignee = 'none'
closed = True
closed_date = <Date 2021-07-15.20:13:07.907>
closer = 'lukasz.langa'
components = []
creation = <Date 2021-07-15.17:50:20.422>
creator = 'lukasz.langa'
dependencies = []
files = []
hgrepos = []
issue_num = 44647
keywords = ['patch']
message_count = 6.0
messages = ['397572', '397576', '397578', '397579', '397621', '397630']
nosy_count = 2.0
nosy_names = ['lukasz.langa', 'miss-islington']
pr_nums = ['27161', '27169', '27170', '27187', '27191']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44647'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    Activity

    1. ambv commented on Jul 15, 2021

      @ambv
      ambv
      on Jul 15, 2021
      ContributorAuthor

      #67826 introduced a new test for Accept: headers in CGI HTTP servers. This test serializes all of os.environ on the server side. For non-UTF8 locales this can fail for some Unicode characters found in environment variables.

      This started failing this week on Azure Pipelines with their rollout of a new Windows 2019 image version that included a "BUILD_SOURCEVERSIONAUTHOR" env variable. For me specifically it includes a leading Unicode character so all my PRs started failing on Azure Pipelines Windows 2019 alone.

      The result was truncated output from the CGI HTTP server, like:

      ======================================================================
      FAIL: test_accept (test.test_httpservers.CGIHTTPServerTestCase) [OrderedDict([('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8')])]
      ----------------------------------------------------------------------

      Traceback (most recent call last):
  File "D:\a\1\s\lib\test\test_httpservers.py", line 860, in test_accept
    self.assertIn(expected.encode('ascii'), res.read())
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b"'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'" not found in b''

      The root cause is that the CGI script in question (cgi_file6 in test_httpservers.py) is crashing on server-side reaching the print(repr(os.environ)) line. However, this exception isn't visible on the client side where the test is running. The only visible issue is truncated output.

      I suggest adding ENSURE_UNICODE_WORKS=Łukasz to the testing env so that this never regresses.

    2. ambv
      added
      3.9 (EOL)end of life
      3.10 (EOL)end of life
      3.11only security fixes
      type-bugAn unexpected behavior, bug, or error
      on Jul 15, 2021

    3. ambv commented on Jul 15, 2021

      @ambv
      ambv
      on Jul 15, 2021
      ContributorAuthor

      New changeset 82b218f by Łukasz Langa in branch 'main':
      bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161)
      82b218f

    4. ambv commented on Jul 15, 2021

      @ambv
      ambv
      on Jul 15, 2021
      ContributorAuthor

      New changeset 356bdff by Miss Islington (bot) in branch '3.10':
      bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161) (bpo-27169)
      356bdff

    5. ambv commented on Jul 15, 2021

      @ambv
      ambv
      on Jul 15, 2021
      ContributorAuthor

      New changeset 95596d5 by Miss Islington (bot) in branch '3.9':
      bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161) (bpo-27170)
      95596d5

    6. ambv
      closed this as completedon Jul 15, 2021
    7. ambv
      closed this as completedon Jul 15, 2021

    8. ambv commented on Jul 16, 2021

      @ambv
      ambv
      on Jul 16, 2021
      ContributorAuthor

      New changeset 7915c96 by Łukasz Langa in branch 'main':
      bpo-44647: Add a permanent Unicode-valued env var to regrtest (bpo-27187)
      7915c96

    9. ambv commented on Jul 16, 2021

      @ambv
      ambv
      on Jul 16, 2021
      ContributorAuthor

      New changeset 93d36a5 by Miss Islington (bot) in branch '3.10':
      bpo-44647: Add a permanent Unicode-valued env var to regrtest (GH-27187) (bpo-27191)
      93d36a5

    10. ezio-melotti
      transferred this issue fromon Apr 10, 2022

    11. serhiy-storchaka commented on Apr 25, 2022

      @serhiy-storchaka
      serhiy-storchaka
      on Apr 25, 2022
      Member

      It introduced a regression. See #91904.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifetype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions